Medicine Meets Virtual Reality 18: NextMed - Volume 163 Studies in Health Technology and Informatics

MEDICINE MEETS VIRTUAL REALITY 18 Studies in Health Technology and Informatics This book series was started in 1990 to...

16 downloads 1938 Views 16MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Report copyright / DMCA form

DOWNLOAD PDF

MEDICINE MEETS VIRTUAL REALITY 18

Studies in Health Technology and Informatics This book series was started in 1990 to promote research conducted under the auspices of the EC programmes’ Advanced Informatics in Medicine (AIM) and Biomedical and Health Research (BHR) bioengineering branch. A driving aspect of international health informatics is that telecommunication technology, rehabilitative technology, intelligent home technology and many other components are moving together and form one integrated world of information and communication media. The complete series has been accepted in Medline. Volumes from 2005 onwards are available online. Series Editors: Dr. O. Bodenreider, Dr. J.P. Christensen, Prof. G. de Moor, Prof. A. Famili, Dr. U. Fors, Prof. A. Hasman, Prof. E.J.S. Hovenga, Prof. L. Hunter, Dr. I. Iakovidis, Dr. Z. Kolitsi, Mr. O. Le Dour, Dr. A. Lymberis, Prof. J. Mantas, Prof. M.A. Musen, Prof. P.F. Niederer, Prof. A. Pedotti, Prof. O. Rienhoff, Prof. F.H. Roger France, Dr. N. Rossing, Prof. N. Saranummi, Dr. E.R. Siegel, Prof. T. Solomonides and Dr. P. Wilson

Volume 163 Recently published in this series Vol. 162. E. Wingender (Ed.), Biological Petri Nets Vol. 161. A.C. Smith and A.J. Maeder (Eds.), Global Telehealth – Selected Papers from Global Telehealth 2010 (GT2010) – 15th International Conference of the International Society for Telemedicine and eHealth and 1st National Conference of the Australasian Telehealth Society Vol. 160. C. Safran, S. Reti and H.F. Marin (Eds.), MEDINFO 2010 – Proceedings of the 13th World Congress on Medical Informatics Vol. 159. T. Solomonides, I. Blanquer, V. Breton, T. Glatard and Y. Legré (Eds.), Healthgrid Applications and Core Technologies – Proceedings of HealthGrid 2010 Vol. 158. C.-E. Aubin, I.A.F. Stokes, H. Labelle and A. Moreau (Eds.), Research into Spinal Deformities 7 Vol. 157. C. Nøhr and J. Aarts (Eds.), Information Technology in Health Care: Socio-Technical Approaches 2010 – From Safe Systems to Patient Safety Vol. 156. L. Bos, B. Blobel, S. Benton and D. Carroll (Eds.), Medical and Care Compunetics 6 Vol. 155. B. Blobel, E.Þ. Hvannberg and V. Gunnarsdóttir (Eds.), Seamless Care – Safe Care – The Challenges of Interoperability and Patient Safety in Health Care – Proceedings of the EFMI Special Topic Conference, June 2–4, 2010, Reykjavik, Iceland Vol. 154. B.K. Wiederhold, G. Riva and S.I. Kim (Eds.), Annual Review of Cybertherapy and Telemedicine 2010 – Advanced Technologies in Behavioral, Social and Neurosciences Vol. 153. W.B. Rouse and D.A. Cortese (Eds.), Engineering the System of Healthcare Delivery ISSN 0926-9630 (print) ISSN 1879-8365 (online)

Medicine Meets Virtual Reality 18 NextMed

Edited by

James D. Westwood Susan W. Westwood MA Li Felländer-Tsai MD PhD Randy S. Haluck MD FACS Helene M. Hoffman PhD Richard A. Robb PhD Steven Senger PhD and

Kirby G. Vosburgh PhD

Amsterdam • Berlin • Tokyo • Washington, DC

© 2011 The authors. All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without prior written permission from the publisher. ISBN 978-1-60750-705-5 (print) ISBN 978-1-60750-706-2 (online) Library of Congress Control Number: 2011920396 Publisher IOS Press BV Nieuwe Hemweg 6B 1013 BG Amsterdam Netherlands fax: +31 20 687 0019 e-mail: [email protected] Distributor in the USA and Canada IOS Press, Inc. 4502 Rachael Manor Drive Fairfax, VA 22032 USA fax: +1 703 323 3668 e-mail: [email protected]

LEGAL NOTICE The publisher is not responsible for the use which might be made of the following information. PRINTED IN THE NETHERLANDS

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved.

v

Preface James D. WESTWOOD Aligned Management Associates, Inc. ENIAC, the first electronic universal digital computer, was born on Valentine’s Day 1946—a lifetime ago. It and its emerging peers were elephantine contraptions, but they evolved rapidly, increasing in speed and shrinking in size, adopting efficiencies of scale in reproduction and mutating continuously. Who are their offspring today? Five billion mobile phones and similarly ubiquitous personal and business computers in countless variations. What was once a costly academic and military project is now an everyday tool. When Medicine Meets Virtual Reality launched in 1992, computers were already popular in most of the industrialized world, although relatively expensive and clunky. (Remember the dot-matrix printer?) The Internet was about to make its commercial debut, providing a means to link all these solitary devices into a communicating, sharing, interactive meta-forum. More so than print, the computer was image-friendly. Unlike television and cinema, the computer-plus-Internet was multi-directional—users could create and share a moving image. Cinema and TV were meeting their eventual heir as “virtual reality” arrived on the scene. At MMVR, virtual reality becomes a theater for medicine, where multiple senses are engaged—sight, sound, and touch—and language and image fuse. (Taste and smell are still under-utilized, alas.) Simulation lets actors rehearse in any number of ways, interrupting and reconfiguring the plot to create the most compelling finale. Visualization alters costumes to clarify relationships, and shifts sets and lighting to sharpen focus or obscure a background. Impromptu lines are recorded for possible adoption into the standard repertoire. Audience members, who need not be physically present, may chat with the actors mid-performance or take on a role themselves. Critics can instantly share their opinions. Whether the actors and audience are physicians, patients, teachers, students, industry, military, or others with a role in contemporary healthcare, the theater of virtual reality provides a singular tool for understanding relationships. Medical information can be presented in ways not possible in books, journals, or video. That information can be manipulated, refined, recontextualized, and reconsidered. Experience finds a wider audience than would fit in a surgical suite or classroom. Therapeutic outcomes can be reverse engineered. Precisely because the theater is unreal, the risks of experimentation and failure vanish, while the opportunity to understand remains. The availability and veracity of this educational virtual theater are improving due to steady technological improvement: this is the purpose of MMVR. Most of the industrialized world is currently undergoing an economic correction whose end result is far from clear. The happier news is that many emerging economies continue to flourish during the downturn. Furthermore, knowledge resources that were once the privilege of wealthier countries are now more easily shared, via computers and the Internet, with those who are catching up. Children (and adults) are being

vi

trained on inexpensive and interconnected devices, acquiring literacy and a better chance at higher education. Healthcare is an important part of this worldwide dissemination of expertise enabled by the virtual theater of learning. As developing regions progress, their most creative minds can take part in the quest for what’s next in medicine. The vision of a better educated, more productive, and healthier global population is clarified. Someone born in 1992, as was MMVR, could be attending a university now. She or he might be working on research that is shared at this conference. We who organize MMVR would like to thank the many researchers who, for a generation, have come from around the world to meet here with the aim of making very real improvements in medicine.

vii

MMVR18 Proceedings Editors James D. Westwood MMVR18 Conference Organizer Aligned Management Associates, Inc. Susan W. Westwood MA MMVR18 Proceedings Coordinator Aligned Management Associates, Inc. Li Felländer-Tsai MD PhD Professor, Department of Orthopedics Director, Center for Advanced Medical Simulation and Training Chair, Department of Clinical Science, Intervention and Technology Karolinska University Hospital Karolinska Institutet Randy S. Haluck MD FACS Professor of Surgery Chief, Minimally Invasive Surgery and Bariatrics Vice Chair for Technology and Innovation Penn State, Hershey Medical Center Helene M. Hoffman PhD Assistant Dean, Educational Computing Adjunct Professor of Medicine Division of Medical Education School of Medicine University of California, San Diego Richard A. Robb PhD Scheller Professor in Medical Research Professor of Biophysics & Computer Science Director, Biomedical Imaging Research Laboratory Mayo Clinic College of Medicine Steven Senger PhD Professor and Chair, Department of Computer Science Professor, Department of Mathematics University of Wisconsin – La Crosse Kirby G. Vosburgh PhD Assistant Professor of Radiology Brigham & Women’s Hospital Harvard Medical School

viii

MMVR18 Organizing Committee Michael J. Ackerman PhD National Library of Medicine Kóan Jeff Baysa MD Vera List Center for Art and Politics; The New School Steve Charles MD MicroDexterity Systems; University of Tennessee Patrick C. Cregan FRACS Nepean Hospital, Sydney West Area Health Service Li Felländer-Tsai MD PhD Karolinska University Hospital; Karolinska Institutet Cali M. Fidopiastis PhD University of Alabama at Birmingham Henry Fuchs PhD University of North Carolina Walter J. Greenleaf PhD Greenleaf Medical Systems; InWorld Solutions; Virtually Better Randy S. Haluck MD FACS Penn State, Hershey Medical Center David M. Hananel CAE Healthcare Wm. LeRoy Heinrichs MD PhD Stanford University School of Medicine Helene M. Hoffman PhD University of California, San Diego Kanav Kahol PhD Arizona State University Mounir Laroussi PhD Old Dominion University Heinz U. Lemke PhD Technical University Berlin Alan Liu PhD Uniformed Services University

ix

Bertalan Meskó MD University of Debrecen; Webicina.com Greg T. Mogel MD Kaiser Permanente Kevin N. Montgomery PhD Stanford University Makoto Nonaka MD PhD Foundation for International Scientific Advancement Roger Phillips PhD CEng FBCS CIPT University of Hull; Vertual, Ltd. Carla M. Pugh MD PhD Northwestern University Giuseppe Riva PhD Università Cattolica del Sacro Cuore di Milano Albert A. Rizzo PhD University of Southern California Richard A. Robb PhD Mayo Clinic College of Medicine Jannick P. Rolland PhD University of Rochester; University of Central Florida Anand P. Santhanam PhD University of California, Los Angeles Richard M. Satava MD FACS University of Washington Steven Senger PhD University of Wisconsin – La Crosse Ramin Shahidi PhD Stanford University School of Medicine Yunhe Shen PhD University of Minnesota Marshall Smith MD PhD Banner Good Samaritan Medical Center Thomas Sangild Sørensen PhD University of Aarhus

x

Don Stredney Ohio Supercomputer Center; The Ohio State University Julie A. Swain MD U.S. Food and Drug Administration Robert M. Sweet MD University of Minnesota Kirby G. Vosburgh PhD Brigham & Women’s Hospital; Harvard Medical School Dave Warner MD PhD MindTel LLC; Institute for Interventional Informatics Suzanne J. Weghorst MA MS University of Washington Brenda K. Wiederhold PhD MBA BCIA Virtual Reality Medical Institute Mark Wiederhold MD PhD Virtual Reality Medical Center Ozlem Yardimci PhD Baxter Healthcare Corporation

xi

Contents Preface James D. Westwood Conference Organization Evaluation of a VR and Stereo-Endoscopic Tool to Facilitate 3rd Ventriculostomy Kamyar Abhari, Sandrine de Ribaupierre, Terry Peters and Roy Eagleson Sleep Dysfunctions Influence Decision Making in Undemented Parkinson’s Disease Patients: A Study in a Virtual Supermarket Giovanni Albani, Simona Raspelli, Laura Carelli, Lorenzo Priano, Riccardo Pignatti, Francesca Morganti, Andrea Gaggioli, Patrice L. Weiss, Rachel Kizony, Noomi Katz, Alessandro Mauro and Giuseppe Riva Visual Tracking of Laparoscopic Instruments in Standard Training Environments Brian F. Allen, Florian Kasper, Gabriele Nataneli, Erik Dutson and Petros Faloutsos On the Use of Laser Scans to Validate Reverse Engineering of Bony Anatomy Joseph B. Anstey, Erin J. Smith, Brian Rasquinha, John F. Rudan and Randy E. Ellis Classification of Pulmonary System Diseases Patterns Using Flow-Volume Curve Hossein Arabalibeik, Samaneh Jafari and Khosro Agin Cost-Efficient Suturing Simulation with Pre-Computed Models Venkata Sreekanth Arikatla, Ganesh Sankaranarayanan and Suvranu De Anesthesia Residents’ Preference for Learning Interscalene Brachial Plexus Block (ISBPB): Traditional Winnie’s Technique vs. Ultrasound-Guided Technique Imad T. Awad, Colin Sinclair, Ewen W. Chen, Colin J.L. McCartney, Jeffrey J.H. Cheung and Adam Dubrowski Fuzzy Control of a Hand Rehabilitation Robot to Optimize the Exercise Speed in Passive Working Mode Mina Arab Baniasad, Mohammad Akbar, Aria Alasty and Farzam Farahmand Engaging Media for Mental Health Applications: The EMMA Project R. Baños, C. Botella, S. Quero, A. García-Palacios and M. Alcañiz NeuroSim – The Prototype of a Neurosurgical Training Simulator Florian Beier, Stephan Diederich, Kirsten Schmieder and Reinhard Männer Low-Cost, Take-Home, Beating Heart Simulator for Health-Care Education Devin R. Berg, Andrew Carlson, William K. Durfee, Robert M. Sweet and Troy Reihsen An Adaptive Signal-Processing Approach to Online Adaptive Tutoring Bryan Bergeron and Andrew Cline Comparison of a Disposable Bougie Versus a Newly Designed Malleable Bougie in the Intubation of a Difficult Manikin Airway Ben H. Boedeker, Mary Bernhagen, David J. Miller and W. Bosseau Murray Improving Fiberoptic Intubation with a Novel Tongue Retraction Device Ben H. Boedeker, Mary Bernhagen, David J. Miller, Thomas A. Nicholas IV, Andrew Linnaus and W.B. Murray

v vii 1

8

11

18

25 31

36

39

44 51 57

60

65 68

xii

Combined Intubation Training (Simulated and Human) for 4th Year Medical Students: The Center for Advanced Technology and Telemedicine Airway Training Program Ben H. Boedeker, Mary Bernhagen, Thomas A. Nicholas IV and W. Bosseau Murray Battlefield Tracheal Intubation Training Using Virtual Simulation: A Multi Center Operational Assessment of Video Laryngoscope Technology Ben H. Boedeker, Kirsten A. Boedeker, Mary A. Bernhagen, David J. Miller and Timothy Lacy Intubation Success Rates and Perceived User Satisfaction Using the Video Laryngoscope to Train Deploying Far Forward Combat Medical Personnel Ben H. Boedeker, Mary A. Barak-Bernhagen, Kirsten A. Boedeker and W. Bosseau Murray Field Use of the STORZ C-MAC™ Video Laryngoscope in Intubation Training with the Nebraska National Air Guard Ben H. Boedeker, Mary A. Bernhagen, David J. Miller, Nikola Miljkovic, Gail M. Kuper and W. Bosseau Murray The Combined Use of Skype™ and the STORZ CMAC™ Video Laryngoscope in Field Intubation Training with the Nebraska National Air Guard Ben H. Boedeker, Mary Bernhagen, David J. Miller, Nikola Miljkovic, Gail M. Kuper and W. Bosseau Murray Online Predictive Tools for Intervention in Mental Illness: The OPTIMI Project Cristina Botella, Inés Moragrega, R. Baños and Azucena García-Palacios An Integrated Surgical Communication Network – SurgON Richard D. Bucholz, Keith A. Laycock, Leslie L. McDurmont and William R. MacNeil Web-Accessible Interactive Software of 3D Anatomy Representing Pathophysiological Conditions to Enhance the Patient-Consent Process for Procedures D. Burke, X. Zhou, V. Rotty, V. Konchada, Y. Shen, B. Konety and R. Sweet Fast Adaptation of Pre-Operative Patient Specific Models to Real-Time Intra-Operative Volumetric Data Streams Bruce M. Cameron, Maryam E. Rettmann, David R. Holmes III and Richard A. Robb Realistic Visualization of Living Brain Tissue Llyr ap Cenydd, Annette Walter, Nigel W. John, Marina Bloj and Nicholas Phillips A Virtual Surgical Environment for Rehearsal of Tympanomastoidectomy Sonny Chan, Peter Li, Dong Hoon Lee, J. Kenneth Salisbury and Nikolas H. Blevins Acquisition of Technical Skills in Ultrasound-Guided Regional Anesthesia Using a High-Fidelity Simulator Jeffrey J.H. Cheung, Ewen W. Chen, Yaseen Al-Allaq, Nasim Nikravan, Colin J.L. McCartney, Adam Dubrowski and Imad T. Awad MeRiTS: Simulation-Based Training for Healthcare Professionals David Chodos, Eleni Stroulia and Sharla King A Framework for Treatment of Autism Using Affective Computing Seong Youb Chung and Hyun Joong Yoon

71

74

77

80

83

86 93

96

99

105

112

119

125 132

xiii

Modification of Commercial Force Feedback Hardware for Needle Insertion Simulation Timothy R. Coles, Nigel W. John, Giuseppe Sofia, Derek A. Gould and Darwin G. Caldwell Visualization of Pelvic Floor Reflex and Voluntary Contractions Christos E. Constantinou, Daniel Korenblum and Bertha Chen Mixed Virtual Reality Simulation – Taking Endoscopic Simulation One Step Further O. Courteille, L. Felländer-Tsai, L. Hedman, A. Kjellin, L. Enochsson, G. Lindgren and U. Fors A Serious Game for Off-Pump Coronary Artery Bypass Surgery Procedure Training Brent Cowan, Hamed Sabri, Bill Kapralos, Fuad Moussa, Sayra Cristancho and Adam Dubrowski Progressive Simulation-Based Program for Training Cardiac Surgery-Related Skills Sayra Cristancho, Fuad Moussa, Alex Monclou, Camilo Moncayo, Claudia Rueda and Adam Dubrowski MSMS Software for VR Simulations of Neural Prostheses and Patient Training and Rehabilitation Rahman Davoodi and Gerald E. Loeb Virtual Reality System in Conjunction with Neurorobotics and Neuroprosthetics for Rehabilitation of Motor Disorders Alessandro De Mauro, Eduardo Carrasco, David Oyarzun, Aitor Ardanza, Anselmo Frizera Neto, Diego Torricelli, José Luis Pons, Angel Gil and Julian Florez Modeling the Thermal Effect of the Bipolar Electrocautery for Neurosurgery Simulation Sébastien Delorme, Anne Cabral, Fábio Ayres and Di Jiang CliniSpace™: A Multiperson 3D Online Immersive Training Environment Accessible Through a Browser Parvati Dev, W. LeRoy Heinrichs and Patricia Youngblood Medical Education Through Virtual Worlds: The HLTHSIM Project Roy Eagleson, Sandrine de Ribaupierre, Sharla King and Eleni Stroulia Ubiquitous Health in Practice: The Interreality Paradigm Andrea Gaggioli, Simona Raspelli, Alessandra Grassi, Federica Pallavicini, Pietro Cipresso, Brenda K. Wiederhold and Giuseppe Riva Bench Model Surgical Skill Training Improves Novice Ability to Multitask: A Randomized Controlled Study Lawrence Grierson, Megan Melnyk, Nathan Jowlett, David Backstein and Adam Dubrowski A Design of Hardware Haptic Interface for Gastrointestinal Endoscopy Simulation Yunjin Gu and Doo Yong Lee Open Surgery Simulation of Inguinal Hernia Repair Niels Hald, Sudip K. Sarker, Paul Ziprin, Pierre-Frederic Villard and Fernando Bello SML: SoFMIS Meta Language for Surgical Simulation Tansel Halic and Suvranu De

135

138

144

147

150

156

163

166

173 180 185

192

199 202

209

xiv

A Software Framework for Multimodal Interactive Simulations (SoFMIS) Tansel Halic, Sreekanth A. Venkata, Ganesh Sankaranarayanan, Zhonghua Lu, Woojin Ahn and Suvranu De Simulation of Vaginal Wall Biomechanical Properties from Pelvic Floor Closure Forces Map Shin Hasegawa, Yuki Yoshida, Daming Wei, Sadao Omata and Christos E. Constantinou A Generalized Haptic Feedback Approach for Arbitrarily Shaped Objects Rui Hu, Kenneth E. Barner and Karl V. Steiner Piezoelectric Driven Non-Toxic Injector for Automated Cell Manipulation H.B. Huang, Hao Su, H.Y. Chen and J.K. Mills Virtual Arthroscopy Trainer for Minimally Invasive Surgery Vassilios Hurmusiadis, Kawal Rhode, Tobias Schaeffter and Kevin Sherman Design for Functional Occlusal Surface of CAD/CAM Crown Using VR Articulator Tomoko Ikawa, Takumi Ogawa, Yuko Shigeta, Shintaro Kasama, Rio Hirabayashi, Shunji Fukushima, Asaki Hattori and Naoki Suzuki Biopsym: A Learning Environment for Trans-Rectal Ultrasound Guided Prostate Biopsies Thomas Janssoone, Grégoire Chevreau, Lucile Vadcard, Pierre Mozer and Jocelyne Troccaz Comparison of Reaching Kinematics During Mirror and Parallel Robot Assisted Movements Zahra Kadivar, Cynthia Sung, Zachary Thompson, Marcia O’Malley, Michael Liebschner and Zhigang Deng Serious Games in the Classroom: Gauging Student Perceptions Bill Kapralos, Sayra Cristancho, Mark Porte, David Backstein, Alex Monclou and Adam Dubrowski Influence of Metal Artifacts on the Creation of Individual 3D Cranio-Mandibular Models Shintaro Kasama, Takumi Ogawa, Tomoko Ikawa, Yuko Shigeta, Shinya Hirai, Shunji Fukushima, Asaki Hattori and Naoki Suzuki Web-Based Stereoscopic Visualization for the Global Anatomy Classroom Mathias Kaspar, Fred Dech, Nigel M. Parsad and Jonathan C. Silverstein Expanding the Use of Simulators as Assessment Tools: The New Pop Quiz Abby R. Kaye, Lawrence H. Salud, Zachary B. Domont, Katherine Blossfield Iannitelli and Carla M. Pugh Validation of Robotic Surgery Simulator (RoSS) Thenkurussi Kesavadas, Andrew Stegemann, Gughan Sathyaseelan, Ashirwad Chowriappa, Govindarajan Srimathveeravalli, Stéfanie Seixas-Mikelus, Rameella Chandrasekhar, Gregory Wilding and Khurshid Guru Practical Methods for Designing Medical Training Simulators Thomas Knott, Sebastian Ullrich and Torsten Kuhlen The Minnesota Pelvic Trainer: A Hybrid VR/Physical Pelvis for Providing Virtual Mentorship Vamsi Konchada, Yunhe Shen, Dan Burke, Omer B. Argun, Anthony Weinhaus, Arthur G. Erdman and Robert M. Sweet

213

218

224 231 236

239

242

247

254

261

264 271

274

277

280

xv

Registration Stability of Physical Templates in Hip Surgery Manuela Kunz, John F. Rudan, Gavin C.A. Wood and Randy E. Ellis Real-Time 3D Avatars for Tele-Rehabilitation in Virtual Reality Gregorij Kurillo, Tomaz Koritnik, Tadej Bajd and Ruzena Bajcsy Fundamentals of Gas Phase Plasmas for Treatment of Human Tissue Mark J. Kushner and Natalia Yu. Babaeva VR-Based Training and Assessment in Ultrasound-Guided Regional Anesthesia: From Error Analysis to System Design Erik Lövquist, Owen O’Sullivan, Donnchadh Oh’Ainle, Graham Baitson, George Shorten and Nick Avis Real-Time Electrocautery Simulation for Laparoscopic Surgical Environments Zhonghua Lu, Venkata Sreekanth Arikatla, Dingfang Chen and Suvranu De Guidewire and Catheter Behavioural Simulation Vincent Luboz, Jianhua Zhai, Tolu Odetoyinbo, Peter Littler, Derek Gould, Thien How and Fernando Bello Design and Implementation of a Visual and Haptic Simulator in a Platform for a TEL System in Percutaneuos Orthopedic Surgery Vanda Luengo, Aurelie Larcher and Jérôme Tonetti Computational Modeling of Human Head Electromagnetics for Source Localization of Milliscale Brain Dynamics Allen D. Malony, Adnan Salman, Sergei Turovets, Don Tucker, Vasily Volkov, Kai Li, Jung Eun Song, Scott Biersdorff, Colin Davey, Chris Hoge and David Hammond Simulation and Modeling of Metamorphopsia with a Deformable Amsler Grid Anabel Martin-Gonzalez, Ines Lanzl, Ramin Khoramnia and Nassir Navab Development of a Customizable Software Application for Medical Imaging Analysis and Visualization Marisol Martinez-Escobar, Catherine Peloquin, Bethany Juhnke, Joanna Peddicord, Sonia Jose, Christian Noon, Jung Leng Foo and Eliot Winer Pneumoperitoneum Technique Simulation in Laparoscopic Surgery on Lamb Liver Samples and 3D Reconstruction F. Martínez-Martínez, M.J. Rupérez, M.A. Lago, F. López-Mir, C. Monserrat and M. Alcañíz Technology Transfer at the University of Nebraska Medical Center Kulia Matsuo, Henry J. Runge, David J. Miller, Mary A. Barak-Bernhagen and Ben H. Boedeker CvhSlicer: An Interactive Cross-Sectional Anatomy Navigation System Based on High-Resolution Chinese Visible Human Data Q. Meng, Y.P. Chui, J. Qin, W.H. Kwok, M. Karmakar and P.A. Heng Generation of Connectivity-Preserving Surface Models of Multiple Sclerosis Lesions Oscar Meruvia-Pastor, Mei Xiao, Jung Soh and Christoph W. Sensen A Comparison of Videolaryngoscopic Technologies David J. Miller, Nikola Miljkovic, Chad Chiesa, Nathan Schulte, John B. Callahan Jr. and Ben H. Boedeker Telemedicine Using Free Voice over Internet Protocol (VoIP) Technology David J. Miller, Nikola Miljkovic, Chad Chiesa, John B. Callahan Jr., Brad Webb and Ben H. Boedeker

283 290 297

304

311 317

324

329

336

343

348

351

354

359 366

369

xvi

iMedic: A Two-Handed Immersive Medical Environment for Distributed Interactive Consultation Paul Mlyniec, Jason Jerald, Arun Yoganandan, F. Jacob Seagull, Fernando Toledo and Udo Schultheis Patient Specific Surgical Simulator for the Evaluation of the Movability of Bimanual Robotic Arms Andrea Moglia, Giuseppe Turini, Vincenzo Ferrari, Mauro Ferrari and Franco Mosca CyberMedVPS: Visual Programming for Development of Simulators Aline M. Morais and Liliane S. Machado A Bloodstream Simulation Based on Particle Method Masashi Nakagawa, Nobuhiko Mukai, Kiyomi Niki and Shuichiro Takanashi Laser Induced Shockwaves on Flexible Polymers for Treatment of Bacterial Biofilms Artemio Navarro, Zachary D. Taylor, David Beenhouwer, David A. Haake, Vijay Gupta, Warren S. Grundfest Virtual Reality Haptic Human Dissection Caroline Needham, Caroline Wilkinson and Roger Soames The Tool Positioning Tutor: A Target-Pose Tracking and Display System for Learning Correct Placement of a Medical Device Douglas A. Nelson and Joseph T. Samosky A Cost Effective Simulator for Education of Ultrasound Image Interpretation and Probe Manipulation S.A. Nicolau, A. Vemuri, H.S. Wu, M.H. Huang, Y. Ho, A. Charnoz, A. Hostettler, C. Forest, L. Soler and J. Marescaux A Portable Palpation Training Platform with Virtual Human Patient Tyler Niles, D. Scott Lind and Kyle Johnsen A Development of Surgical Simulator for Training of Operative Skills Using Patient-Specific Data Masato Ogata, Manabu Nagasaka, Toru Inuiya, Kazuhide Makiyama and Yoshinobu Kubota Virtual Reality Image Applications for Treatment Planning in Prosthodontic Dentistry Takumi Ogawa, Tomoko Ikawa, Yuko Shigeta, Shintaro Kasama, Eriko Ando, Shunji Fukushima, Asaki Hattori and Naoki Suzuki The Initiation of a Preoperative and Postoperative Telemedicine Urology Clinic Eugene S. Park, Ben H. Boedeker, Jennifer L. Hemstreet and George P. Hemstreet Modeling Surgical Skill Learning with Cognitive Simulation Shi-Hyun Park, Irene H. Suh, Jung-hung Chien, Jaehyon Paik, Frank E. Ritter, Dmitry Oleynikov, Ka-Chun Siu Virtual Reality Stroop Task for Neurocognitive Assessment Thomas D. Parsons, Christopher G. Courtney, Brian Arizmendi and Michael Dawson Implementation of Virtual Online Patient Simulation V. Patel, R. Aggarwal, D. Taylor and A. Darzi Patient-Specific Cases for an Ultrasound Training Simulator Kresimir Petrinec, Eric Savitsky and Cheryl Hein

372

379

386 389

394

397

400

403

408

415

422

425

428

433

440 447

xvii

Stereo Image-Based Arm Tracking for In Vivo Surgical Robotics Eric Psota, Kyle Strabala, Jason Dumpert, Lance C. Pérez, Shane Farritor and Dmitry Oleynikov A Simulation Framework for Wound Closure by Suture for the Endo Stitch Suturing Instrument Sukitti Punak and Sergei Kurenov Simplified Cosserat Rod for Interactive Suture Modeling Sukitti Punak and Sergei Kurenov A Design for Simulating and Validating the Nuss Procedure for the Minimally Invasive Correction of Pectus Excavatum Krzysztof J. Rechowicz, Robert Kelly, Michael Goretsky, Frazier W. Frantz, Stephen B. Knisley, Donald Nuss and Frederic D. McKenzie AISLE: An Automatic Volumetric Segmentation Method for the Study of Lung Allometry Hongliang Ren and Peter Kazanzides Development of a Wireless Hybrid Navigation System for Laparoscopic Surgery Hongliang Ren, Denis Rank, Martin Merdes, Jan Stallkamp and Peter Kazanzides Visualization of Probabilistic Fiber Tracts in Virtual Reality Tobias Rick, Anette von Kapri, Svenja Caspers, Katrin Amunts, Karl Zilles and Torsten Kuhlen NeuroVR 2 – A Free Virtual Reality Platform for the Assessment and Treatment in Behavioral Health Care Giuseppe Riva, Andrea Gaggioli, Alessandra Grassi, Simona Raspelli, Pietro Cipresso, Federica Pallavicini, Cinzia Vigna, Andrea Gagliati, Stefano Gasco and Giuseppe Donvito Personal Health Systems for Mental Health: The European Projects Giuseppe Riva, Rosa Banos, Cristina Botella, Andrea Gaggioli and Brenda K. Wiederhold An Intelligent Virtual Human System for Providing Healthcare Information and Support Albert A. Rizzo, Belinda Lange, John G. Buckwalter, Eric Forbell, Julia Kim, Kenji Sagae, Josh Williams, Barbara O. Rothbaum, JoAnn Difede, Greg Reger, Thomas Parsons and Patrick Kenny Virtual Reality Applications for Addressing the Needs of Those Aging with Disability Albert Rizzo, Phil Requejo, Carolee J. Winstein, Belinda Lange, Gisele Ragusa, Alma Merians, James Patton, Pat Banerjee and Mindy Aisen The Validation of an Instrumented Simulator for the Assessment of Performance and Outcome of Knot Tying Skill: A Pilot Study David Rojas, Sayra Cristancho, Claudia Rueda, Lawrence Grierson, Alex Monclou and Adam Dubrowski Manual Accuracy in Comparison with a Miniature Master Slave Device – Preclinical Evaluation for Ear Surgery A. Runge, M. Hofer, E. Dittrich, T. Neumuth, R. Haase, M. Strauss, A. Dietz, T. Lüth and G. Strauss

454

461 466

473

476 479

486

493

496

503

510

517

524

xviii

Are Commercially Available Simulators Durable Enough for Classroom Use? Jonathan C. Salud, Katherine Blossfield Iannitelli, Lawrence H. Salud and Carla M. Pugh Toward a Simulation and Assessment Method for the Practice of Camera-Guided Rigid Bronchoscopy Lawrence H. Salud, Alec R. Peniche, Jonathan C. Salud, Alberto L. de Hoyos and Carla M. Pugh Use of Sensor Technology to Explore the Science of Touch Lawrence H. Salud and Carla M. Pugh Real-Time “X-Ray Vision” for Healthcare Simulation: An Interactive Projective Overlay System to Enhance Intubation Training and Other Procedural Training Joseph T. Samosky, Emma Baillargeon, Russell Bregman, Andrew Brown, Amy Chaya, Leah Enders, Douglas A. Nelson, Evan Robinson, Alison L. Sukits and Robert A. Weaver Toward a Comprehensive Hybrid Physical-Virtual Reality Simulator of Peripheral Anesthesia with Ultrasound and Neurostimulator Guidance Joseph T. Samosky, Pete Allen, Steve Boronyak, Barton Branstetter, Steven Hein, Mark Juhas, Douglas A. Nelson, Steven Orebaugh, Rohan Pinto, Adam Smelko, Mitch Thompson and Robert A. Weaver A Fixed Point Proximity Method for Extended Contact Manipulation of Deformable Bodies with Pivoted Tools in Multimodal Virtual Environments Ganesh Sankaranarayanan, Zhonghua Lu and Suvranu De Collision and Containment Detection Between Biomechanically Based Eye Muscle Volumes Graciela Santana Sosa and Thomas Kaltofen Visualization of 3D Volumetric Lung Dynamics for Real-Time External Beam Lung Radiotherapy Anand P. Santhanam, Harini Neelakkantan, Yugang Min, Nicolene Papp, Akash Bhargava, Kevin Erhart, Xiang Long, Rebecca Mitchell, Eduardo Divo, Alain Kassab, Olusegun Ilegbusi, Bari H. Ruddy, Jannick P. Rolland, Sanford L. Meeks and Patrick A. Kupelian Laser Surgery Simulation Platform: Toward Full-Procedure Training and Rehearsal for Benign Prostatic Hyperplasia (BPH) Therapy Yunhe Shen, Vamsi Konchada, Nan Zhang, Saurabh Jain, Xiangmin Zhou, Daniel Burke, Carson Wong, Culley Carson, Claus Roehrborn and Robert Sweet 3D Tracking of Surgical Instruments Using a Single Camera for Laparoscopic Surgery Simulation Sangkyun Shin, Youngjun Kim, Hyunsoo Kwak, Deukhee Lee and Sehyung Park Perceptual Metrics: Towards Better Methods for Assessing Realism in Laparoscopic Simulators Ravikiran B. Singapogu, Christopher C. Pagano, Timothy C. Burg and Karen J.K.L. Burg Role of Haptic Feedback in a Basic Laparoscopic Task Requiring Hand-Eye Coordination Ravikiran B. Singapogu, Christopher C. Pagano, Timothy C. Burg, Karen J.K.L. Burg and Varun V. Prabhu

531

535

542

549

552

555

560

567

574

581

588

591

xix

A Model for Flexible Tools Used in Minimally Invasive Medical Virtual Environments Francisco Soler, M. Victoria Luzon, Serban R. Pop, Chris J. Hughes, Nigel W. John and Juan Carlos Torres Segmentation of 3D Vasculatures for Interventional Radiology Simulation Yi Song, Vincent Luboz, Nizar Din, Daniel King, Derek Gould, Fernando Bello and Andy Bulpitt EEG-Based “Serious” Games and Monitoring Tools for Pain Management Olga Sourina, Qiang Wang and Minh Khoa Nguyen A New Part Task Trainer for Teaching and Learning Confirmation of Endotracheal Intubation Cyle Sprick, Harry Owen, Cindy Hein and Brigid Brown Mobile Three Dimensional Gaze Tracking Josef Stoll, Stefan Kohlbecher, Svenja Marx, Erich Schneider and Wolfgang Einhäuser High-Field MRI-Compatible Needle Placement Robot for Prostate Interventions Hao Su, Alex Camilo, Gregory A. Cole, Nobuhiko Hata, Clare M. Tempany and Gregory S. Fischer Electromyographic Correlates of Learning During Robotic Surgical Training in Virtual Reality Irene H. Suh, Mukul Mukherjee, Ryan Schrack, Shi-Hyun Park, Jung-hung Chien, Dmitry Oleynikov and Ka-Chun Siu Web-Based Interactive Volume Rendering Stefan Suwelack, Sebastian Maier, Roland Unterhinninghofen and Rüdiger Dillmann A Method of Synchronization for Haptic Collaborative Virtual Environments in Multipoint and Multi-Level Computer Performance Systems Kazuyoshi Tagawa, Tatsuro Bito and Hiromi T. Tanaka A Hybrid Dynamic Deformation Model for Surgery Simulation Kazuyoshi Tagawa and Hiromi T. Tanaka Single and Multi-User Virtual Patient Design in the Virtual World D. Taylor, V. Patel, D. Cohen, R. Aggarwal, K. Kerr, N. Sevdalis, N. Batrick and A. Darzi Terahertz Imaging of Biological Tissues Priyamvada Tewari, Zachary D. Taylor, David Bennett, Rahul S. Singh, Martin O. Culjat, Colin P. Kealey, Jean Pierre Hubschman, Shane White, Alistair Cochran, Elliott R. Brown and Warren S. Grundfest Quantifying Surgeons’ Vigilance During Laparoscopic Operations Using Eyegaze Tracking Geoffrey Tien, Bin Zheng and M. Stella Atkins Modeling of Interaction Between a Three-Fingered Surgical Grasper and Human Spleen Mojdeh Tirehdast, Alireza Mirbagheri, Mohsen Asghari and Farzam Farahmand Quantizing the Void: Extending Web3D for Space-Filling Haptic Meshes Sebastian Ullrich, Torsten Kuhlen, Nicholas F. Polys, Daniel Evestedt, Michael Aratow and Nigel W. John Dissecting in Silico: Towards a Taxonomy for Medical Simulators Sebastian Ullrich, Thomas Knott and Torsten Kuhlen

594

599

606

611 616

623

630

635

638 645 650

653

658

663

670

677

xx

Computed Tomography as Ground Truth for Stereo Vision Measurements of Skin Amy M. Vanberlo, Aaron R. Campbell and Randy E. Ellis Towards the Visualization of Spiking Neurons in Virtual Reality Anette von Kapri, Tobias Rick, Tobias C. Potjans, Markus Diesmann and Torsten Kuhlen The Use of Virtual Training to Support Insertion of Advanced Technology at Remote Military Locations Madison I. Walker, Robert B. Walker, Jeffrey S. Morgan, Mary Bernhagen, Nicholas Markin and Ben H. Boedeker Three Dimensional Projection Environment for Molecular Design and Surgical Simulation Eric Wickstrom, Chang-Po Chen, Devakumar Devadhas, Matthew Wampole, Yuan-Yuan Jin, Jeffrey M. Sanders, John C. Kairys, Martha L. Ankeny, Rui Hu, Kenneth E. Barner, Karl V. Steiner and Mathew L. Thakur Reality Graded Exposure Therapy with Physiological Monitoring for the Treatment of Combat Related Post Traumatic Stress Disorder: A Pilot Study Dennis Patrick Wood, Jennifer Webb-Murphy, Robert N. McLay, Brenda K. Wiederhold, James L. Spira, Scott Johnston, Robert L. Koffman, Mark D. Wiederhold and Jeff Pyne Applications of Tactile Feedback in Medicine Christopher Wottawa, Richard Fan, James W. Bisley, Erik P. Dutson, Martin O. Culjat and Warren S. Grundfest Needle Insertion Simulation by Arbitrary Lagrangian-Eulerian Method Satoshi Yamaguchi, Koji Satake, Shigehiro Morikawa, Yoshiaki Shirai and Hiromi T. Tanaka Clinical Performance of Dental Fiberscope Image Guided System for Endodontic Treatment Yasushi Yamazaki, Takumi Ogawa, Yuko Shigeta, Tomoko Ikawa, Shintaro Kasama, Asaki Hattori, Naoki Suzuki, Takatsugu Yamamoto, Toshiko Ozawa and Takashi Arai A Novel Virtual Reality Environment for Preoperative Planning and Simulation of Image Guided Intracardiac Surgeries with Robotic Manipulators Erol Yeniaras, Zhigang Deng, Mushabbar A. Syed, Mark G. Davies and Nikolaos V. Tsekos Enabling Surgeons to Create Simulation-Based Teaching Modules Young In Yeo, Saleh Dindar, George Sarosi and Jörg Peters Using a Virtual Integration Environment in Treating Phantom Limb Pain Michael J. Zeher, Robert S. Armiger, James M. Burck, Courtney Moran, Janid Blanco Kiely, Sharon R. Weeks, Jack W. Tsao, Paul F. Pasquina, R. Davoodi and G. Loeb Validation of a Virtual Preoperative Evaluation Clinic: A Pilot Study Corey V. Zetterman, Bobbie J. Sweitzer, Brad Webb, Mary A. Barak-Bernhagen and Ben H. Boedeker Multifunction Robotic Platform for Natural Orifice Surgery Xiaoli Zhang, Wei Jian Chin, Chi Min Seow, Akiko Nakamura, Michael Head, Shane Farritor, Dmitry Oleynikov and Carl Nelson Maintaining Forward View of the Surgical Site for Best Endoscopic Practice Bin Zheng, Maria A. Cassera, Lee L. Swanström, Adam Meneghetti, Neely O.N. Panton and Karim A. Qayumi

680 685

688

691

696

703

710

713

716

723 730

737

740

743

xxi

Phenomenological Model of Laser-Tissue Interaction with Application to Benign Prostatic Hyperplasia (BPH) Simulation Xiangmin Zhou, Nan Zhang, Yunhe Shen, Dan Burke, Vamsi Konchada and Robert Sweet Subject Index Author Index

749

757 763

This page intentionally left blank

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-1

1

Evaluation of a VR and Stereo-Endoscopic Tool to Facilitate 3rd Ventriculostomy Kamyar ABHARI a,b , Sandrine de RIBAUPIERRE c , Terry PETERS a,b and Roy EAGLESON a,d a Imaging Research Laboratories, Robarts Research Institute b Biomedical Engineering Program, The University of Western Ontario Ontario c Department of Clinical Neurological Sciences, The University of Western Ontario, London Health Sciences Centre d Department of Electrical and Computer Engineering, Faculty of Engineering, The University of Western Ontario, London, Ontario, Canada Abstract. Endoscopic third ventriculostomy is a minimally invasive technique to treat hydrocephalus, which is a condition in which the patient is retaining excessive amount of cerebrospinal ﬂuid in the head. While this surgical procedure is fairly routine, it carries some risks, mainly associated with the lack of depth perception, since monocular endoscopes provide only 2D views. We studied the advantages given by a 3D stereoendoscope over a 2D monocular endoscope, ﬁrst by assessing the variability of stereoacuity in each subject, then in analyzing their overall correct response rate in differentiating between heights of two different images with 2D and 3D vision. Keywords. Hydrocephalus, Endoscopic Third Ventriculostomy, Stereo-endoscopy

Introduction Hydrocephalus is an abnormal accumulation of cerebreospinal ﬂuid (CSF) within the brain, and is one the most common source of developmental disability among children as it affects one in every 500-1000 live births [1]. Obstructive hydrocephalus can be treated either with a shunt, draining ﬂuid away from the head, or with an Endoscopic Third Ventriculostomy (ETV) which involves making a hole in the ventricular system to bypass the obstruction. In the last decade, ETV gradually has become the procedure of choice for obstructive hydrocephalus. The technique involves making a small perforation on the ﬂoor of the third ventricle to allow extra CSF to drain into the interpeduncular cistern. The ETV operation involves using an endoscope to navigate within the ventriclar system. There are different types of endoscope used but they all produce 2D images. Although ETV is an effective approach, it is not without risk to the patient, and the speed and accuracy of the intervention is dependent on visualization of the ﬂoor of the third ventricle and basilar artery. The basilar artery is located a few millimeters behind the clivus. Accurate localization of the basilar artery and its two most important branches

2

K. Abhari et al. / Evaluation of a VR and Stereo-Endoscopic Tool to Facilitate 3rd Ventriculostomy

(posterior cerebral arteries) is essential to avoid injuring them during the procedure. Injuring the basilar artery might lead to a major stroke or be lethal. Although the ﬂoor of the third ventricle can be transparent, in some cases it is thickened by an old infection, hemorrhage, or tumoral cells, and therefore it is impossible to see through and visualize all the structures. In these cases, the task of localization of the basilar artery is extremely difﬁcult and in some cases impossible. Unfortunately, there are no reliable visible textures on the membrane of the third ventricle that can be helpful to locate and avoid basilar artery. However, there are different structures which deform the ﬂoor of the third ventricle and provide some relief. In hydrocephalus patients, the pressure of CSF inside the ventricular system gradually reshapes the structure of the ventricles. As a result, the surface of third ventricle is also pushed down. With the pressure, the ﬂoor of the third ventricle will then adopt the shape of the underlying structures. This mechanism creates a bump on the ﬂoor of the third ventricle above the basilar artery. We believe that this curvature can be used in order to locate and avoid the basilar artery if it can be detected within the stereoendoscopic view. Without providing 3D information, however, surgeons are not able to observe this landmark and differentiate it from the other parts of the third ventricle. These structures may not be visualized with monocular endoscopes where observers suffer from the lack of depth perception. Stereoscopic endoscopes, on the other hand, can provide necessary depth information required to properly locate and visualize these structures. Two clinical centres have evaluated the use of stereoscopic neuroendoscopes in a few patients [2], but the technique has not been fully investigated, and the difference between the 2D and 3D never been studied objectively. The objective of this study is to establish a methodology to determine whether 3Dendoscopy can improve accuracy of the ETV in cases where impaired depth perception can be problematic or even catastrophic during surgery. Using a stereo-endoscopic camera, the physical structure of the brain can be observed in 3D which offers the observer appropriate depth perception of the brain’s anatomy. In this paper, we evaluate the feasibility of this approach using experiments which involve comparing the established 2D method and the proposed 3D technique in terms of its sensitivity to depth discrimination. Our hypothesis is that this method will signiﬁcantly improve the localization and avoidance of basilar artery with the goal of having safer and faster ETV interventions.

Materials and Methods 1. Materials 1.1. Virtual Environment Stereoacuity, similar to visual acuity, is a measure of the perceptual capacity to detect small differences in depth using stereo vision. Although there are commercially available tests for stereoacuity such as Random Dot Stereograms [8], they usually vary the point positions in depth, and not the size and shape of the perceived stimulus. In clinical settings, it is important to appreciate not only the relative distance between structures, but also the curvature of the surface. In addition, there are some monocular cues that can be present at the area of operation, such as small blood vessels, different opacity of the membrane, etc. Building our own stereoacuity test, allowed us to control these

K. Abhari et al. / Evaluation of a VR and Stereo-Endoscopic Tool to Facilitate 3rd Ventriculostomy

3

Figure 1. Model of the third ventricle

factors, and therefore we were able to correlate our results with results obtained in a clinical setting. It was therefore necessary to design a specialized visualization tool. The system described here extends a 3D biomedical visualization platform developed within our VASST lab (AtamaiViewer, (c) Atamai Inc., London, ON). The system makes use of a FakeSpace TM display with a high-resolution stereoscopic projection system that offers a large slanted table-top display for use in immersive visualization experiments. To begin with, a set of scale models were fabricated based on the real anatomy and workspace geometry of the third ventricle (Figure 1). First, the models were speciﬁed using a computer-aided design program. A texture map, acquired endoscopically, of the ﬂoor of the third ventricle is mapped onto the surface of our virtual models. The selected texture has no distinguishable monocular cues. Each model may include a bump on the surface similar to what the basilar artery induces in hydrocephalus patients. In live ETV surgery scenarios, this bump may have a range of prominence. The height of the bump on our models ranges from zero (no bump) to 3mm with step value of 0.1mm (i.e. 31 models in total). 1.2. VisionSense Camera The VisionSense stereoendoscope camera (VisionSense Ltd., Isreal) is a compact (3.8mm – 4.9mm) FDA-approved device, which makes it a good candidate for neurosurgical procedures. Previously designed stereoendoscopes, were not suitable for minimally invasive neurosurgeries as they are signiﬁcantly larger than commonly used endoscopes. Several studies have demonstrated the practicality of the VisionSense camera and its overall advantage over the monocular endoscopes [3] [4] [5]. 1.3. Physical Environment: Preliminary Prototype Using stereolithographic rapid-prototyping technology, seven different phantoms (ranging from 0mm to 3mm with 0.5mm step value) were created based on our computergenerated models as seen in Figure 1. A number of experiments were conducted using these models in order to determine some of key variables required for our ﬁnal prototype (Refer to section 2.3. for details). 1.4. Physical Environment: Final Prototype In order to collect preliminary data, two rapid-prototyped phantoms were placed under the VisionSense camera in each trial. This set-up brought us some undesirable effects including a gradient around the edges and a glare due to the reﬂection of the endoscope’s light. Although these effects were not pronounced, they could potentially be used as

4

K. Abhari et al. / Evaluation of a VR and Stereo-Endoscopic Tool to Facilitate 3rd Ventriculostomy

monocular cues. For this reason, new series of phantoms were made out of two parts silicone to provide smooth surface with the accuracy of 0.1mm in any preferable colour.

2. Method In this section, we outline a methodology for the design and evaluation of a stereoendoscopic method to facilitate ETV, in comparison with the current method involving monocular endoscopy. Our main goal was to examine the use of stereopsis to identify the location of the basilar artery by detecting the bump in the tissue. In order to test this, we are consequently testing the role of stereo in the task of differentiating between two different surfaces in terms of their depth. 2.1. Virtual Environment Experiment The stereoacuity test involves number of trials and a simple discrimination task. In each trial, subjects are presented with a pair of models side-by-side from the top view angle in stereo. They are asked to sit comfortably viewing the FakeSpace TM screen while wearing LCD shutter glasses. The task involves selecting the model with bigger bump by pressing the corresponding keys on the keyboard. The methodology employed is based on a standard psychophysical ’staircase’ design for establishing stereoacuity [6]. 2.2. VisionSense Experiment: Preliminary The second stage of the experiments involves using the VisionSense stereoendoscope to compare 3D and 2D visualization in terms of the accuracy of completing a task. In this stage, the experiments involved using a set of plastic phantoms and the VisionSense endoscope to make a comparison between 2D and 3D ETV interventions. Subjects’ ability to discriminate bump heights was compared in two conditions: (i) using the VisionSense camera with Stereo and Monocular cues present, and (ii) with similar views but with no stereo. Each trial involves placing two different phantoms side by side on a plexiglass holder and asking the subjects to select the one with taller bump. Using this set-up, users could observe the phantoms on the dedicated display similar to the way in which neurosurgeons do in the operating room. The task consisted of locating and selecting the target (the bump in this case) which was most prominent in 3D. The experiments are conducted once in monocular view and then later using stereo. In order to include the subjects’ stereo sensitivity proﬁle in our analysis, the virtual stereo environment and the VisionSense stereoendoscope are required to provide the same stereo effect or disparity. To fulﬁll this requirement, we calculated and varied the inter-ocular distance as well as the focal point of the virtual cameras. The distance between the blocks and the lens is also kept the same for both real and virtual environments. 2.3. VisionSense Experiment: Final In this stage of our study, the task is identical to the previous one with the difference of using the silicone phantom. For any psychophysical experiment, it is necessary to deter-

K. Abhari et al. / Evaluation of a VR and Stereo-Endoscopic Tool to Facilitate 3rd Ventriculostomy

5

mine the following variables: Start Point, Step Size, Stop Point, and Modiﬁcation of step sizes [9]. Collected data from the previous phase of study provide required information to identify these variables: 2.3.1. Initializing the sequence of trials In order to lower the total number of trials for each subject, we produce stimulus values based on an estimate of their stereoacuity threshold. By using our preliminary data, the mean of overall correct responses reaches 85% when the height difference is approximately 1.25mm for both the VisionSense and FakeSpace. Therefore, 1.25mm was selected as the ﬁrst stimulus in our series. 2.3.2. Step size In our series of stimulus displays, the step size can be deﬁned as the minimum difference between the bump height values as we move from one trial to the next. Ideally, the proper step size value should be selected as the intensity at which subjects perform discriminations halfway between chance guessing and perfect responses [9]. Since in our preliminary experiments, subjects’ performance reached almost 75% when the step size was 0.5mm, we lowered the minimum height difference by half (0.25 mm) in order to achieve the proper step size value. 2.3.3. Where to stop Finding the stop point is challenging since it can be a compromise between a large series of stimuli for higher accuracy and a small number of trials for economy in time and therefore minimizing the effect of fatigue. To be fair and efﬁcient, it is necessary to ﬁnd the trial where the correct hit value reaches its plateau as our stop point. To fulﬁll this condition, the number of trials was increased to 44 from 38 as the mean value of correct responses reached 96% in our preliminary experiments. 2.3.4. Modiﬁcation of step sizes As a common approach in psychophysical experiments [9], steps were set to be larger in the beginning of the series and gradually get smaller when the ﬁnal threshold value has been reached.

Results The data recorded during the ﬁrst and second phase (section 2.1. and 2.2. respectively) were the basis for our ﬁnal experimental design (Refer to section 2.3. for details). The overall quantitative result from the series of experiments in the ﬁnal phase is shown in Table 1 and illustrated in Figure 2. We choose a threshold of 90% of correct answers to analyze what height difference in the bumps subjects would be able to see. As seen in the psychometric graph, all subjects perform above the threshold (90%), for a height difference between the two bumps of 0.75mm in stereo. The same pool of subjects did not achieve the same threshold value for the height difference of less than 2.5mm. Since the stereo and mono conditions were run using the same basic heights for the bumps, a paired t-test was used to analyze the data. The result of the t-test indicates that

6

K. Abhari et al. / Evaluation of a VR and Stereo-Endoscopic Tool to Facilitate 3rd Ventriculostomy

Table 1. Results: Average Correct Response Rate (CR) vs. Height Difference (HD) Mode [n] [M] [SD] [STDerr]

HD: 0mm, 0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75, 2, and 2.5mm

Stereo [10] [83.8] [23.06] [7.29]

CR: 45%, 45, 65, 90, 95, 98.2, 100, 100, 100, and 100%

Mono [10] [73.6] [14.68] [4.643]

CR: 45%, 60, 60, 70, 75, 79, 85, 85, 87, and 90%

Figure 2. Correct Response Rate vs Height Difference

stereo-endoscopes, compare to the monocular ones, will signiﬁcantly improve the localization and avoidance of basilar artery by increasing our ability to detect small differences in depth. (t=2.93, p=0.01, with the CI of 95%).

Discussion and Conclusion Our data show that if the basilar artery is impinging on the membrane, deforming it by at least 0.75mm, this stereo cue can allow the surgeon to avoid that area with 90% conﬁdence. Other monocular cues may be present, and consequently can be used in addition to these cues which are purely stereoscopic. This paper advocates for the stereoendoscopic extension of the monocular stereoscopic approach, since with minimal incremental cost, this can dramatically improve the performance of locating the basilar artery. The low value of the threshold demonstrates the ability of our subjects in order to differentiate about the location of the basilar artery and the rest of the third ventricle. The results obtained using the VisionSense camera showed that subjects’ performance when making use of monocular cues and stereo cues is subject-dependent. Note that it is impossible to eliminate completely the monocular cues when surfaces are presented to an observer using stereovision. Our data indicate that subjects have the ability to make use of one cue or the other, preferentially, according to personal choice or perceptual capacity. In all cases, however, the subjects were never worse when using Stereo and Mono cues, as compared with Monocular vision alone; and in several cases, their acuity thresholds

K. Abhari et al. / Evaluation of a VR and Stereo-Endoscopic Tool to Facilitate 3rd Ventriculostomy

7

were improved signiﬁcantly for the Stereo and Mono presentation cases. The next phase of this series of experiments will be to determining the accuracy to which the subjects can localize a target (the place where they would make the endoscopic hole), again using the VisionSense camera, and comparing the 2D and 3D cue conditions. We will report on our data collection covering data for the stereoacuity in more subjects, and also acquiring data on proper localization of the target and differences between 2D and 3D images. We also plan to study the feasibility of overlapping the 3D stereoendoscopy with an ultrasonic doppler image of the basilar artery in order to increase the accuracy. A further step would be to be able to map both of those images into a preoperative imaging and use them in order to update the neuronavigation system in real-time. Some teams have tried to use endoscopic ultrasound to increase the accuracy of their operations [8], but were limited by the short penetration depth and the inability to scan anteriorly. Incorporating different technologies (Doppler US, neuronavigation) with stereoendoscopy, should lead to a more accurate way of localizing the target, and therefore to safer operations. In addition, our methodology can then be applied to more complicated neuroendoscopic procedures (ie multiple cysts, tumors etc). Overall, our results show that although there seem to be some inter-subject variability in the stereoacuity, stereoendoscopy facilitates neuroendoscopic performance, especially when the anatomical cues are poor. Acknowledgements The authors would like to thank J. Moore, C. Wedlake, and E. Cheng for valuable discussions and technical support. This project was supported by the Canadian Institutes for Health Research (Grant MOP 74626), the National Science and Engineering Research Council of Canada (Grants R314GA01 and A2680A02), the Ontario Research and Development Challenge Fund, the Canadian Foundation for Innovation and Ontario Innovation Trust. Graduate student funding for K. Abhari was provided by scholarships from the National Science and Engineering Research Council of Canada and by the University of Western Ontario. References [1] [2] [3] [4] [5]

[6] [7] [8] [9]

National Institute of Neurological Disorders and Stroke, http://www.ninds.nih.gov. Chen, J.C., Levy, M.L., Corber, Z., Assi, M.M., Concurrent three dimensional neuroendoscopy: initial descriptions of application to clinical practice, Minim Invasive Neurosurg., 6(4) (1999). Fraser, J.F., Allen, B., Anand, V.K., Schwartz, T.H., Three-dimensional neurostereoendoscopy: subjective and objective comparison to 2D, Neurosurgical focus, 52 (1) (2009) 25-31. Tabaee, A., Anand, V.K., Fraser, J.F., Brown, S., Singh, A., Schwartz, T.H., Three-dimensional endoscopic pituitary surgery, Neurosurgery, 65 (2009) 288-295. Roth, J., Singh, A., Nyquist, G., Fraser, J., Bernardo, A, Anand, V.K., Schwartz, T.H., ThreeDimensional and 2-Dimensional Endoscopic Exposure of Midline Cranial Base Targets Using Expanded Endonasal and Transcranial Approache, Neuro-surgery, 65(6) (2009) 1116-1130 Andrews, T., Glennerster, A., Parker, A.: Stereoacuity Thresholds in the Presence of a Reference Surface, J. of Vision Research, 41 (2001) 3051-3061 Resch, K.D., Transendoscopic ultrasound in ventricular lesions, Surgical neurology, 69(4) (2008) 375382 Julesz B., Foundations of Cyclopean Perception, The University of Chicago Press ISBN 0-226-41527-9. Cornsweet T. N., The Staircase Method in Psychophysics, The American Journal of Psychology,Â˘a75(3) (1962) 485–491

8

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-8

Sleep Dysfunctions Influence Decision Making in Undemented Parkinson’s Disease Patients: A Study in a Virtual Supermarket Giovanni ALBANI 1, Simona RASPELLI 2, Laura CARELLI 3 Lorenzo PRIANO 1, Riccardo PIGNATTI 1, Francesca MORGANTI 3 Andrea GAGGIOLI 2-4, Patrice L. WEISS 5, Rachel KIZONY 5-6, Noomi KATZ 6 Alessandro MAURO 1, Giuseppe RIVA 2-4 1

Department of Neurosciences and Neurorehabilitation, Istituto Auxologico Italiano, IRCCS, Piancavallo-Verbania, Italy 2 Applied Technology for Neuro-Psychology Lab, Istituto Auxologico Italiano, IRCCS, Milan, Italy 3 Department of Human Sciences, University of Bergamo, Bergamo, Italy 4 Psychology Department, Catholic University of Milan, Italy 5 Department of Occupational Therapy, University of Haifa, Haifa, Israel 6 Research Institute for the Health & Medical Professions, Ono Academic College, Kiryat Ono, Israel

Abstract. In the early-middle stages of Parkinson’s disease (PD), polysomnographic studies show early alterations of the structure of the sleep, which may explain frequent symptoms reported by patients, such as daytime drowsiness, loss of attention and concentration, feeling of tiredness. The aim of this study was to verify if there is a correlation between the sleep dysfunction and decision making ability. We used a Virtual Reality version of the Multiple Errand Test (VMET), developed using the NeuroVR free software (http://www.neurovr2.org), to evaluate decision-making ability in 12 PD notdemented patients and 14 controls. Five of our not-demented 12 PD patients showed abnormalities in the polysomnographic recordings associated to significant differences in the VMET performance. Keywords: Virtual Reality, Assessment, Parkinson’s disease, NeuroVR, VMET

1. Introduction In the early-middle stages of Parkinson’s disease (PD), polysomnographic studies show early alterations of the structure of the sleep, which may explain frequent symptoms reported by patients, such as daytime drowsiness, loss of attention and concentration, feeling of tiredness. Apparently these symptoms may involve a deficit in the executive functions, so the goal of this study was to verify the existence of a correlation between the sleep dysfunction and decision making ability in PD not-demented patients.

G. Albani et al. / Sleep Dysfunctions Inﬂuence Decision Making

9

Specifically, polysomnographic data were associated with the performance obtained by the PD patients in the virtual version of a neuropsychological test, the Multiple Errand Test (MET). The MET is an assessment of executive functions in daily life originally developed by Shallice and Burgess [1] specifically for high functioning patients and adapted into the simple version and the hospital version. It consists of three tasks that abide by certain rules and is performed in a mall-like setting or shopping centre.

2. Methods We evaluated 12 PD not-demented patients and 14 controls. In particular, patients who had a severe cognitive impairment (MMSE < 19), a severe motor impairment, auditory language comprehension difficulties (score at the Token Test < 26,5), object recognition impairments (score at the Street Completion Test < 2,25), spatial hemiinattention and neglect, excessive state and trait anxiety (score at the State and Trait Anxiety Index > 40) and excessive depression state (score at the Beck Depression Inventory > 16) were excluded from the study. A neuropsychological evaluation was conducted on the patients selected according to the above criteria, with the aim to obtain an accurate overview of patients’ cognitive functioning. More, the decision making ability was assessed using a virtual version of MET (VMET), which was presented within a virtual supermarket [2-3]. In particular, subjects were invited to buy some items following a defined shopping list and to obtain some information (e.g., the closing time of the supermarket) following specific rules (e.g., you are not allowed to go into the same aisle more than once). While completing the MET procedure, the time of execution, total errors, inefficiencies, rule breaks, strategies, interpretation failures and partial tasks failures (e.g., maintained task objective to completion; maintained sequence of the task; divided attention between components of task and components of other VMET tasks and no evidence of perseveration) were measured. All patients and controls performed a videopolysonnographyc study within a week after the VMET evaluation.

3. Results In normal subjects, neuropsychological tests correlated with the findings of VMET. In PD patients, on the other hand, while traditional neuropsychological test were normal, VMET scores showed significant differences between patients and controls (Table 1). More, five (group A) of our not-demented 12 PD patients of this study showed abnormalities in the videopolysomnographic recordings, such as insomnia, sleep fragmentation and REM behaviour disorders. Concerning VMET analysis, group A in comparison with those patients with normal polysomnographic data (group B), showed significant differences in time of execution (mean p= 0.05) and errors (p = 0.05).

10

G. Albani et al. / Sleep Dysfunctions Inﬂuence Decision Making

4. Conclusions VMET gave us important additional data concerning the cognitive status of PD patients, telling us that also PD not-demented patients may present an underlying unknown cognitive dysfunction. Moreover, this study also suggested a correlation between dysexecutive syndrome and sleep abnormalities in PD: five of our not-demented 12 PD patients showed abnormalities in the polysomnographic recordings associated to significant differences in the VMET performance. Table 1. Differences between groups in the VMET performance

Errors Searched item in the correct area Maintained task objective to completion Maintained sequence of the task Divided attention Organized materials appropriately throughout task Self corrected upon errors made during the task No evidence of perseveration Sustained attention through the sequence of the task Buying a chocolate bar Buying toilet paper Buying a sponge Buying two products from refrigerated products aisle Going to the beverage aisle and asking about what to buy Rule breaks Strategies

Group Healthy subjects Patiets Healthy subjects Patiets Healthy subjects Patients Healthy subjects Patients Healthy subjects Patients Healthy subjects Patients Healthy subjects Patients Healthy subjects Patients Healthy subjects Patients Healthy subjects Patients Healthy subjects Patients Healthy subjects Patients Healthy subjects Patients Healthy subjects

N 14 12 14 12 14 12 14 12 14 12 14 12 14 12 14 12 14 12 14 12 14 12 14 12 14 12

Mean 17,64 25,08 8,86 11,92 8,86 11,83 8,93 12,08 9,29 12,25 9,50 12,25 9,86 12,50 8,50 11,92 9,43 12,17 9,29 13,25 9,07 13,33 9,07 13,33 9,64 12,83

Std. Deviation 3,895 4,757 1,512 2,314 1,351 2,368 1,328 2,234 1,437 2,379 1,990 2,454 1,834 1,931 1,160 2,429 1,342 2,082 2,555 3,888 2,165 3,939 2,556 3,939 2,590 3,326

14

10,50

2,312

Patients Healthy subjects Patients Healthy subjects Patients

12 14 12 14 12

15,17 28,50 24,92 37,36 47,33

1,992 2,378 3,423 8,608 3,339

5. References [1] [2]

[3]

Shallice, T., & Burgess, P. W. (1991). Deficits in strategy application following frontal lobe damage in man. Brain 114, 727-741. S. Raspelli, L. Carelli, F. Morganti, B. Poletti, B. Corra, V. Silani, and G. Riva, Implementation of the multiple errands test in a NeuroVR-supermarket: a possible approach, Studies in Health Technology and Informatics 154, 115-119. G. Albani, S. Raspelli, L. Carelli, F. Morganti, P.L. Weiss, R. Kizony, N. Katz, A. Mauro, and G. Riva, Executive functions in a virtual world: a study in Parkinson's disease, Studies in Health Technology and Informatics 154, 92-96.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-11

11

Visual Tracking of Laparoscopic Instruments in Standard Training Environments Brian F. ALLEN a Florian KASPER a Gabriele NATANELI a Erik DUTSON b Petros FALOUTSOS a a Department of Computer Science, University of California, Los Angeles b Department of Surgery, University of California, Los Angeles Abstract. We propose a method for accurately tracking the spatial motion of standard laparoscopic instruments from video. By exploiting the geometric and photometric invariants common to standard FLS training boxes, the method provides robust and accurate tracking of instruments from video. The proposed method requires no modiﬁcations to the standard FLS training box, camera or instruments. Keywords. Laparoscopic Surgery, Surgery Training, Machine Vision

Introduction Laparoscopic surgery is the most common and widely available minimally invasive surgical technique employed today. With a ﬁber-optic camera and specialized instruments, entire procedures can be accomplished through keyhole incisions. In comparison to open surgery, laparoscopic procedures are less invasive, require shorter periods of hospitalization and entail faster recovery time and less pain for the patient. However, such beneﬁts do not come without costs. In the case of a laparoscopic surgery, perhaps the primary trade-oﬀ is the diﬃculty of the operation and the need for a specialized repertoire of motor skills. To address the diﬃculty of training and evaluating the skill of surgeons, theSociety of American Gastrointestinal and Endoscopic Surgeons (SAGES) adopted the Fundamentals of Laparoscopic Surgery (FLS) as a standardized toolset for certiﬁcation and assessment. FLS is a set of experimentally validated training tasks and equipment [6], providing a standardized means to assess the motor skills speciﬁc to laparoscopy. Such objective measure of skill is particularly important in light of studies that show that training surgeons have little ability to self-assess [5]. FLS assessment gauges manual skills entirely on two features of task performance: movement eﬃciency (measured by the time taken to complete the task) and a precision measure speciﬁc to the task. Precision measures include transient, observed actions, such as dropping a block in the peg transfer task, as well as after-the-fact measures, such as divergence from the target circle in the cutting task, or security of a suture knot. Improvement in the accuracy of assessment has been demonstrated by considering more information than FLS records. In

12

B.F. Allen et al. / Visual Tracking of Laparoscopic Instruments in Standard Training Environments

Camera

FLS Training Box FLS Task

Analog Signal Video Digitizer Frames Instrument Edges 2D Tracking

Estimate Trocar Position

Instrument Tip Position (2D) 3D Tracking Instrument Tip Position (3D)

(a) Overview of the process showing data ﬂow.

(b) A standard FLS box trainer.

Figure 1.

particular, tracking the full spatial motion of the instruments during the course of the task performance provided signiﬁcant gains by considering metrics such as the path length instrument tips travelled [8]. Unfortunately, the equipment needed to acquire detailed spatial tracking data is expensive and specialized. The researchers have predominately employed either (1) precise magnetic tracking [1], (2) mechanical linkages attached to the instruments [7], or (3) virtual reality (VR) simulators with joysticks replacing laparoscopic instruments [11]. Note that (1) and (2) require physical attachments to instruments, while VR simulators typically rely on joysticks that simulate actual laparoscopic instruments. Notably, and most comparable to our work, Tonet et al. [9] considered tracking actual instruments using computer vision. However, that method requires modifying the instruments by aﬃxing a ring of Lambertian material at a know position. In addition, machine vision techniques for laparoscopy have been proposed to control robotic camera holders [10], and for visual-servoing of laparoscopic robots [4]. In this work, we make use of several methods employed by other authors. Voros et al. proposed the use of a probabilistic Hough transform [10] for tracking instruments to automate control of a laparoscope. Doignon et al. [3] describe a least-squares ﬁt of the instrument positions across a series of images to estimate the trocar position. The main contribution of this work is the synthesis of a complete system for tracking tools in FLS training boxes, including the accurate detection of the instrument shafts within the image, the estimation of tool-tip position along the shaft, the automatic registration of the trocar’s position, and the geometric computation of the camera-space position. This method, summarized in ﬁgure 1(a), and is speciﬁcally tailored to tracking laparoscopic instruments in standard FLS trainer boxes. Our goal is purposefully less ambitious than attempts to track instruments in general settings, such as in vivo. We happily exchange generality for reliability and accuracy in this particularly useful setting.

B.F. Allen et al. / Visual Tracking of Laparoscopic Instruments in Standard Training Environments

13

1. Methods and Materials The primary equipment of FLS is a “box trainer,” pictured in ﬁgure 1(b), with several ports and a ﬁxed camera. Our system accepts the video recorded by the camera included in the standard FLS training box. 1.1. Image-Space Position (2D) Distinct photometric features of the FLS toolset allow us to robustly track the 2D position of instrument tool-tips within each frame of video. Our algorithm has three phases: (1) color space analysis and extraction of the instrument contours, (2) line ﬁtting to estimate the direction of each instrument shaft , (3) linear search to identify the most probable position of the tool-tip along each instrument. In the standard FLS setup, both the pegs and instrument shafts have a distinct black color. A simple thresholding operation provides a binary probability map of both the pegs and the shafts (shown in ﬁgure 2(b)), which we then ﬁlter with the application of the erosion and dilation morphological operators. By carefully picking the number of iterations for these operators, we isolate the contours of the two instruments in one step, as shown in ﬁgure 2(c). The number of iterations is determined automatically, as described in Section 1.1.2. Automated tuning greatly improves the robustness of this step. By applying the Hough transform on the isolated instrument maps, we extract the lateral contours of each shaft (shown in ﬁgure 2(d)). Considering that the instruments are always posed diagonally in the frame, we use the inclination of the lateral contours to group them as belonging to the left and right instruments. We ﬁt a line by least-squares that corresponds to the major axis of each instrument, to each group. The forward direction (from the image borders to the center) of each axis deﬁnes a line along which we are going to search for the instrument tool-tips. Figure 2(e) shows the best-ﬁt lines for the example frame. 1.1.1. Searching for the Instrument Tool-Tips with a Conﬁdence Estimate The demarcation point between the instrument and the tool-tip is clearly deﬁned by the abrupt transition between the black color of the shaft and the lighter color of the tool-tip metal body. For added robustness, we search for this point along the direction of each instrument in two functional domains: (1) color space, and (2) gradient space. If we call α the angle between the instrument axis and the the Y-axis, the directional gradient of the image along this angle is given by convolving the image with a rotated forward diﬀerencing kernel: ⎡

⎤ cos(α) − sin(α) cos(α) cos(α) + sin(α) ⎣ ⎦ −sin(α) 0 sin(α) −cos(α) − sin(α) −cos(α) −cos(α) + sin(α) The point TG found in the gradient domain is consistently more accurate than TC found in the color space. Therefore, we always use TG for tracking the position of the tool-tip. On the hand, we use TC to produce an estimate of the conﬁdence we have in TG . We found experimentally that the accuracy of tracking is greatly aﬀected by a shift in the color space characteristics of the instrument region, due to the tool-tips getting out of focus. Hence, by estimating the discrepancy

14

B.F. Allen et al. / Visual Tracking of Laparoscopic Instruments in Standard Training Environments

(a) Unmodiﬁed frame from (b) Binary probability map of (c) Binary mask with isolated the FLS camera during a black regions. instruments. training task.

(d) Extracted lateral contours (e) Instrument direction estiof instruments. mated using line-ﬁtting.

(f) Tracked position in 2D.

Figure 2.

between TC and TG , which are two measurements of the same quantity, we obtain a rather reliable estimate of the conﬁdence of TG . We express this notion as C P (TG ) = 1 − TG −T where β is a normalization constant. β The linear search for TG assumes that there is a single sharp peak in the gradient. However, this assumption is often violated by the presence of specular highlights along the instrument shaft. Noting that such highlight are typically clustered in the region of the shaft closer to the edge of the screen, we mitigate their eﬀect by starting the search from the middle of the instrument axis as opposed to the beginning. 1.1.2. Automated Tuning One parameter that greatly aﬀects the robustness of our approach is the number of iterations for the erosion operator: too many steps remove the instrument regions completely, while too few leave additional noise in the initial binary mask computed in the ﬁrst step of the algorithm. To address this problem, we consider the raw binary probability map of the pegs and instruments, use a heuristic to remove the instrument contours, and determine the minimum number of erosion steps required to remove all the noise. We repeat this approach for a window of frames to ﬁnd the best value for the given video sequence. 1.2. Camera-Space Position (3D) The key idea that allows locating the 3D position of the tool-tip from a single frame of video is recognizing that the vanishing point of the edges of the instrument’s image provides the 3D direction of the instrument d [2]. That is, the vector from the camera (i.e., the origin of the camera frame using the pin-hole model) to the vanishing point is equal to the direction of the instrument itself. Figure 2(g) illustrates this property, with R representing the ﬁxed point through which the instrument passes. Likewise, the diagram illustrates that the 3D position of the

B.F. Allen et al. / Visual Tracking of Laparoscopic Instruments in Standard Training Environments

15

(g) The geometry of the image formation of (h) Calculation of the depth λ of the trothe instrument. car. The shown plane contains the major axis of the projected ellipse of the trocar (x0 x1 ) and the camera.

Figure 2. Continued.

tool-tip is the intersection of two lines: the line that passes through the trocar R in the direction of the instrument d , and the pre-image of the tool-tip (i.e., the line passing through both the camera point and the image of the tool-tip). This approach assumes that the camera-space position of the trocar R is known. Unfortunately, it is not possible to locate R from a single frame. 1.2.1. Edges’ Vanishing Point and Direction of the Instrument Once the images of the framing edges of the instrument (eu , el ) are found, the vanishing point is V = eu × el , assuming lines eu , el and point V are in 2D homogeneous coordinates. Thus, all lines in the scene that are parallel to the direction of the instrument d will have images that pass through V . Now consider the line that passes through the camera point C and is parallel to d: C + td. The image of this line must also pass through V , as V is the vanishing point for the ¯ is equivalent to C + td. Since the world frame direction d . Therefore, the line CV is simply the camera frame, the direction of the instrument is simply d = V||V−0 || . 1.2.2. Position of Tool-Tip The tool-tip point T is the point on the instrument that corresponds to the distal end of the instrument (see ﬁgure 2(g)). The tool-tip is some unknown distance k from R in the direction of the tool, T = R + sd. But note that T is also located on the pre-image of point T , i.e., on the line C + t(T − C) = t(T − 0) = tb with b ≡ (T − 0). The procedure for locating T in the image will be considered in the next section. Ideally, T is simply at the intersection of lines L1 (s) = R + sd and L2 (t) = tb, however such precision is unlikely. Instead, consider the points on each line s) and L2 (t˜). The segment L1 (˜ s)L2 (t˜) is uniquely closest to the other line, L1 (˜ perpendicular to both L1 (s) and L2 (t).

16

B.F. Allen et al. / Visual Tracking of Laparoscopic Instruments in Standard Training Environments

d, bb, (R − 0) − b, bd, (R − 0) d, db, b − (d, b)2

(1)

d, db, (R − 0) − d, bd, (R − 0) t˜ = d, db, b − (d, b)2

(2)

s˜ =

s)L2 (t˜) as the estimate of T gives Taking the midpoint of L1 (˜

T =

(R + s˜d) + (t˜b) . 2

(3)

1.2.3. Locating the Trocar from a Sequence of Images So far we have assumed that the position of the trocar R (the center of the region of space through which all instruments pass) is known. To determine R , the framing edge pairs (eiu , eil ) for each frame i are collected. If there were no errors in the (eiu , eil ), the image of the trocar would be the locus of points on the image plane between the edges for all frames. Due to inevitable noise, the actual image of the trocar is smaller than the observed locus. To more robustly ﬁnd the trocar’s image, an image point R is found as the point closest to all of the (eiu , eil ), that is, for E = {eiu , eil }, ∀i and v i ⊥ li for all l ∈ E, R = arg max p∈I

projv (l0i − p) .

(4)

i

With the center of the image of the trocar R determined, the ellipse centered at R with one axis of (R − 0) that best matches the set of E is found. Deﬁne x0 ≡ ||R − [w/2, h/2]T || and x1 ≡ x0 + m, where w, h are the width and height of the image, and 2m is the length of the major axis of the ellipse. The geometry of the trocar’s projection is shown in ﬁgure 1(a), in the plane containing the ellipse’s major axis and the camera. Deﬁning a, b, r , d as in ﬁgure 1(a), the depth of R, λ is determined by r = d tan(b) x0 d = sin(a)

b = tan−1 (x1 ) − tan−1 (x0 ) r λ = d . r

With both λ and the image of the trocar R , the 3D position of the trocar is known. 2. Results For our experiments, we captured several video sequences of the FLS peg transfer task with the standard camera included in the box trainer and a completely unaltered setup. The illumination is provided by an array of LED lights included in the box. One group of tasks was performed by an expert surgeon featuring controlled smooth motions, while a second group was performed by a novice and

B.F. Allen et al. / Visual Tracking of Laparoscopic Instruments in Standard Training Environments

17

is aﬀected by jittery non-smooth motions. In both cases, we recorded robust 2D tracking of the instrument tips that were validated visually. Figure 2(f) shows the tracked position (in yellow) of the two instrument tips from the unmodiﬁed FLS video, shown in ﬁgure 2(a). The accompanying video shows the performance of our tracker for a short clip with thumbnails of the intermediate steps. The measure of conﬁdence of the tracked position allows us to automatically disable tracking of an instrument tip when it is no longer visible in the scene. The tracker is unable to track the position of the instrument tip accurately when the instrument is too close to the camera and thus very blurry. However, in such cases, the measure of conﬁdence is very low, as expected. 3. Conclusion In this paper we presented a complete system for tracking the 3D position of the instrument tips of a standard FLS box trainer. Our approach is robust, does not require any physical alteration of the toolset, and works with the standard camera included in the kit. In the future, we would like to combine our existing tracking capabilities with a more thorough analysis of the entire scene as a means to produce a more accurate assessment of FLS tasks. References [1]

[2] [3]

[4]

[5]

[6]

[7]

[8] [9]

[10]

[11]

B. Allen, V. Nistor, E. Dutson, G. Carman, C. Lewis, and P. Faloutsos. Support vector machines improve the accuracy of evaluation for the performance of laparoscopic training tasks. Surgical endoscopy, 24(1):170–178, 2010. A. Cano, P. Lamata, F. Gay´ a, and E. G´ omez. New Methods for Video-Based Tracking of Laparoscopic Tools. Biomedical Simulation, pages 142–149, 2006. C. Doignon, F. Nageotte, and M. de Mathelin. The role of insertion points in the detection and positioning of instruments in laparoscopy for robotic tasks. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2006, pages 527–534, 2006. A. Krupa, C. Doignon, J. Gangloﬀ, and M. de Mathelin. Combined image-based and depth visual servoing applied to robotized laparoscopic surgery. In Proc. of the 2002 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2002. VA Pandey, JHN Wolfe, SA Black, M. Cairols, CD Liapis, and D. Bergqvist. Selfassessment of technical skill in surgery: the need for expert feedback. Annals of The Royal College of Surgeons of England, 90(4):286, 2008. J. Peters, G.M. Fried, L.L. Swanstrom, N.J. Soper, L.F. Sillin, B. Schirmer, K. Hoﬀman, et al. Development and validation of a comprehensive program of education and assessment of the basic fundamentals of laparoscopic surgery. Surgery, 135(1):21–27, 2004. J. Rosen, J.D. Brown, L. Chang, M. Barreca, M. Sinanan, and B. Hannaford. The Blue Dragon-a system for measuring the kinematics and the dynamics of minimally invasive surgical tools in-vivo. In Proceedings- IEEE International Conference on Robotics and Automation, volume 2, pages 1876–1881. Citeseer, 2002. C.D. Smith, T.M. Farrell, S.S. McNatt, and R.E. Metreveli. Assessing laparoscopic manipulative skills. The American Journal of Surgery, 181(6):547–550, 2001. O. Tonet, R.U. Thoranaghatte, G. Megali, and P. Dario. Tracking endoscopic instruments without a localizer: A shape-analysis-based approach. Computer Aided Surgery, 12(1):35– 42, 2007. S. Voros, J.A. Long, and P. Cinquin. Automatic detection of instruments in laparoscopic images: A ﬁrst step towards high-level command of robotic endoscopic holders. The International Journal of Robotics Research, 26(11-12):1173, 2007. JD Westwood, HM Hoﬀman, D. Stredney, and SJ Weghorst. Validation of virtual reality to teach and assess psychomotor skills in laparoscopic surgery: Results from randomised controlled studies using the MIST VR laparoscopic simulator. Medicine Meets Virtual Reality: art, science, technology: healthcare and evolution, page 124, 1998.

18

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-18

On the Use of Laser Scans to Validate Reverse Engineering of Bony Anatomy Joseph B ANSTEYa, Erin J SMITHb, Brian RASQUINHAb, John F RUDANc, and Randy E ELLISa,b,c,1 a School of Computing, Queen’s University, Kingston, Ontario, Canada K7L3N6 b Department of Mechanical and Materials Engineering, Queen’s University c Department of Surgery,Queen’s University

Abstract. There is a growing body of evidence to suggest the arthritic hip is an irregularly-shaped, aspherical joint, especially in severely pathological cases. Current methods used to study the shape and motion of the hip in-vivo, are invasive and impractical. This study aimed to assess whether a plastic model of the hip joint can be accurately made from a pelvic CT scan. A cadaver hemi-pelvis was CT imaged and segmented from which a 3D plastic model of the proximal femur and hemi-pelvis were fabricated using rapid-prototyping. Both the plastic model and the cadaver were then imaged using a high-resolution laser scanner. A three-way shape analysis was performed to compare the goodness-of-fit between the cadaver, image segmentation, and the plastic model. Overall, we obtained submillimeter fit accuracy between all three hip representations. Shape fit was least favorable in areas where the boundary between cartilage and bone is difficult to distinguish. We submit that rapid-prototyping is an accurate and efficient mechanism for obtaining 3D specimens as a means to further study the irregular geometry of the hip. Keywords. Hip, Anatomy, Arthritis, Computed Tomography, Stereolithography

Introduction Detailed physical study of population variations in anatomy, including bones, is limited by the availability of specimens but computed tomography (CT) scans of patients are more abundant. This raises the question: how accurate is the reverse engineering of anatomy from a medical image? To take the hip joint as an example, the currently accepted belief is that the hip is a ball-and-socket joint with spherical congruent joint surfaces of the femoral head and acetabulum [1]. However, there is an emerging body of evidence to suggest the contrary – that the arthritic hip is, in fact, aspherical in nature [2]. This is especially true in pathologies such as femoroacetabular impingement, osteoarthritis, and developmental hip dysplasia. Thus it is important to accurately understand the shape and movement of this irregular joint in order to devise appropriate treatments for disease.

1

Corresponding Author: Randy E Ellis, School of Computing, Queen’s University, Kingston, ON, Canada K7L 3N6; E-mail: [email protected]

J.B. Anstey et al. / On the Use of Laser Scans to Validate Reverse Engineering of Bony Anatomy

19

Because of the unique shape and anatomical location of the pelvis, it is difficult to study the motion of this joint in vivo using optoelectrically tracked skin markers. Recent efforts have been made to study the motion of the hip using computer navigation [3]; however, these methods are invasive and thus not practical for largescale in vivo studies. Moreover, cadaver specimens with specific pathologies are difficult to source, expensive, and if un-embalmed (e.g., frozen) short-lived. By comparison, medical images of pathological hips, such as CT scans, are readily available as they are often standard practice in pre-operative care. The purpose of the current investigation was to assess the accuracy of replicating the shape of the hip joint using 3D rapid prototyping. If this process is sufficiently accurate, then plastic models derived from patient CT images could potentially be used as a means to study the kinematics of an irregularly shaped or pathological hip joint.

1. Methods & Materials A formaldehyde-fixed hemi-pelvis was imaged with all soft tissues intact using a 16slice CT scanner (Lightspeed+ XCR , General Electric, Milwaukee, USA) with a slice thickness of 0.625mm. The images were saved and later imported into commercially available Mimics software (Materialise, Leuven, Belgium). The anatomy was systematically segmented into 3D digital models using a step-wise process that ensured the production of precise representations of the imaged anatomy. The process began by applying a threshold to highlight the surface of the bony anatomy with a mask. This mask was then manually edited until satisfactory segmentation of the hip in all three orthogonal planes (coronal, sagittal, and axial) was achieved. The masks for the hip bone and proximal femur were rendered into digital 3D models. The models were then visually examined for unusual bumps or pits that may be considered atypical of the anatomy. If an unusual surface feature was observed, the area was compared to the raw CT images of that location. If the unusual feature was found to accurately represent the CT data no action was taken, otherwise the area was edited to accurately reflect the images in the CT data. When satisfied with the outcome of the digital 3D models, they were saved as a Stereo-Lithography (.STL) file and sent to a rapid-prototyping machine (Dimension sst 1200es, Stratasys, Eden Prairie, USA) for fabrication. Upon printing, the model was again visually examined for any unusual surface features not seen in the CT data and their articulation with one another was examined to ensure that they did indeed articulate (since we knew the cadaver anatomy articulated, it was important to ensure that the modeled anatomy also articulated). To ensure that the articulations were typical of a human joint, a senior orthopedic surgeon (JFR) was consulted to evaluate the articulation. 1.1. Cadaver Preparation The bones comprising the cadaver hip were retrieved by removing all soft tissues using typical dissection techniques with a scalpel, forceps, and a blunt probe. The labrum and fat pad were also removed from the acetabulum, and attention was given to the fovea on the head of the femur to remove the remnants of the ligamentum teres. The bones were scraped clean using a scalpel and a 5% solution of hydrogen peroxide to loosen tissue from the non-articulating bone surfaces. Our goal was to compare the 3D

20

J.B. Anstey et al. / On the Use of Laser Scans to Validate Reverse Engineering of Bony Anatomy

models to the actual anatomy with the articular cartilage intact so extra care was taken to not damage the cartilage. 1.1.1. Shape Analysis The head and acetabular components of both the cadaver and plastic models were scanned using a laser scanner (Central V.5.7.0, ShapeGrabber, Ottawa, Canada) to obtain point-cloud representation of their surfaces. Because of the complex 3D geometry of the components, the scans were acquired in small patches that were within the plane of the laser; the specimens were rotated between each scan to collect data from the entire surface. A three-way analysis (Figure 1) was performed to determine the goodness-of-fit between: i) the cadaver and derived CT segmentation, ii) the CT segmentation and subsequent plastic model, and iii) the cadaver and the plastic model. The laser-scanned point cloud data was used to generate a STL tessellation for each surface patch. These were imported into the Mimics environment along with the 3D segmentation model. Mimics was used to perform a preliminary registration of each surface patch to the 3D segmentation model. This was accomplished both manually (visually) as well as with the local and global Mimics registration tools. The registered surface patches, now in a common global coordinate frame of the 3D segmentation model, were exported as new STL files. These files were imported into MATLAB (MathWorks, Natick, MA) for all subsequent data analysis.

Figure 1: 3-way shape analysis

In the MATLAB environment, a refined registration was performed using an iterative closest point (ICP) algorithm [4] to register each patch to the segmentation model. Subsequently, the 3D segmentation model was triangulated using a Delaunay triangulation [5], and the closest point to each triangle, as well as its corresponding distance (residual), was located on each patch using an iterative closest point algorithm. For the set of point matches, the root-mean-square error (standard deviation, σ) of the residual was computed and used to filter the extreme ~5% outliers (1.96σ). Subsequently, a second refined registration was performed for each patch and new statistical measures computed: residual distance at each point, average and maximum deviations, and root-mean square errors.

J.B. Anstey et al. / On the Use of Laser Scans to Validate Reverse Engineering of Bony Anatomy

21

2. Results A summary of match results for the proximal femur and acetabulum are shown in Tables 1 and 2, respectively. Signed distances were computed to determine whether the matches were inside or outside the target, with positive numbers being outside the target (larger) and negative numbers being inside (smaller). Overall, we obtained sub-millimeter shape accuracy between the shape of the cadaver hip region and both the resulting CT segmentation and 3D plastic model. In both cases, the cadaver was slightly smaller than the CT segmentation. Similarly, the model was slightly smaller than the CT segmentation from which it was derived. As was expected from these findings, the cadaver and model were a close match, with the cadaver being slightly smaller than the model. The residual distances computed from the matched object to the target object were plotted in order to visualize areas of good and poor fit. Figures 2 and 3 depict matches outside ± 1σ for the three-way match. By comparing cadaver specimens to these residual plots, it was noted that mismatches tended to occur in specific regions. These included areas where there was residual soft tissue on the cadaver specimen that was detected by laser scanning, but not in the CT segmentation or consequently the model (these are positive distances, indicating that they are external). Both osteophytic regions on the femur and along the acetabular rim also showed greater mismatch, likely because osteophytes are difficult to segment due to their semi-cartilaginous compositions. For the same reasons, areas of cartilage deterioration on the surface of the femoral head also showed a higher degree of mismatch.

3. Discussion There were potential sources of error in our data collection and analytical process, which were consequences of the time it took to acquire data in the study. The cadaver was first imaged with CT, then dissected, and laser-scanned at a later date. After CT imaging, the cadaver specimen was wrapped in a cloth soaked with a moistening solution (water, glycerin, Potassium Acetate, and Dettol) and stored in a heavy duty plastic bag at room temperature. Post-dissection, the remaining bones were stored using the same method until the completion of our study. It is unknown how this storage process may have affected the size and shape of the bone. Because the bones had to be exposed to the room’s environmental conditions during the laser scanning for extended periods of time (up to 2.5hrs) on multiple occasions, there may have been changes due to dehydration, especially of the articular cartilage. In particular, we noticed that dehydration of the specimen over time led to tiny “hairs” of periosteal tissue to appear on surface of the cadaver specimens. These “hairs” may have affected the quality of registration, and hence the quality of analysis of cadaver-based comparisons. This is further supported by the better matches observed between the smooth plastic model and the CT segmentation.

22

J.B. Anstey et al. / On the Use of Laser Scans to Validate Reverse Engineering of Bony Anatomy

Figure 2: Residual distance maps for proximal femur matches. Poorly-matched areas outside of one standard deviation are shown as black (smaller than the match target) or white (larger than the match target). Areas within one standard deviation are uncolored (gray).

Table 1: Results of proximal femur matches. Root-mean-square error, average and maximum deviation were computed for the residual distance at each point. Signed distances were computed to determine whether the matches were inside or outside the target, with positive numbers being outside the target (larger) and negative numbers being inside (smaller).

RMSE (σ) Average Deviation (unsigned) Average Deviation (signed) Max Deviation (unsigned)

CADAVER-toSEGMENTATION 0.61 mm

MODEL-toSEGMENTATION 0.49 mm

CADAVER-toMODEL 0.48 mm

0.58 mm

0.47 mm

0.42 mm

-0.49 mm

- 0.46 mm

-0.32 mm

1.62 mm

0.94 mm

1. 58 mm

Our results also suggest that there may have been some over-segmentation of the CT scans, mainly in regions containing osteophytes (such as the femoral head-neck junction and the acetabular rim) and along the articular surface, especially in areas of cartilage deterioration. In these regions it was particularly difficult to distinguish a definitive boundary between bone and cartilage on the CT images, even with careful segmentation. Over-segmentation would cause the segmentation and resulting model to be slightly larger than the cadaver, which is implied in our results. We also noted that the plastic model was slightly smaller than the CT segmentation from which it was derived. However, mismatches appeared to be much more uniform over the entire surface, rather than in specific concentrations as we saw with the cadaver-to-CT match. We also observed a tendency for mismatches to follow the directions of material deposition. There are several potential explanations for these observations including the resolution of the 3D printer (approximately ±0.1mm), anisometric plastic hardening following deposition, and thermal fluctuations at the time of laser-scanning that may have an effect on the volume of the plastic model.

J.B. Anstey et al. / On the Use of Laser Scans to Validate Reverse Engineering of Bony Anatomy

23

Figure 3: Residual distance maps for acetabulum matches. Poorly-matched areas outside of one standard deviation are shown as black (smaller than the match target) or white (larger than the match target). Areas within one standard deviation are uncolored (gray).

Table 2: Results of acetabulum matches. Root-mean-square error, average and maximum deviation were computed for the residual distance at each point. Signed distances were computed to determine whether the matches were inside or outside the target, with positive numbers being outside the target (larger) and negative numbers being inside (smaller).

RMSE (σ) Average Deviation (unsigned) Average Deviation (signed) Max Deviation (unsigned)

CADAVER-toSEGMENTATION 0.81 mm

MODEL-toSEGMENTATION 0.58 mm

CADAVER-toMODEL 0.54 mm

0.72 mm

0.55 mm

0.47 mm

-0.58 mm

-0.55 mm

-0.43mm

2.86 mm

1.91 mm

1.94 mm

Additionally, both the CT imaging and laser scanning processes have inherent inaccuracies that may have been propagated through the analytical pipeline. Although we obtained high-quality CT images, our segmentation remained limited to the resolution of the CT scans (0.625mm). Moreover, image quality may have been reduced or distorted through the image processing pipeline, as images were changed from one format to another. For instance, CT slice pixels were combined to form 3D voxels which were then triangulated (without smoothing) to form a surface for subsequent analysis. This study was limited to a single specimen as a proof of the concept of using a rapid prototyping process to reconstruct bony anatomy. Future work could include expanding the number of specimens, using fresh-frozen cadaveric material (or immediately post-mortem animal tissue), and comparing various pathologies to determine whether diseased bone can be accurately reconstructed. On the basis of this work, we are encouraged at the prospect for the use of rapid prototyping as a novel tool in the anatomical sciences. For example, this representation was used to analyze the morphology of the joint by fitting an ellipsoid to the articular surfaces (as in [2]) which quantitatively demonstrated asphericity of the femoral head. If a larger sample size is found to support our current findings we may also begin replicating patient hip joints with no known pathologies and determine whether those hip joints are also aspherical.

24

J.B. Anstey et al. / On the Use of Laser Scans to Validate Reverse Engineering of Bony Anatomy

The results of which have the potential of changing the way the geometry of the hip joint is viewed in mechanical and scientific disciplines.

4. Conclusion Three-dimensional rapid prototyping derived from high-quality CT image segmentations can accurately represent the true shape of the hip joint with submillimeter accuracy. The outcome, however, is clearly dependent on the accuracy of the image segmentation from which the model is derived. Therefore, care must be taken to accurately define the cartilage boundary especially along articular surfaces and in osteophytic regions. Although we can claim that plastic models can accurately depict the shape of the hip joint, more work is needed to draw conclusions concerning use of these models to accurately represent the motion of this joint.

Acknowledgements This work was supported in part by the Canada Foundation for Innovation, the Canadian Institutes for Health Research, Kingston General Hospital, and the Natural Sciences and Engineering Research Council of Canada.

References [1]

Cailliet, R.: The Illustrated Guide to Functional Anatomy of the Musculoskeletal System. American Medical Association, 2004.

[2]

Ellis, R., Rasquinha, B., Wood, G., Rudan, J.: 3D Shape Analysis of Arthritic Hips: A Preliminary Study. Int J Comp Assist Radiol Surg, S137–S142, 2010.

[3]

Thornberry, R. L.: The Combined Use of Simulation and Navigation to Demonstrate Hip Kinematics. J Bone Joint Surg(Am) 91:144-152, 2009.

[4]

Besl, P., McKay, N.: A Method for Registraion of 3-D Shapes. IEEE Trans Pattern Anal Machine Intell 4(2),:239-256, 1992.

[5]

Barber, C., Dobkin, D., Huhdanpaa, H.: The Quick-hull algorithm for convex hulls. ACM Trans Math Software 22(4):469-483, 1996.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-25

25

Classification of Pulmonary System Diseases Patterns Using Flow-Volume Curve a

Hossein ARABALIBEIKa,1, Samaneh JAFARIa and Khosro AGIN b Research Center for Science and Technology in Medicine (RCSTIM), Tehran University of Medical Sciences, Tehran, Iran b Shahid Beheshti University of Medical Sciences, Tehran, Iran

Abstract. Spirometry is the most common pulmonary function test. It provides useful information for early detection of respiratory system abnormalities. While decision support systems use normally calculated parameters such as FEV1, FVC, and FEV1% to diagnose the pattern of respiratory system diseases, expert physicians pay close attention to the pattern of the flow-volume curve as well. Fisher discriminant analysis shows that coefficients of a simple polynomial function fitted to the curve, can capture the information about the disease patterns much better than the familiar single point parameters. A neural network then can classify the abnormality pattern as restrictive, obstructive, mixed, or normal. Using the data from 205 adult volunteers, total accuracy, sensitivity and specificity for four categories are 97.6%, 97.5% and 98.8% respectively. Keywords. pulmonary function test, respirometry, flow-volume curve, artificial neural networks

Introduction Early detection of respiratory system abnormalities raises the chances of successful treatments and drops related costs. Pulmonary function tests (PFTs) measure the efficiency of lungs function. Spirometry is the most widely used PFT. It records the amount of air breathed in and out and the rate at which this process takes place [1]. The preliminary output of the spirometry test is the flow-volume curve. This curve is constructed by calculating the flow and volume of the inhaled and exhaled air during an inspiration and expiration cycle performed with maximum effort (Figure 1a). Normally, Vital Capacity (VC), Forced Vital Capacity (FVC), Forced Expiratory Volume at 1st second (FEV1), ratio of FEV1 to FVC (FEV1%), Peak Expiratory Flow (PEF) and Forced Expiratory Flow at 25 to 75% (FEF 25-75) are extracted from this curve and used as a basis for diagnosis. Age, height, sex and ethnic of the patient influence expected normal values of the measured parameters which in turn affect interpretation of the spirometry results [2].

1

Corresponding Author: Research Center for Science and Technology in Medicine (RCSTIM), Imam Khomeini Hospital, Keshavarz Blvd, Tehran, Iran, Tel: +98 21 66581505, Fax: +98 21 66581533, E-mail: [email protected].

26

H. Arabalibeik et al. / Classiﬁcation of Pulmonary System Diseases Patterns

Various respiratory diseases generate different flow-volume curve patterns. Restrictive lung diseases (e.g. change in lung parenchyma, disease of the pleura, chest wall or neuromuscular apparatus) are identified by reduced lung volume leading to a shrunk version of the normal flow-volume curve [3]. This pattern is characterized by low FVC and comparatively high expiratory flow (Figure 1b). The obstructive pattern is characterized by a decreased flow and FEV1, usually along with normal or increased volume (Figure 1c). This pattern is a consequence of progressive airflow obstruction in the peripheral airways, associated with lung inflammation, emphysema and mucus hyper secretion [4]. Examples of obstructive airway diseases are asthma, chronic bronchitis, chronic obstructive pulmonary disease (COPD) and emphysema.

(a)

(b)

(c)

(d)

Figure 1. Flow–volume curve of (a) Normal, (b) Restrictive, (c) Obstructive and (d) Mixed subjects

H. Arabalibeik et al. / Classiﬁcation of Pulmonary System Diseases Patterns

27

In mixed pattern, respiratory system suffers from both obstructive and restrictive abnormalities. Normally volume reduces more than flow (Figure 1d). So this pattern is characterized by reduced FEV1 and FVC values and increased FEV1%. Automated diagnosis systems generally use the extracted parameters from the curve. Some recent publications have suggested the use of different intelligent systems as decision support systems to help the physicians in diagnosis [2-7]. All of these methods just use the mentioned parameters, while expert physicians use the morphology and pattern of the flow volume curve as well. Are these parameters sufficient to capture the precious data stored behind the obtained curve? In this research we will show that some simple and computationally inexpensive parameters can better capture the pattern of the curves and contribute more in diagnosing the diseases. In this work MLP neural networks are used as classifier to discriminate between four patterns of pulmonary system operation namely normal, obstructive, restrictive and mixed.

1. Methods and Materials Flow-volume data of 205 adult volunteers consisting of 90 normal, 30 restrictive, 32 obstructive and 53 mixed pattern cases are obtained using a Spirojet spirometer (Ganshorn Company: www.ganshorn.de). The data is then divided to 155 training and 50 test samples. Predicted values of FVC, FEV1, FEV1% and PEF are obtained using age, gender, height and race of patients. The standard protocol of a breath cycle in spirometry according to the recommendation of the American Thoracic Society (ATS) consists of an inhaling to total lung capacity and then exhaling as hard and completely as possible. Diseases such as goiter change the inspiration part of the flow-volume curve, while the expiration part is affected by obstructive, restrictive and mixed abnormality patterns. Curve fitting is a parametric model estimation algorithm. According to a cost function, the algorithm tries to find the optimal values of the predefined smooth function coefficients. The cost function is a measure of the error between the real data and their approximation by the fitted curve. Polynomial models given by (1)

are used in this study to extract some simple features regarding the curve patterns, where n is the order of the polynomial Artificial Neural Networks (ANNs) are computational models consisting of simple processing units connected in a layered structure. They provide promising tools for complex systems modeling, function approximation and decision making in nonlinear multi-criteria domains by learning from examples. A Multilayer Perceptron (MLP) neural network stores the extracted knowledge in layer weights. Learning takes place by adapting weights to minimize the output error between the network’s output and the desired values. Various MLP networks, with different hidden layers and diverse number of neurons in each hidden layer are used to classify four respiratory diseases patterns.

28

H. Arabalibeik et al. / Classiﬁcation of Pulmonary System Diseases Patterns

Figure 2. A sample of flow-volume curve. Dashed line: original curve; solid line: fitted curve

2. Results Polynomials of orders 5 and 6 are used which leads to R-squared values of more than 0.99. This shows a good fitting of the curves to the measured data which preserve most of the details needed for the diagnosis (Figure 2). MLPs with 1 and 2 hidden layers and diverse number of neurons in each hidden layer are used as classifier. Tangent-sigmoid transfer function is used for hidden layers and linear transfer function for the 4 neurons of the output layer. Each output neuron, specify one of the patterns. The coefficients (pi) of fitted curves as well as the predicted values of FVC, FEV1 and FEV1% were used as inputs of the neural network. The network is trained for a mean squared error of less than 10e-5, using Levenberg– Marquardt (LM) algorithm during 300 epochs. We use early stopping to avoid decrease in the generalization ability of the network caused by over fitting. To compare the discrimination power of each extracted feature, we used Fisher Discriminant Ratio (FDR) which considers both within class and between class scatterings [8]. Figure 3 shows that polynomial coefficients have considerably higher FDR values than parameters usually used for classification. Accuracy, sensitivity and specificity results for different networks are presented in Table 1. For comparison purposes, the best results of using FEV1, FEV1%, FVC, and their corresponding predicted values as decision parameters are also presented (ANN15).

Figure 3. Fisher Discriminant Ratio for different features

H. Arabalibeik et al. / Classiﬁcation of Pulmonary System Diseases Patterns

29

Table 1. Comparison of different MLP structures No 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Architecture 10-7-4 10-20-4 10-25-4 10-30-4 10-40-4 10-45-4 10-13-10-4 10-13-18-4 10-13-20-4 10-13-30-4 10-13-40-4 10-13-50-4 9-20-4 9-30-4 6-10-10-4

Accuracy 92.5 97.56 95.12 95 96.25 92.7 95.12 95 95.12 95 95.12 90.25 95.12 92.69 87.8

Sensitivity 92.5 97.5 95 95.12 95.12 92.5 92.5 95.12 95 95.12 95 90 95 92.5 82.5

Specificity 98 98.8 98.1 97.61 97.48 97.4 98 99 99 99 98 97 98.1 96.93 95.97

Table 2. Comparison of the results No 1 2 3 4 5

Classifier [2] [5] [6] [7] This work

Accuracy 90 92 92.3 93 97.6

Sensitivity 91.6 92.3 92.6 93 97.5

Specificity 87.5 91.6 91 98 98.8

3. Discussion The 10-20-4 configuration has the best result in classifying respiratory patterns within three layer networks. Most of the networks in the four layer structure present very close results. According to Table 1, ANN2 with a 10-20-4 structure also presents the best results in general. Excessive number of neurons in the hidden layer does not raise the classification performance. In fact, unnecessary modeling power of the network causes over fitting which in turn initiates early stopping. The results of ANN7 to ANN12 show that assuming one more hidden layer results in the same problem. Comparing ANN2 and ANN13, as well as ANN4 and ANN14, show that the polynomial of order 6 outperforms the order 5 polynomial. Although higher polynomial orders preserve more details of the flow-volume curve, but simulations show that polynomials of orders greater than 6 provide unnecessary inputs to the ANN. This makes the MLP more complex without increasing its classification performance. At the other hand, lower order polynomials do not capture the considered necessary details of the curves for appropriate diagnosis and classification. In another word, using the proper order of the polynomial preserves necessary information for the classification and filters out the unnecessary details and noises that not only do not contribute to diagnosing ability but also weaken it. Comparison of accuracy, sensitivity and specificity results of this study and previous works (Table 2) shows that using a set of simple computational features that capture the morphology of the flow-volume curve, results in an improved classification of respiratory disease patterns.

30

H. Arabalibeik et al. / Classiﬁcation of Pulmonary System Diseases Patterns

ANN15 shows that using a simple MLP neural network of the comparable size with normally used parameters of FEV1, FEV1%, FVC, and their predicted values does not tend to good results. It means that better performance of ANN1 to ANN14 could be attributed to the selected features.

4. Conclusions Spirometry is a common and helpful test in evaluating the functionality of pulmonary system. Normally some parameters like FEV1 and FEV1% which are extracted from the flow-volume curve are used for classification of respiratory system disease patterns. These parameters have rather single point characteristics. They do not represent the shape of the curve sufficiently. The results of this research show that the curve contains more precious information than just what these parameters capture. Using some simple parameters such as fitted curve coefficients, one can extract the information behind spirometry output curve more precisely as an expert physician does.

References [1] [2] [3] [4] [5]

[6] [7]

[8]

http://www.thoracic.org (last accessed: 2010/01/01). M. Veezhinathan and S. Ramakrishnan, Detection of obstructive respiratory abnormality using flow– volume spirometry and radial basis function neural networks, J. Med. Syst. 31 (2007), 461–465. C.R. Sweeney, Equine restrictive lung disease Part 1: Overview, in P. Lekeux (Ed.), Equine Respiratory Diseases, International Veterinary Information Service, Ithaca, New York, USA, 2004. A. Husain and S. Habib, Pattern identification of obstructive and restrictive ventilatory, Pak. J. Physiol. 4 (2008), 30–34. V. Mahesh and S. Ramakrishnan, Assessment and classification of normal and restrictive respiratory conditions through pulmonary function test and neural network, J. Med. Eng. Techno. 31 (2007), 300– 304. M.J. Baemani, A. Monadjemi and P. Moallem, Detection of respiratory abnormalities using artificial neural networks, Journal of Computer Science 4 (2008), 663–667. H. Arabalibeik, M.H. Khomami, K. Agin and S. Setayeshi, Classification of restrictive and obstructive pulmonary diseases using spirometry data, In Studies in Health Technology and Informatics 142, IOS press, 2009. G.J. McLachlan, Discriminant analysis and statistical pattern recognition, John Wiley & Sons, New York, 1992.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-31

31

Cost-Efficient Suturing Simulation with Pre-Computed Models Venkata Sreekanth ARIKATLAa , Ganesh SANKARANARAYANANa and Suvranu DE a,1 a Rensselaer Polytechnic Institute, Troy, NY

Abstract. Suturing is currently one of the most common procedures in minimally invasive surgery (MIS). We present a suturing simulation paradigm with precomputed finite element models which include detailed needle-tissue and threadtissue interaction. The interaction forces are derived through a reanalysis technique for haptic feedback. Besides providing deformation updates and high fidelity forces, our simulation is computationally less costly. Keywords. Surgery simulation, Suturing, Real-time simulation, Finite elements

Introduction More often than not, surgery simulation involves intricate procedures being performed over complex geometries. The main cost in most physics-based surgery simulation environments is the deformation update. For this reason, pre-computed methodologies [1] are sometimes preferred over iterative or direct-solution procedures. Precomputation based methodologies aid in dramatic cost reduction during run-time. Nevertheless, some limitations include the restriction mostly to linear formulation and no topology changes being allowed. Suturing is now-a-days one of the most common surgical procedures in MIS (Minimally Invasive Surgery). In this paper, we model the suturing procedure using pre-computed methods to simulate the deformation and interaction forces. Unlike in [2], we aim for detailed needle-tissue and thread-tissue interaction. We specifically use the reanalysis technique [3] in conjunction with the superposition principle for linear elasticity to update the deformation and the reaction forces as a result of needle-tissue and thread-tissue interactions. This culminates in high fidelity tissue response while utilizing fewer computational resources.

1. Tools and Methods The suturing procedure in MIS requires bimanual interaction with the needle and the thread in order to suture on the tissue base. The sections below describe the techniques we employed at various stages to achieve this goal. 1

Corresponding Author: Dr. Suvranu De, Department of Mechanical, Aerospace and Nuclear Engineering, Rensselaer Polytechnic Institute, Email: [email protected]

32

V.S. Arikatla et al. / Cost-Efﬁcient Suturing Simulation with Pre-Computed Models

1.1. Deformation We adopt a linear elastic material model discretized using tetrahedral finite elements. This is a standard procedure and results in a set of simultaneous equations of the form (1) Where,

is the stiffness matrix,

is the displacement vector and

is the external

force vector. We pre-compute the inverse of as for runtime use. During the user interaction, we use the reanalysis technique as in [3] to compute the deformation field and force. If the degrees of freedom are rearranged based on the interacted node, we can write

(2) The sub-matrices in the above equation are derived according to which node the , . user interacts with. Expanding the above matrix, we obtain Since

is small in dimension, its inverse can be computed on the fly. This technique

can only be used if the interaction is local. In order to handle multiple needle/thread interactions simultaneously, we exploit the property of superposition in our linear formulation.

Figure 1. Various stages during the suturing procedure

1.2. Modeling the Suturing Procedure We model the suturing procedure based on the aforementioned reanalysis technique given that the interaction paradigm in the simulation is point-based. Figure 1 shows the

V.S. Arikatla et al. / Cost-Efﬁcient Suturing Simulation with Pre-Computed Models

33

division of the suturing procedure into logical steps based on the needle and thread interaction with tissue. These four stages: (1) The sharp tip of the needle enters the tissue (2) The tip of the needle emerges from the tissue (3) The blunt end goes inside the tissue when the sharp end is pulled by the grasper. The thread also interacts with the tissue (4) the needle is out of the tissue. Only the thread remains inside the tissue. 1.3. Needle-Tissue Interaction When the needle first enters the tissue, the surface triangle which it pierces is determined from the dynamic point algorithm [4] and recorded. The boundary condition from the interaction is applied through displacement conditions on the nearest node of the entry triangle. In essence, the nearest node should follow the point on the needle that is on the tissue surface. Since the needle is curved, we divide it into a set of straight line segments (see Figure 2(a)). At every time step, the nearest point on the needle to the entry/exit is calculated and its displacement is set accordingly. In Stage 2, when the sharp end pierces out of the tissue, the triangle of exit is recorded. Since at this stage two different points on the needle intersect the surface, boundary conditions at both the entry and exit triangles are applied separately and are superimposed to obtain the resulting deformation since the underlying formulation is linear.

Dynamic Point

(a)

(b)

Figure 2. (a) The curved needle is divided into several segments. Each segment has one dynamic point (b) Type 1 and Type 2 interactions of the thread with tissue

1.4. Thread-Tissue Interaction The suture thread is attached to the end of the needle and is meant to hold the tissue in place after the knot is tied. This is modeled with a follow-the-leader (FTL) algorithm [5]. In the FTL algorithm, the nodes away from the one that is being controlled by the user are moved toward their corresponding leader to preserve the length. Although this is a non-physical technique, it is very stable as well as less costly to employ within the requirements of simulating the suture thread. The thread interacts with the base (modeled with FEM) after Stage 2 in the suturing process. Specifically, the thread interacts with the model in two ways. Type 1: The thread is only guided through the point where the needle enters or exits. Type 2: Part of the thread inside the tissue is pulled on either side. Force is imparted to the user.

34

V.S. Arikatla et al. / Cost-Efﬁcient Suturing Simulation with Pre-Computed Models

This force is proportional to the frictional resistance that the tissue imparts to the thread while it is being slid through it. These interactions are shown in Figure 2(b). In Type 1 interaction, the nearest node is snapped to the entry point on the tissue after the FTL update. Hence the FTL update is overridden. In case of type 2 interaction, the entry point on the model follows the node i that is associated with the entry point. This associated node may change as the user pulls the suture thread using the grasper. to reset the association. If the number of segments on We use a threshold force the suture thread is high enough, one can feel the stiction force between the thread and the tissue as in reality. After Stage 4 is reached, the suture should be secured by tying a knot. For this purpose, a simple and real-time knot tying algorithm, as proposed in [6], was employed. It is built around the FTL algorithm for simulating the knot. After the user closes the knot onto the tissue, the nodes that form the knot are frozen from any further moment. A snapshot of the suturing simulation is shown in Figure 3.

Algorithm for Type 2 interaction LOOP 1. 2. 3.

Update Thread using FTL1 & FTL2 Find the nearest node to entry/exit IF a.

Set the boundary of the nearest vertex on the entry/exit triangle to node i on suture

b.

Compute

c.

IF i. ii.

d.

Recalculate and set ELSE

i. 4.

Reset boundary condition to follow node (i+1)

Set the already calculated force

ELSE a. Set nearest node’s position to entry/exit triangle’s centroid.

2. Results Our simulation was run on a desktop computer equipped with an Intel core2 quad 2.66 GHz processor, 2.75 GB RAM and NVIDIA® Quadro4 XGL graphics card. Two PHANTOM® OmniTM devices were used to render the reaction forces calculated from the reanalysis technique.

V.S. Arikatla et al. / Cost-Efﬁcient Suturing Simulation with Pre-Computed Models

35

The cost in a particular time step was divided among dynamic point update, FTL update and the deformation update through reanalysis. The collision detection and FTL were run in a separate thread and run at a frequency of 295Hz with 25 segments for the suture thread and five segments on the curved needle.

(a)

(b)

Figure 3. Suturing simulator (a) With tool interfaces (b) Snapshot of suturing simulation

3. Conclusion/Discussion We have developed algorithms for detailed needle-tissue and thread-tissue interaction with pre-computed models for laparoscopic suturing procedures. These algorithms can simulate the deformation and forces in real-time with minimal cost. Some of the limitations of the present work include being unable to simulate extended tool-tissue contact and large deformation of tissues, which will constitute our future work.

Acknowledgement This work was supported by grant R01 EB005807 from NIH/NIBIB.

References [1]

[2] [3] [4] [5] [6]

Berkley, J., Turkiyyah, G., Berg, D., Ganter, M., and Weghorst, S. 2004. Real-Time Finite Element Modeling for Surgery Simulation: An Application to Virtual Suturing. IEEE Transactions on Visualization and Computer Graphics 10, 3 (May. 2004) M. Bro-Nielsen, Fast Finite Elements for Surgery Simulation, Studies in Health Technological Information, vol. 39, pp. 395-400, 1997. De, S.; Lim, Y.-J.; Muniyandi, M. & Srinivasan, M. A. Physically Realistic Virtual Surgery Using the Point-Associated Finite Field (PAFF) Approach. Presence, 2006, 15, 294-308. Maciel, A. and De, S. 2008. An efficient dynamic point algorithm for line-based collision detection in real time virtual environments involving haptics. Comput. Animat. Virtual Worlds 19, 2 (May. 2008) Brown, J., Latombe, J.-C., and Montgomery, K. 2004. Real-time knot-tying simulation. The Visual Computer 20, 2-3, 165–179. Sankaranarayanan G, De S. A real-time knot detection algorithm for suturing simulation, Stud Health Technol Inform. 2009; 142: 289-91.

36

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-36

Anesthesia Residents’ Preference for Learning Interscalene Brachial Plexus Block (ISBPB): Traditional Winnie’s Technique vs. Ultrasound-Guided Technique Imad T AWADa, Colin SINCLAIRa, Ewen W CHENa, Colin JL MCCARTNEYa, Jeffrey JH CHEUNGa and Adam DUBROWSKIb a Department of Anesthesia, Sunnybrook Health Sciences Centre b Sick Kids Learning Institute, University of Toronto

Abstract. There is a recent shift from traditional nerve stimulation (NS) to ultrasound-guided (UG) techniques in regional anesthesia (RA). This shift prompted educators to readdress the best way to teach these two modalities. Development of a more structured curriculum requires an understanding of student preferences and perceptions. To help in structuring the teaching curriculum of RA, we examined residents’ preferences to the methods of instruction (NS Vs. UG techniques). Novice residents (n=12) were enrolled in this parallel crossover trial. Two groups of 6 residents received a didactic lecture on NS or UG techniques. The groups then crossed over to view the other lecture. After they observed a demo of ISBPB on two patients using NS and US. The residents completed a questionnaire regarding their impression of each technique and the learning experience. UG technique was perceived to be safer and to have more educational value than NS. However, residents felt both techniques should be mandatory in the teaching curriculum. Keywords. Regional anesthesia, teaching curriculum, interscalene block

Introduction The inconsistency of RA teaching in the majority of anesthesia residency programs in North America is due in part to the lack of sufficient clinical exposure [1-3]. As well, the clinical practice of RA in the last six years has undergone transition from traditional NS to UG techniques. This has perhaps diluted the experience residents receive in both traditional landmark and ultrasound imaging techniques. Such transition necessitates a change in our educational models, with an increasing need to develop guidelines and teaching curricula to standardize the practice and teaching of regional anesthesia [4]. In the current study we surveyed novice anesthesiology residents about their preferences of teaching traditional NS and UG methods, as well as their perceptions of the safety and educational value of these two approaches. Understanding the trainees’ needs by assessing their preferences and perceptions is the first necessary step in developing better-structured future educational models.

I.T. Awad et al. / Anesthesia Residents’ Preference for Learning Interscalene Brachial Plexus Block

37

1. Material and Methods With Institutional Ethics Board approval, 12 novice anesthesia residents were recruited in this prospective observational crossover study. The students completed an initial survey to ensure they had no significant experience in either NS or UG regional anesthesia techniques. The residents were then given two 30-minute lectures on interscalene brachial plexus block (ISBPB), one with NS technique and the other with UG technique. Both lectures were standardized in time and content, and were delivered by an expert in the respective technique. Afterward, the residents viewed two real-time demonstration of ISBPB by an expert, one with a NS and the other with US. To avoid an order of training effect (recency effect), the residents were randomized into two groups to counter-balance the viewing order for both lectures and the demonstrations. Residents then completed a questionnaire looking at their understanding of the basic anatomy and ISBPB technique, their preference in technique for future teaching for this block, perceived safety, risk of complications, and educational value of the each technique. The questionnaire was peer-reviewed by a group of five regional anesthetists and a medical psychologist. All reviewers were uninvolved in the development of the original questionnaire. Descriptive statistics were used to summarize the data, with counts and percentages presented for question responses. Analyses were carried out with SAS Version 9.1 (SAS Institute, Cary, North Carolina, USA).

2. Results The initial survey revealed that the residents had minimal experience with regional blocks, in particularly with ISBPB. Residents preferred to have equal emphasis of training in their residency using both traditional NS and UG techniques compared to traditional alone. Residents felt nerve blocks performed under UG would result in fewer complications overall (p<0.01). More specifically that it would decrease the incidence of inadvertent arterial puncture (p<0.01), pneumothorax (p<0.01), intravascular injection (p<0.01), and that it provides a more definite endpoint for needle placement (p<0.01). As well, more residents thought there would be greater overall patient satisfaction in anesthesia (p=0.01) with UG imaging and its use would lead to better patient outcomes (p=0.01). Also, 66.7% of residents felt UG imaging would provide better educational value compared to 8.3% traditional (P=0.003) and 25.0% no difference (p=0.04).

3. Discussion The purpose of this pilot survey was to assess novice anesthesia residents’ preferences for training in regional anesthesia using traditional NS and US techniques. During preand post-education session surveys, twelve-novice anesthesia resident indicated that they had equal preferences in receiving teaching in regional anesthesia using both NS and US-guided techniques. More importantly however, the residents’ perceptions indicated that the US imaging may lead to fewer patient complications and it provided greater educational value. Literature searches have identified a potential gap in education practice, where to date there is no standard approach or structure as how educators would teach regional

38

I.T. Awad et al. / Anesthesia Residents’ Preference for Learning Interscalene Brachial Plexus Block

anesthesia. With the recent developments of ultrasound guided regional anesthetic techniques, and the need to maintain the traditional techniques, as well as with the development of simulation augmented teaching models, evidence based guidance is needed to develop educational curricula that expose the trainees to a balanced variety of techniques in a safe, simulation-based environment. In most institutions training of regional anesthesia is didactic and hands-on with no set target of competency at the end of training. These standards have been developed based on educators experience and professional intuition rather than systematic, evidence based approach. Therefore our study is the initial effort to generate evidence for the development of such programs. One of the limitations of this pilot study is the small sample size of residents who are from the same university. We have chosen novice residents with limited experience in regional anesthesia to avoid any bias in opinion. It would be interesting to compare the novice opinion against senior experienced residents’ preference of training. The training preferences of more experienced anesthesia residents (PGY5), before and after their one-month elective rotation in regional anesthesia need to be systematically assessed. It is possible that that cohort may have different educational requirements to novices. Based on the responses from this cohort of junior trainees, we conclude that it would be reasonable and advisable to train anesthesia residents in both traditional landmark and ultrasound-guided techniques for ISBPB, rather than with one technique exclusively. It may be inferred that this preference could be extrapolated to other peripheral nerve block techniques. We therefore propose that teaching of novice residents in regional anesthesia could become more structured and integrate anatomical and ultrasound-guidance techniques.

Conclusion Residents expressed equal preference for both techniques to be part of the training curriculum. There was a clear preference towards using the US technique in terms of safety and educational value.

References [1] Hadzic A, Vloka JD, Kuroda MM, Koorn R and Birnbach DJ. The practice of peripheral nerve blocks in the United States: a national survey. Reg Anesth Pain Med 1998; 23: 241-6. [2] Smith MP, Sprung J, Zura A, Mascha E and Tetzlaff JE. A survey of exposure to regional anesthesia techniques in American anesthesia residency training programs. Reg Anesth Pain Med 1999; 24: 11-16. [3] Chelly JE, Greger J, Gebhard R, Hagberg CA, Al-Samsam T and Khan A. Training of residents in peripheral nerve blocks during anesthesiology residency. J Clin Anesth 2002; 14: 584-8. [4] Marhofer P and Chan VW. Ultrasound-guided regional anesthesia: current concepts and future trends. Anesth Analg 2007; 104: 1265-9.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-39

39

Fuzzy Control of a Hand Rehabilitation Robot to Optimize the Exercise Speed in Passive Working Mode Mina Arab BANIASADa,b, Mohammad AKBARa, Aria ALASTYa, Farzam FARAHMANDa,b,1 a School of Mechanical Engineering, Sharif University of Technology, Tehran, Iran b RCSTIM, Tehran University of Medical Sciences, Tehran, Iran

Abstract. The robotic rehabilitation devices can undertake the difficult physical therapy tasks and provide improved treatment procedures for post stroke patients. During passive working mode, the speed of the exercise needs to be controlled continuously by the robot to avoid excessive injurious torques. We designed a fuzzy controller for a hand rehabilitation robot to adjust the exercise speed by considering the wrist angle and joint resistive torque, measured continuously, and the patient’s general condition, determined by the therapist. With a set of rules based on an expert therapist experience, the fuzzy system could adapt effectively to the neuromuscular conditions of the patient’s paretic hand. Preliminary clinical tests revealed that the fuzzy controller produced a smooth motion with no sudden change of the speed that could cause pain and activate the muscle reflexive mechanism. This improves the recovery procedure and promotes the robot’s performance for wide clinical usage. Keywords. Fuzzy Control, Rehabilitation Robotics, Stroke

Introduction Thirty percents of patients that survive from stroke suffer from disability in the forms of partial or complete motor limitation of upper and lower limbs [1]. The recovery of power and motor control of post stroke patients is only possible via intensive rehabilitation exercises, which traditionally require strenuous manual efforts by the therapists, due to the high resistance of the spastic muscles. The robotic rehabilitation devices, introduced in recent years, can undertake this difficult task and provide improved treatment procedures, using new sensory-motor rehabilitation strategies. It has been reported that robotic rehabilitation shortens the recovery time, reduces the ennui of patient and therapist and prevent soft tissue and joint injuries [2]. Wrist-RoboHab is an upper limb robot-mediated rehabilitation device, designed by our group, to ease the therapeutic practices for therapists and post stroke patients [3]. It provides passive and active exercises unilaterally or bilaterally in three different states: pronation/supination of the forearm, and flexion/extension and abduction/adduction of the wrist (Fig 1). Five working modes have been implemented in the system: (1) Passive Mode, in which both the paretic and healthy hands of the patient are derived by the 1 Corresponding Author: School of Mechanical Engineering, Sharif University of Technology, and RCSTIM, Tehran University of Medical Sciences, Tehran, Iran; E-mail:[email protected]

40

M.A. Baniasad et al. / Fuzzy Control of a Hand Rehabilitation Robot

robot. This is the most routine working mode of the robot for treatment of spastic patients and needs adjustment of velocity and range of motion by the therapist while the torque is limited to a prescribed level. (2) Active mode, in which the patient moves his healthy hand (master) to activate the robot (slave) that drives the paretic extremity in the mirrored direction. The torque at the slave is measured and applied to the master as a force feedback to provide a real-time dynamic sensation. (3) Resistive mode which is almost the same as the active mode except that a prescribed torque is applied to the master which drives the paretic hand at the slave. This working mode is designed for flaccid patients for whom no resistive torque is generated by the paretic hand. (4) Active-assisted mode, in which the patient makes effort to move the paretic hand against an adjustable resistance and is assisted, to some extent, by the healthy hand in case he is not capable to complete the range of motion. (5) Active-constrained mode in which the patient makes effort to move his healthy hand while the corresponding handle is fixed. This can generate small movements at the paretic side due to the brain irradiation effect.

(a)

(b)

Figure 1. Wrist-RoboHab robot during flexion-extension (a) and supination-pronation (b) exercises.

It has been shown that during the passive manipulation of a spastic extremity, a higher exercise speed would lead to a higher resistance due to the viscoelastic behavior of the spastic muscles [4]. So, it is necessary to decrease the exercise speed when the muscle resistance rises excessively, in order to avoid injurious loads at the tendons and bones. This requirement has been implemented in previous robotic rehabilitation devices with an immediate drop of the exercise speed as soon as the measured torque at the paretic extremity exceeded a prescribed torque. However, the resulting sudden changes of the speed might activate the reflexive mechanism of muscles [5] making the exercises painful and disturbing the recovery process [4]. An effective solution for this problem is to develop a detail musculoskeletal model of the paretic extremity and employ a modelbased controlling algorithm. However, considering the fact that such a complicated model has not been developed yet, the purpose of this study was to design a fuzzy controller, based on the experience of an expert therapist, to adjust the exercise speed smoothly and safely during the rehabilitation process of the spastic extremity.

Method In the conventional physical therapy of the hand of the post stroke patients, the therapist determines the optimal exercise speed based on three parameters: (1) the wrist angle (2) the corresponding resistive torque, and (3) the patient’s general condition. We assumed

41

M.A. Baniasad et al. / Fuzzy Control of a Hand Rehabilitation Robot

these parameters as the inputs of the fuzzy controller and defined their membership functions considering their importance, evaluated through consultation with an expert therapist. For the patient’s general condition, which is a main determinant of the maximum allowable torque and consequently the exercise speed, we defined the membership function of Figure 2(a) with the FL, CVA, and SP representing the flaccid, intermediate, and spastic conditions, respectively. Also membership functions of figures 2(b) and 2(c) were allocated to the wrist angle, which affects the muscles’ lengths and effective moment arms, and the joint resistive torque, as the main determinant of the optimal exercise speed, respectively. The output of the fuzzy system was considered to be the exercise speed with a membership function defined in Figure 2(d).

Figure 2. Membership functions for (a) patient's general condition, (b) wrist angle, (c) joint resistive torque, and (d) exercise speed.

Following defining the fuzzy system inputs and output and theirs membership functions, the fuzzy rules were established by using the experiences of an expert physical therapist. Table 1 indicates the 36 rules of the fuzzy system which were used to obtain the system’s output. For instance, Table 1(c) indicates that for a spastic patient (Pa=SP) in zero wrist angle (=Z) with 5 Nm resistive torque (T=S1), the appropriate exercise speed is 5 (V=S1). Also for a patient with SP condition and wrist angle of Z and hand resistance of S1, the appropriate exercise speed is S1. Table 1. Fuzzy rule base S1

CEV

S1

S2

B2

B1

CEV

CEV

S1

CEV

CV

CEV

CEV

S2

B1

B2

CEV

CEV

CEV

S1

B1

S1

S1

S2

S1

S1

S2

S1

S1

S2

B2

S1

S2

S2

S1

S1

S1

S1

S1

S1

N

Z

P

N

Z

P

N

Z

P

T

(a) Patient’s condition: FL

(b) Patient’s condition : CVA

(c) Patient’s condition: SP

42

M.A. Baniasad et al. / Fuzzy Control of a Hand Rehabilitation Robot

To find the value of the system’s output, i.e., the speed of the exercise, we used the singleton fuzzifier, product inference engine and center average defuzzifier [6]:.

(1)

where !" # are respectively the values of ith patient condition, jth wrist angle and kth hand resistance membership functions. Also $% " is the center of the output of the rule that fires the membership function of each parameter. In order to implement the designed fuzzy controller on the Wirst-RoboHab, a visual C++ code, was developed. An encoder (Amtec Robotics Co., Munich, Germany) and a 10 Nm capacity torque sensor TCN 16 (Dacell Co., Chung-buk, Korea), respectively, were used to measure the wrist angle and the corresponding resistive torque continuously during the passive working mode of the robot. These data were communicated to a PC to calculate the optimal speed of the exercise and control the driving motors PR 070 (Amtec Robotics Co., Munich, Germany).

(a)

(b)

Figure 3. Speed vs. wrist angle for (a) non-fuzzy, and (b) fuzzy controlling systems.

Results After ensuring the safety and acceptable technical performance of the controlling algorithm, some preliminary clinical tests were accomplished. The passive rehabilitation exercise was applied to a female patient, suffered from CVA at her left side, in the presence of an expert therapist. The resistive torque and the applied exercise speed, as well as the joint angle, were recorded and analyzed in one exercise cycle. The exercise cycle was defined as a supination motion of the wrist from zero to 60º, followed by a pronation motion from 60º to -60º and then a supination motion returning the wrist to the initial position. Figures 3(a) and 3(b) illustrate the variation of the exercise speed versus the wrist angle obtained by the non-fuzzy and fuzzy controlling systems, respectively. The fuzzy controller changed the exercise speed more smoothly in comparison with non fuzzy system. With such a smooth change of the speed, the rehabilitation exercise of the wrist joint of the paretic hand would be painless and much more effective. Figures 4(a) and 4(b) illustrate the change of the resistive torque in the non-fuzzy and fuzzy controlling systems, respectively. Comparison of the two graphs indicates that

M.A. Baniasad et al. / Fuzzy Control of a Hand Rehabilitation Robot

43

both systems reached the maximum allowable torque at the end range while the resistance was small in the midrange. This is not unexpected considering the spastic behavior of the paretic hand. However, the change of the resistive torque in the nonfuzzy controlling system was not smooth and included several sharp peaks in the exercise cycle. These sudden changes of the resistive torque are due to the stepwise change of the exercise speed (Fig 3(a)) and indicate that the reflexive mechanism of the paretic hand has been activated. When the fuzzy controlling system was used, however, the resistive torque varied smoothly with the wrist angle with no instantaneous changes in the torque-angle curve.

(a)

(b)

Figure 4. Resistance vs. wrist’s angle for (a) non-fuzzy, and (b) fuzzy controlling systems.

Conclusion The fuzzy based controlling system developed in this study can adapt effectively to the neuromuscular conditions of the post stroke patients during passive rehabilitation exercises. Thanks to the intelligent approach employed, the paretic extremity might be derived smoothly by the rehabilitation robot with no sudden changes of the exercise speed that might cause pain and activate the muscle reflexive mechanism. This is a major improvement is comparison with the previous controlling systems, which were only based on the maximum allowable torque, and can extend the application of the robot-mediated rehabilitation devices in real clinical practice.

References [1] W. Rosamond, et al., Heart disease and stroke statistics--2007 update: A report from the American heart association statistics committee and stroke statistics subcommittee. Circulation., 115(5): e69-171, 2007 [2] H. I. Krebs, et al., Rehabilitation robotics: pilot trial of a spatial extension for MIT-Manus. J Neuroeng Rehabil., 1: 5, 2004. [3] E. Rashedi, et al., Design and development of a hand robotic rehabilitation device for post stroke patients, Proceedings of the 31st Annual International Conference of the IEEE EMBS. September 2-6 2009, Minneapolis, Minnesota, USA. [4] E. R. Kandel, et al., Principles of neural science, 4th ed., McGraw-Hill, Health Professions Division, 2000. [5] V.B. Brooks, The neural basis of motor control, Oxford University Press, London, 1986. [6] L.X. Wang, A course in fuzzy systems and control, Prentice-Hall Inc., New York, 1997.

44

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-44

Engaging Media for Mental Health Applications: the EMMA Project R. BAÑOS ad,1, C. BOTELLA bd, S. QUERO bd, A.GARCÍA-PALACIOS bd, M. ALCAÑIZ cd a Universidad de Valencia b Universidad Jaume I c Universidad Politécnica de Valencia d CIBER de Fisiopatología de la Obesidad y Nutrición (CIBEROBN) Abstract. EMMA project has been focused on how the sense of presence in virtual environments mediates or generates emotional responses, and how to use presence and emotional responses in virtual environments effectively in clinical and non clinical settings. EMMA project has developed two different virtual environments. The first one acts as a 'mood device' and is aimed to induce and enhance several moods on clinical and non clinical subjects. The second one is a virtual environment that acts as an adaptive display to treat emotional disorders (Posttraumatic Stress Disorder, Adjustment Disorder and Pathological Grief). This virtual world varies the contents that are presented depending on the emotions of the patient at each moment. The goal of this paper is to outline the main goals achieved by this project Keywords. Emotions, Virtual Reality, Presence, Computer Aided Psychotherapy, Psychological Treatments, Stress-related Disorders, Cyberpsychology

Introduction The notion of “being present” in the virtual worlds has been considered central to Virtual Environments (VE) endeavours since its conception. Presence is traditionally thought of as the psychological feeling of “being in” or “existing in” the VE in which one is immersed. But beyond this simple definition, what exactly presence is and what causes presence remains somewhat of a mystery. In the last years, scientific literature has mainly paid attention to the cognitive and environmental determinants of presence. However, emotional responses can also play an important role in generating and enhancing presence, specifically for some Virtual Reality (VR) applications, such as mental health field. If we are able to understand better presence and emotional reactions to virtual environments, we will be able to design more effective “virtual experiences”. EMMA research project has been funded by the European Union (Engaging Media for Mental Health Applications, IST-2001-39192) with the main purpose of studying the relationships between presence and emotions in VEs. Specifically, EMMA project has been focused on two strategic objectives: 1) how the sense of presence mediates or generates affective and emotional responses, and 2) how to use presence and emotional responses in VEs effectively in clinical and non clinical settings. 1

Dpto. Personalidad, Evaluación y Tratamientos Psicológicos. Facultad de Psicología, Avda. Blasco Ibáñez, 21, 46010-Valencia, Spain.

R. Baños et al. / Engaging Media for Mental Health Applications: The EMMA Project

45

In order to achieve these goals, EMMA project has developed two different VEs. The first one works as a 'mood device' and is aimed to induce and enhance several moods on clinical and non clinical subjects. The second one is a VE that acts as an adaptive display to treat emotional disorders: Post-traumatic Stress Disorder (PTSD), Adjustment Disorder (AD) and Pathological Grief (PG). This virtual world varies the contents that are presented depending on the emotions of the patient at each moment. The aim of this paper is to briefly describe these VEs and outline the main goals achieved by this project.

1. EMMA as a ‘Mood Device’ 1.1. EMMA’s Parks In order to generate emotions in the user, an emotionally significant environment was developed. It consisted of a VR Mood Induction Procedure (VR-MIP) that progressively changes depending on the mood state to be evoked in the user (sadness, joy and anxiety). The scenario is a neutral park. We chose this environment because it includes elements of nature (trees, flowers, water, etc.), and because changing some of the light parameters (tone, direction, brightness) easily modifies the aspect of these elements, inducing different moods in the user. For example, in the case of sadness, the park is grey, it is a cloudy day, the trees have no leaves, and the music that is heard is very sad (see Figure 1). The aspect of this initially neutral park changes depending on the emotion. From the technical point of view, changes in the textures have been made, but also some specific objects appear at the park according to the emotion. Furthermore, with the aim of building the different environments, variations of several traditional MIP were included: music, narratives, Velten self-statements [1], as well as pictures (selected from the International Affective Picture System IAPS, [2]), movies, and autobiographical recalls. For a more detailed description see Baños et al. [3].

Figure 1. One view of the sad park.

46

R. Baños et al. / Engaging Media for Mental Health Applications: The EMMA Project

1.2. Main Goals Achieved with EMMA’s Parks With the aim of studying the relationship between emotions and presence, we conducted several studies using the EMMA’s Parks with non clinical populations. In a first study [4], we tested the role of immersion and media content in the sense of presence. Three immersive systems were used (a PC monitor, a rear projected video wall, and a head-mounted display) and two VEs, one involving emotional content (sadness) and the other not; results indicated that although immersion had an impact on presence, this role was more relevant for non-emotional VE than for emotional VE. In the study, we concluded that presence is not a direct function of immersion alone, and a one-to-one relationship cannot be assumed to exist between immersion and presence. In a later work [5] the goal was to test how stereoscopy (the illusion of depth and 3D imaging) affects the sense of presence and the intensity of the positive mood that users felt in VEs. Results showed that there were no differences between stereoscopic and monoscopic presentation neither subjective sense of presence and emotional reactions in VEs. These results did not replicate the previous findings [6,7] showing that subjective feelings of presence are enhanced by stereoscopic stimuli presentation. However, unlike the VEs used in these studies, the EMMA`s Park was designed specially to induce positive moods (not neutral). It may be that for this specific purpose, stereoscopic presentation is not as critical and technological factors are more relevant for non-emotional environments than for emotional ones. Furthermore, our results support the hypothesis that stereoscopy does not affect the intensity of mood induction. And, finally, results replicated previous findings [3,4] that indicated that the subjective sense of presence is related to emotional reactions. Participants who experienced a strong feeling of presence also reported stronger positive emotional reactions. EMMA’s Parks as a VR-MIP has also been tested as an efficacious procedure to evoke different emotions in users. In a study carried out with non clinical population [3], results showed the efficacy of this procedure not only to induce a target mood (sadness), but also to change this previous induced mood to an opposite emotion (happiness). However, there are still many unanswered questions that future research should clarify. For instance, an important question is: how long do induced moods last, and to what extent are they transitory? Regarding to this, our research group is currently conducting a work whose aim is to test the role of consecutive positive mood induction virtual procedures on satisfaction of life of people. ! $ % &"This is a work in progress, but preliminary data [8] shows that VR- $

! #

" !

" Therefore, if mood induction has positive long term effects, VR-MIP could be a useful device to include in treatment protocols as a therapeutic tool not only to induce specific moods (relax or joy) in people who need it but also to increase their well being.

R. Baños et al. / Engaging Media for Mental Health Applications: The EMMA Project

47

2. EMMA as a ‘Clinical Adaptive Display’ 2.1. EMMA`s World The second VE developed in EMMA project is an adaptive display to treat emotional disorders, named EMMA’s world. Most of the VEs currently available in the field of psychological treatments were designed to solve a specific problem. ‘EMMA’s world’ is a versatile VR system (an “adaptive display”), which can address a wide range of psychological problems. Initially, it was designed with the aim of treating PTSD, AD and CG. This virtual world was designed to assist patients to confront situations where they have suffered or are suffering a stressful experience. To accomplish therapeutic goals, a series of emotional virtual elements are used and personalized so that they are meaningful to the user and contain the fundamental emotional elements that the person must confront. In this VR application, the therapist and the patient can represent the experience suffered by the patient according to the specific therapeutic needs. The aim is to design clinically significant environments for each individual, emphasizing more the meaning of the event than the realism of the VR environment to recreate the situation, the physical characteristics of the event. The aim is not realism, but using customized aspects that help to process the emotional event in a safe and protective environment. The main elements of EMMA’s World are EMMA`s room and the Book of life. EMMA’s room is an architectonical structure that contains several elements. The first one is a Database Screen, where a list of icons shows all the elements that users can manipulate, including three-dimensional objects, sounds, images, coloured lights, movies and texts. All these elements have been designed to help patients confront and manage the emotions and experiences that they have gone in his/her life. Another important tool is the Book of Life, a virtual book where patients can reflect upon feelings and experiences. The objective is to represent the most important moments, people and situations in the person’s life (related to the traumatic or negative experience). Anything that is meaningful for the patient can be incorporated in the system: photos, drawings, phrases, videos, etc.. EMMA’s World also includes five different pre-defined scenarios or ‘landscapes’ (see Figure 2): a desert, an island, a threatening forest, a snow-covered town and meadows. These environments have been designed to reflect different emotions (relaxation, elation, sadness, etc). Their specific use depends on the context of the session and can be selected by the therapist in real time. The aim is to reflect and enhance the emotion that the user is experiencing or to induce certain emotions. It is possible to include modifications in the scenario and to graduate their intensity in order to reflect the changes in the participants’ mood states. For example, in the elation landscape, EMMA’s room is surrounded by green hills and trees. It is a beautiful sunny day. The therapist can also create different effects in the environment such as rain, snow, earthquakes, etc. A more detailed description of the kind of actions that can be performed by patients and therapists can be found in Rey et al. [9].

48

R. Baños et al. / Engaging Media for Mental Health Applications: The EMMA Project

Figure 2. Various scenarios in the virtual environment.

2.2. Results Obtained with EMMA’s World To treat diverse Stress-related-Disorders (PTSD, AD or CG), flexible VR applications capable to evoke the different stressful events could be very useful. EMMA’s world provides clinically significant and personalized environments for each patient, focusing on his/her particular trauma meaning. This emotional adaptive display has the advantages highlighted in several review works [10-12] for the traditional VR systems which simulate the reality. In the concrete case of PTSD, EMMA’s world can help to overcome the limitations of prolonged imaginal exposure, which even being the treatment of choice for this problem [13], is underused in the clinical practice [14] due to several reasons. First, there is the cognitive avoidance showed by some patients who are unwilling to cope with trauma reminders because they find it too aversive. Others are able to think about the trauma but they are detached from the experience resulting in poor therapeutic results. A second problem could be the poor imagination capacity presented by some patients. Lastly, therapists are also reluctant to use this technique. EMMA’s World by using symbols to carry out the exposure to the trauma reminders can prevent cognitive avoidance and, then, increase the emotional involvement, a critical issue for the exposure efficacy. On the other hand, the main advantage of EMMA’s World for the Clinical psychology field is its versatility, which allows the treatment of a big variety of psychological disorders in which negative emotions play an important role. This is the case of the stress-related disorders aforementioned. EMMA’s World is an open VR system where the main objective is not to simulate the physical characteristics of the negative or traumatic event, but to use personalized symbols and elements that evoke a emotional reaction to the person and, therein, help the person to emotionally process the negative event, creating simultaneously a safe and protected timeless space. In this space the person can rest and recover resilience capacity to cope with the future.

R. Baños et al. / Engaging Media for Mental Health Applications: The EMMA Project

49

Several works conducted by our research team have shown the utility of EMMA’s World for the treatment of the aforementioned stress-related disorders: PTSD [15,16]; PG [17] and AD [18]. Furthermore, results obtained in another two studies where participants with the three diagnoses were included found preliminary results about the efficacy of EMMA’s World and participants also informed to have a good opinion and satisfaction with the treatment [19, 20]. Lastly, EMMA’s World has been used with success in the treatment of other disorders like specific phobias (specifically, storm phobia and darkness phobia) and with other populations; elderly people in the case of storm phobia [21] and darkness phobia in the case of children [22].

3. Discussion What distinguishes VR from other media or communication systems is the sense of presence: VR can be considered as the leading edge of a general evolution of present communication interfaces like television, computer and telephone whose ultimate goal is the full immersion of the human sensormotor channels into a vivid and interactive communication experience. The presence feeling induced by VR has helped this medium to find a significant space in mental health treatment. VR is playing an important role as presence-enhanced supportive technique. Through presence, VR helps patients to confront their problems in a significant but controlled and safe setting. In recent years, several VR applications have proven useful for applying exposure techniques to the treatment of various psychological disorders. VR has opened the possibility to patients of experiencing their life in another more satisfactory way. The applications developed in the EMMA project tried to be a step forward in this way. The EMMA project has proposed to use the sense of presence in VEs to mediate and generate emotional responses, and to use presence and emotional responses in VEs to effectively treat emotional problems in clinical settings. The focus has been on designing applications to elicit emotions with the goal to reduce or modify them, and on designing affectively significant environments, including those elements with the potential of activating emotions. To do that, EMMA has proved that it is not necessary to copy physical reality exactly as it is. A user could experience virtual presence even when the VE does not represent completely or with total precision the real world. Even more, VR could produce alternative and fantastic worlds, which is one of its more attractive features. Furthermore, EMMA project has proposed to use more flexible VEs than other virtual systems. Because EMMA world permits customization of the environments according to the needs and preferences of users, it can be applied to a wide variety of problems. VR therapists’ can use presence to provide meaningful experiences able to induce a deep and permanent change in their patients. We think that this is the first step in a line of research that will be very productive. This will allow us to work with ‘virtual worlds’ and ‘real worlds’ that fit the specific needs of every patient in different situations.

50

R. Baños et al. / Engaging Media for Mental Health Applications: The EMMA Project

References [1] E. Velten. A laboratory task for induction of mood states. Behaviour Research and Therapy 6 (1968), 473-482. [2] P.J. Lang, M.M. Bradley & B.N Cuthbert. International Affective Picture System (IAPS): Technical Manual and Affective Ratings. NIMH Cent. Study Emot. Atten. University of Florida, 1995. [3] R.M. Baños, V. Liaño, C. Botella, M. Alcañiz, B. Guerrero & B. Rey. Changing induced moods via virtual reality. In W. IJsselsteijn, Y. de Kort, C. Midden, B. Eggen, E. van den Hoven, (Eds.) Persuasive Technology: Lecture Notes in Comnputer Science. Berlin/Heilderberg: Springer-Verlag, 2006. [4] R.M. Baños, C. Botella, M. Alcañiz, V. Liaño, B. Guerrero & B. Rey. Immersion and emotion: their impact on the sense of presence. CyberPsychology & Behavior 7 (6) (2004), 734-741. [5] R.M. Baños, C. Botella, I. Rubió, S. Quero, A. García-Palacios & M. Alcañiz. Presence and emotions in virtual environments: the influence of stereoscopy. CyberPsychology and Behavior 11 (1) (2008), 1-8. [6] C. Hendrix & W. Barfield. Presence within Virtual Environments as a Function of Visual Display Parameters, Presence: Teleoperators and Virtual Environments, 5(3) (1996), 274-289. [7] A. IJsselsteijn, H. de Ridder, J. Freeman, S. Avons & D. Bouwhuis. Effects of stereoscopic presentation, image motion and screen size on subjective and objective corroborative measures of presence. Presence: Teleoperators and Virtual Environments 10 (2001), 298-311. [8] R.M. Baños, G. García-Soriano, C. Botella, E. Oliver, E. Etchemendy, J. Bretón & M. Alcañiz. Positive mood induction and well being. Proceedings - 2009 2nd Conference on Human System Interactions, HSI '09, art. no. 5091032, 517-519. [9] B. Rey, J. Montesa, M. Alcañiz, R.M. Baños & C. Botella. A Preliminary study on the use of an Adaptive Display for the treatment of emotional disorders. PsychNology Journal 3 (1) (2005), 101-112. [10] C. Botella, S. Quero, R.M. Baños, C. Perpiñá, A. García-Palacios & G. Riva. Virtual Reality and Psychotherapy. In G. Riva, C. Botella, P. Legeron & Optale (Eds.) Cybertherapy. Amsterdam: IOSS Press 2004. [11] B.K. Wiederhold & M.D. Wiederhold. A review of virtual reality as a psychotherapeutic tool. Cyberpsychology & Behavior, 1 (1998), 45-52 [12] E., Zimand, B. Rothbaum, L. Tannenbaum, M.S. Ferrer & L. Hodges. Technology meets psychology: Integrating virtual reality into clinical practice. The Clinical Psychologist, 56 (2003), 5-11. [13] E.B. Foa, T. M. Keane & M.S. Friedman (Eds.). Practice guidelines from the international society for traumatic stress studies: effective treatments for PTSD. The Guilford Press, New York, 2000 [14] C.B. Becker, C. Zayfert, & E. Anderson. A survey of psychologists’ attitudes towards and utilization of exposure therapy for PTSD. Behaviour Research and Therapy, 42 (2004), 277-292. [15] C. Botella, S. Quero, N. Lasso de la Vega, R.M. Baños, V. Guillén, A. García-Palacios & D. Castilla. Clinical issues in the application of virtual reality to the treatment of PTSD. In M. Roy (Ed.) Novel approaches to the diagnosis and treatment of posttraumatic stress disorder. NATO Security Through Science Series vol. 6. IOS Press, Amsterdam, 2006. [16] C. Botella, A. García-Palacios, V. Guillen, R.M. Baños, S. Quero & M. Alcaniz. An Adaptive Display for the Treatment of Diverse Trauma PTSD Victims. Cyberpsychology & Behavior 13 (2010), 67-71. [17] C. Botella, J. Osma, A. García Palacios, V. Guillén & R.M. Baños. Treatment of Complicated Grief using Virtual Reality. A Case Report. Death Studies, 32 (7) (2008), 674-692 [18] R.M. Bños, C. Botella, V. Guillen, A. García-Palacios & S. Quero. Un programa de tratamiento para los trastornos adaptativos: un estudio de caso. Apuntes de Psicología, 26 (2008), 303-316. [19] C. Botella, R.M. Baños, B. Rey, M. Alcañiz, V. Guillén, S. Quero & A. García-Palacios. Using an Adaptive Display for the Treatment of Emotional Disorders: A preliminary analysis of effectiveness. CHI 2006, April 22-27, 2006, Montreal, Canadá. [20] R.M. Baños, C. Botella, V. Guillen, A. Garcia-Palacios, S. Quero, J. Bretón-López & M. Alcaniz. An adaptive display to treat stress-related disorders: the EMMA’s world. British Journal of Guidance and Counselling, 37 (3) (2009), 347-356. [21] C. Botella, R.M. Baños, B. Guerrero, A. García-Palacios, S. Quero & M. Alcañiz. Using a flexible Virtual Environment for Treating a Storm Phobia. PsychNology, 4 (2) (2006), 129-144. [22] M. Alcañiz, C. Botella, B. Rey, R. Baños, J.A. Lozano, N. Lasso de la Vega, D. Castilla, J. Montesa & A. Hospitaler. EMMA: an adaptive display for virtual reality. Lecture Notes in Computer Sciencie, Volume 4565 LNAI (2007), 258-265.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-51

51

NeuroSim - The Prototype of a Neurosurgical Training Simulator , Stephan DIEDERICH a , Kirsten SCHMIEDER b and Reinhard MÄNNER a,c a Institute for Computational Medicine, University of Heidelberg b Department of Neurosurgery, Medical Faculty Mannheim, University of Heidelberg c Department of Computer Science V, University of Heidelberg Florian BEIER

a,1

Abstract. We present NeuroSim, the prototype of a training simulator for open surgical interventions on the human brain. The simulator is based on virtual reality and uses real-time simulation algorithms to interact with models generated from MRTor CT-datasets. NeuroSim provides a native interface by using a real surgical microscope and original instruments tracked by a combination of inertial measurement units and optical tracking. Conclusively an immersive environment is generated. In a ﬁrst step the navigation in an open surgery setup as well as the hand-eye coordination through a microscope can be trained. Due to its modular design further training modules and extensions can be integrated. NeuroSim has been developed in cooperation with the neurosurgical clinic of the University of Heidelberg and the VRmagic GmbH in Mannheim. Keywords. Virtual Reality, Medical Training Simulator, Neurosurgery

Introduction Neurosurgical interventions on the human brain are complicated and highly risky. Although minimal invasive techniques are used more often, there is still need for open surgical interventions, which can be accomplished only by very well trained and experienced surgeons. “See one, do one, teach one” is the most common axiom for acquiring medical skills although this method might endanger patients. Another possibility is the training on plastic models, living animals or dead bodies. So there is a great need for an efﬁcient training environment that is realistic without involving real patients or animals. Virtual reality (VR) can be used in order to implement such a training system. Apart from the properties mentioned, VR-simulators have several advantages: Surgical tasks are reproducible and can be trained at any time, even if the case is rare. The surgeon’s skills are measured objectively and the result can be compared to other users. Although there are some groups that are developing neurosurgical simulators [DeMauro08,NeuroTouch], we are not aware of any project that combines the native interface of a moveable surgical microscope with original instruments. 1 Corresponding Author: Florian Beier, Institute for Computational Medicine, University of Heidelberg, Germany, E-mail: ﬂ[email protected]

52

F. Beier et al. / NeuroSim – The Prototype of a Neurosurgical Training Simulator

We present NeuroSim, a VR-based simulator, that uses original instruments and a real surgical microscope. The ﬁrst training module features an abstract task in order to train basic skills. The software design is modular and based on training modules, so further tasks like tumor resection or aneurysm clipping can be added.

1. Methods While developing NeuroSim our main focus was to combine a realistic interface with an immersive real-time simulation. Our setup consists of a phantom of the head, original instruments, a surgical microscope, several cameras and a standard personal computer (see ﬁgure 1). NeuroSim uses a modular software platform which includes a plugin structure and is easily extendable.

(a) Surgical microscope

(b) Optics carrier and phantom of the head

Figure 1. NeuroSim

1.1. Instrument Tracking The phantom of the head hosts an optical tracking system (see ﬁgure 2(a)) which consists of three CMOS cameras, several white LEDs and one FPGA (ﬁeld programmable gate array). Passive color markers are attached to the tip of the original instruments. The FPGA gathers and preprocesses the data from the cameras in order to reduce latency and the amount of data being transferred to the PC [Koepﬂe04]. Only one color per instrument is used, the reconstruction is done by a relational method described in [Koepﬂe07]. An inertial measurement unit connected via USB and consisting of three accelerometers and three gyroscopes is tied to the instruments (see ﬁgure 2(b)) in order to estimate their orientation and gather data that can be used to stabilize the optical tracking. Sensor fusion combines the data from the optical tracking, the gyroscopes and the accelerometers in order to determine the position and orientation of the instruments and to ﬁlter glitches. Future work will include a more sophisticated sensor fusion that uses the inertial measurement unit to make the tracking more robust in cases of occluded markers. In addition, more instruments such as a needle holder or scissors will be integrated in the system.

F. Beier et al. / NeuroSim – The Prototype of a Neurosurgical Training Simulator

(a) Tracking system

53

(b) Instrument with inertial measurement unit

Figure 2. Instrument Tracking

1.2. Surgical Microscope Almost all surgical interventions on the human brain require microsurgical skills and are performed with a neurosurgical microscope that can be freely positioned above the operating ﬁeld. Position and orientation of the microscope as well as the state of the pistol grip buttons like zoom or focus have to be determined. NeuroSim uses the mechanical and electrical part of a real surgical microscope to provide a native moveable interface. A tracking system, mounted on the microscope (see ﬁgure 3(a)), is used to track active infrared markers that are integrated in the phantom of the head (see ﬁgure 3(b) and 3(c)). The inside-out tracking takes advantage of the fact that the optical axis of the microscope is always positioned in such a way that the camera system points towards the phantom. The use of infrared markers reduces the negative inﬂuence of changing light environments and guarantees a stable tracking. Each pistol grip includes a joystick that controls the precise movement of the microscope on two axes. As the tracking system is directly attached to the head of the microscope, its movement is already included in the tracking process. The optical oculars are substituted by a stereo display (see ﬁgure 3(d)) in which the computer generated scene is shown in 3D. All devices mentioned can be added to an original surgical microscope, so that costs for a future product can be reduced. Buttons like focus and zoom will be readout via a CAN-bus interface in a future process. 1.3. Model generation The models used in NeuroSim are generated from MRT- or CT-images. The generation is done in three steps: First the raw images are segmented, then a surface model is extracted and, in a last step, the surface is used to generate a tetrahedron mesh. The ﬁrst abstract training module uses a part of the brain as a background tissue that can be deformed by interacting with the instruments. For the medical training modules that will be implemented next, more complex models of the brain are generated from different datasets. For the generation of vessels, CTangiography datasets will be used. As a result, many different but still realistic sets of models will be available.

54

F. Beier et al. / NeuroSim – The Prototype of a Neurosurgical Training Simulator

(a) Trackingsystem

(b) Inside of the phantom

(c) Infrared LED marker

(d) Stereo display

Figure 3. Microscope tracking and setup

1.4. Simulation Real-time tissue modelling is based on a high-performance and reusable framework developed within the ViPA group which was presented in [Grimm05]. The framework is currently being developed in cooperation with the VRmagic GmbH and the ViPA group. The simulation used in the ﬁrst training module is based on an approach presented by [Teschner04] which has been modiﬁed in order to support real-time cutting of tetrahedrons and can be accelerated using GPUs. 1.5. Simulator Framework The simulator is based on a modular software framework developed within the ViPA group. It allows rapid prototyping of medical simulators by using a plugin based architecture. Highly reusable plugins form the basis of the framework and can be shared across different simulators. The plugin themselves are decoupled via an abstract interface layer. Communication is done via message-passing, so single components like input interfaces (e.g. tracking device) can easily be swapped or simulated by other devices (e.g. keyboard). Persistence and record/replay functionality can be included in the frame-

F. Beier et al. / NeuroSim – The Prototype of a Neurosurgical Training Simulator

55

work. The VR itself uses a similarly modular but more lightweight approach called component based entity system, where entities in the VR are aggregated from components. This approach offers highly reusable components and allows an object in the VR to be constructed via a graphical editor or simple text ﬁles.

2. Results By putting all the components described above together, the prototype of a neurosurgical simulator was created. The ﬁrst training module consists of a rigid-body-simulation of several small spheres. These spheres have to be broached with the instruments. If the tip of the instrument does not touch the sphere perpendicular to its surface, the sphere slides away and the instrument does not enter. If the position of the tip inside the sphere is near the center, the color of the sphere turns slowly from red to green (see ﬁgure 4). Some of the spheres are positioned behind the skull, outside the volume that is initially visible. In order to see all spheres through the microscope, the microscope has to be repositioned during the procedure. Although the task is quite abstract, it meets several demands: ﬁrst, the trainee has to get familiar with the positioning of the surgical microscope. He or she has to navigate it in a way so that all spheres are visible. Second, the indirect and steady handling of the instruments is trained.

Figure 4. Abstract training module

3. Conclusions We presented the prototype of a neurosurgical training simulator. Through the combination of original instruments and a real surgical microscope, NeuroSim is able to create an immersive environment. Thus we were able to perform abstract tasks. By doing that, several basic skills that are the fundament of a successful surgery can be trained. Current development includes training modules focusing on medical content like the suturing of two blood vessels and a more complex sensor fusion for the instrument tracking. Due to the modular platform design more training modules can be added easily. It is planned to add modules for tumor resection and aneurysm clipping. Furthermore, brain models will be generated from real datasets in order to build up a case database. Finally, an objective evaluation will be integrated.

56

F. Beier et al. / NeuroSim – The Prototype of a Neurosurgical Training Simulator

Acknowledgements This work is kindly supported by Leica Microsystems2 , sponsor of the neurosurgical microsocpe, and VRmagic GmbH3 .

References [DeMauro08] A. De Mauro, J. Raczkowsky, R. Wirtz, H. Wörn. Development of a Microscope Embedded Training System for Neurosurgery, Lecture Notes in Computer Science, Volume 5104, 2008 [Grimm05] J. Grimm. Interaktive Echtzeitmodellierung von biologischem Gewebe für Virtuelle Realitäten in der medizinischen Ausbildung, PhD thesis, University of Mannheim, Department for Mathematics and Computer Science, 2005. [Koepﬂe04] A. Köpﬂe, M. Schill, M. Rautmann, M. Schwarz, A. Pott, A. Wagner, R. Männer, E. Badreddin, P. Weiser, H. P. Scharf. Occlusion-Robust, Low-Latency Optical Tracking using a Modular Scalable System Archituecture, Medical Robotics, Navigation & Visualization MRNV, Remagen, Germany, March 2004. [Koepﬂe07] A. Köpﬂe, F. Beier, C. Wagner, R. Männer. Real-time Marker-based Tracking of a Non-rigid Object, Stud Health Technol Inform 125 (2007), 232-234, Published by IOS Press. [NeuroTouch] http://www.nrc-cnrc.gc.ca/eng/dimensions/issue2/virtual_surgery.html [Teschner04] M. Teschner, B. Heidelberger, M. Mueller, M. Gross. A Versatile and Robust Model for Geometrically Complex Deformable Solids, Proc. Computer Graphics International CGI’04, Crete, Greece, pp. 312-319, June 16-19, 2004.

2 http://www.leica-microsystems.com 3 http://www.vrmagic.com

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-57

57

Low-Cost, Take-Home, Beating Heart Simulator for Health-Care Education Devin R. BERG a,1, Andrew CARLSON a, William K. DURFEE a, Robert M. SWEET b, and Troy REIHSEN b a Department of Mechanical Engineering, University of Minnesota, Minneapolis, MN b Department of Urologic Surgery, University of Minnesota, Minneapolis, MN

Abstract. Intended for medical students studying the evaluation and diagnosis of heart arrhythmias, the beating heart arrhythmia simulator combines visual, auditory, and tactile stimuli to enhance the student's retention of the subtle differences between various conditions of the heart necessary for diagnosis. Unlike existing heart arrhythmia simulators, our simulator is low cost and easily deployable in the classroom setting. A design consisting of solenoid actuators, a silicon heart model, and a graphical user interface has been developed and prototyped. Future design development and conceptual validation is necessary prior to deployment. Keywords. simulator, cardiac, arrhythmia, education

Introduction Memory retention can be enhanced by providing multiple forms of sensory input, such as auditory, visual, and tactile data [1]. Using this premise, we set out to design a heart simulator that supplements didactic and textbook material for beginning or advanced students in the medical field and enable them to cater the learning experience to their own personal learning style at home. Additionally, the kit should have a total part cost limited to approximately $150.00 in order to facilitate widespread classroom distribution. Prior art in the area of cardiac simulators includes computer models, simulated electrocardiogram (EKG) graphs, and physical beating heart simulators [2,3,4,5]. Much of the previous work focuses on one aspect such as having an advanced computer model. However, this approach lacks a comprehensive learning environment for the user. Other simulators offer a comprehensive learning environment but are too expensive for the classroom. The present design provides the benefits of a comprehensive learning environment through multiple forms of stimuli while maintaining a price level that makes the unit readily available for the classroom setting.

1

Corresponding Author: Department of Mechanical Engineering, University of Minnesota, 111 Church St. SE, Minneapolis, MN 55455, USA; Email: [email protected]

58

D.R. Berg et al. / Low-Cost, Take-Home, Beating Heart Simulator for Health-Care Education

Figure 1. USB connected, beating heart simulator.

1. Methods In pursuit of designing a suitable heart arrhythmia simulator, several possible configurations were conceptualized and prototyped. Initial prototyping included investigating the use of servo motors as a drive mechanism and writing rudimentary control code for use on a microprocessor. Ultimately, the use of servo motors was abandoned in favor of solenoid actuators to manipulate the simulator due to high noise levels and low power capacity. Additionally, a control algorithm and coupled graphical user interface (GUI) was developed to both control the solenoid actuation and provide the user with enhanced feedback and control. After cycling through several design iterations, a final design was reached that satisfied the design requirements and the budget constraints.

2. Results The beating heart simulator prototype was designed to provide the user with an immersive environment for learning the variations between several heart arrhythmias. The device incorporates computer software to provide the visual and auditory feedback to the user while simultaneous tactile information is fed to the user through a USB connected heart model (Figure 1). The tactile portion of the device uses pull-style linear solenoid actuators to provide conversion between input electrical signal from the printed circuit board (PCB) and output translational motion at the heart model. The prototype uses four solenoid actuators, one for each chamber of the heart model. When the solenoid actuator is activated by an electrical signal from the PCB, the solenoid core retracts into the stationary coil thus applying tension to the rigidly connected wire. The wire applies a force to the heart model, through an anchor, which produces a displacement in the model wall.

D.R. Berg et al. / Low-Cost, Take-Home, Beating Heart Simulator for Health-Care Education

59

Figure 2. Main program screen.

The computer software, written in Visual C#, presents the user with a welcome screen when first opened to make sure the USB kit is connected and communicating properly. Once the program has started, the main user interface is shown (Figure 2). From here the user can select from a variety of heart conditions which will automatically start the EKG script that reads a pre-recorded data file approximately 30 seconds in length [6]. The script will continue to play on loop until the user selects a new option or quits the program.

3. Conclusions Further development is needed in several areas. Continued enhancement of the GUI to include additional information about each heart condition could be incorporated to enhance the user experience. Additionally, a more sophisticated EKG analysis algorithm could provide more complete data to the end user. Beyond this, the simulator should undergo comprehensive validation studies to ensure that it meets the intended design goal of being a useful educational tool for health-care students. In the long-term outlook, the simulator could potentially be marketed as a learning tool for K-12 schools as well as hospitals or clinics as a means of enhancing patient comprehension.

References [1] D.S. Ruchkin, J. Grafman, K. Cameron, R.S. Berndt, Working memory retention systems: A state of activated long-term memory, Behavioral and Brain Sciences 26 (2003), 709–728. [2] R.L. Stanbridge, D. O’Regan, A. Cherian, R. Ramanan, Use of a pulsatile beating heart model for training surgeons in beating heart surgery, Heart Surgery Forum 2 (1999), 300–304. [3] M. Nakao, H. Oyama, M. Komori, T. Matsuda, G. Sakaguchi, M. Komeda, T. Takahashi, Haptic reproduction and interactive visualization of a beating heart for cardiovascular surgery simulation, International Journal of Medical Informatics 68 (2002), 155–163. [4] L. Xia, M.M. Huo, X. Zhang, Q. Wei, F. Liu, Beating heart modeling and simulation, In Proceedings of the Computers in Cardiology Conference (2004), 137-140. [5] The Chamberlain Group, Beating heart with great vessels, . [6] Goldberger A.L., Amaral L.A.N., Glass L., Hausdorff J.M., Ivanov P.Ch., Mark R.G., Mietus J.E., Moody G.B., Peng C.K., Stanley H.E. PhysioBank, PhysioToolkit, and PhysioNet: Components of a New Research Resource for Complex Physiologic Signals. Circulation 101(2000), e215-e220.

60

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-60

An Adaptive Signal-Processing Approach to Online Adaptive Tutoring Bryan BERGERON, MDa,1 and Andrew CLINE, MPH b a Accella Learning LLC b Director of Protective Medicine, UofL Dept of Emergency Medicine

Abstract. Conventional intelligent or adaptive tutoring online systems rely on domain-specific models of learner behavior based on rules, deep domain knowledge, and other resource-intensive methods. We have developed and studied a domain-independent methodology of adaptive tutoring based on domainindependent signal-processing approaches that obviate the need for the construction of explicit expert and student models. A key advantage of our method over conventional approaches is a lower barrier to entry for educators who want to develop adaptive online learning materials. Keywords. Intelligent tutoring systems, adaptive signal processing, ECRTM, electronic competency recordTM

Introduction At a time of national emergency from natural or manmade events, civilian and military first responders must train or, minimally, actively review the knowledge and skills relevant to the events. Moreover, because time is limited, this training must be as efficient as possible. In addition, because printed materials and experts are typically unavailable in emergencies, online training is a natural mode of learning in these situations. In online training, the most time-efficient approaches to education are adaptive, meaning that the content is matched, in real-time, to the learner’s demonstrated level of content mastery. In contrast, a one-size-fits-all approach may be appropriate for the average learner, but may be excessively tedious and time consuming for the expert, and yet may not provide enough of the fundamentals for learners at the other end of the spectrum. However, developing adaptive online courses is extremely resource intensive, in part because detailed, domain-specific models of the learner and the expert teacher must be developed. For example, an adaptive course in physics requires models of the learner and of a physics teacher. The student or learner model typically contains definitions of stereotypic misconceptions or mistakes commonly made of a learner. Similarly, the expert model is typically configured in a way that enables the adaptive system to respond in a stereotypic manner. That is, if a learner confuses mass and

1

Corresponding Author: Bryan Bergeron, MD, 258 Harvard Street #315, Brookline MA 02446; Email: [email protected].

B. Bergeron and A. Cline / An Adaptive Signal-Processing Approach to Online Adaptive Tutoring

61

weight, then the adaptive system might present illustrations to differentiate the two concepts. The rule base or other means of representing experts and learners is much more involved in complex adaptive tutoring situations, such as learning to operate a complex piece of equipment or in learning a new surgical skill. Often, there is no established rule base or other validated basis for determining whether a particular error on the learner’s part is due to a misconception, eye-hand-coordination problem, or other issue. As such, conventional adaptive tutoring would be impractical in theses situations. To address these limitations, we set out to develop domain-independent methods of adaptive tutoring, including the incorporation of adaptive serious games and ties to instrumentation through augmented reality approaches [1, 2]. As part of our research, we conducted several studies on the efficacy of our approach, including a major study at the University of Louisville’s Division of Protective Medicine, summarized here.

1. Methods & Materials To test the hypothesis related to efficacy, efficiency, and positive learning outcomes, the University of Louisville recruited study participants to take one of the three course curriculums. The design included: •

Recruitment of Medical, Business, First Responder and Collegiate Student populations for testing.

•

Participants ranged in age from 21-65.

•

The target number of subjects for each of six groups was 15 or greater.

•

Messages were sent to 90 potential study participants requesting their participation. The 90 participants were previous students in the University of Louisville’s Division of Protective Medicine who showed interest in furthering their knowledge.

•

The 90 possible participants that were contacted were randomly assigned one of the three courses to take. Therefore, 30 participants per course were assigned.

•

Each participant was provided a unique login and password to the system.

•

Participants were only identified in post testing by these unique login names and reporting only categorized the students in one of four areas of profession: Student, Business, First Responder, or Medical.

•

Randomly, some students were chosen to take more than one course, while others were assigned to complete only one course.

•

Participants were contacted and given instructions to complete testing within 5 days of the initial notice. After the 5 days passed, a follow-up reminder was provided and testing ended within the following 5 days.

62

B. Bergeron and A. Cline / An Adaptive Signal-Processing Approach to Online Adaptive Tutoring

2. Results Analysis of the results of the six groups (N=15 per group) of learners conducted by the University of Louisville’s Division of Protective Medicine showed significantly shorter interaction times as well as equivalent to superior retention of information imparted through adaptive tutoring versus standard online training. Retention was measured at 6 weeks after the initial training. Analysis of the type of questions asked and how the responder answered the questions revealed that our adaptive tutoring system utilized the student’s answers to decide on the difficultly level of the material presented. When learners demonstrated knowledge of a subject, they were presented with more difficult/advanced content. We also showed that our adaptive system ends the student’s session earlier with fewer total questions being asked when compared to a learner that is less knowledgeable with the material. Conversely, learners that demonstrated shallow understanding of the subject matter were presented with content a lower difficulty level. Based on our adaptive systems ability to respond to each particular learner’s needs with content suitable to their specific level of knowledge, the adaptive system was shown to have a positive impact on knowledge transfer and learning as hypothesized.

3. Discussion Data from the three courses administered at the University of Louisville and the teaching staff support our hypothesis. Not only were the instructors pleased with the quality of instruction, but tracking files clearly demonstrated that the nature of the content presented to learners was dynamically modulated during each course, as defined by our adaptive system. Moreover, we demonstrated that a general-purpose adaptive system (as opposed to a domain-limited system) could be developed and used for practical teaching applications. Results of our work with adaptive signal processing as the basis for adaptive online tutoring have been positive. In this report, we focus on the key enabling technology or our approach, adaptive signal processing. Unlike fixed statistical methods, properly configured adaptive signal processing algorithms adjust to suit the demands of the changing environment. Whereas a fixed system degrades with change, the performance of a properly configured adaptive system improves through interaction with a changing environment. When applied to the signal produced by tracking learner interaction with an online tutoring system, adaptive signal processing can provide the basis for determining whether the learner should be presented with more or less challenging materials, or given more or less time. One of the advantages of our approach is the wide variety of well-documented analog and digital signal processing algorithms available. Off-the-shelf applications, including the digital signal processing algorithm libraries for MathWorks MatLab and National Instruments LabVIEW enabled us to experiment with a variety of signal processing approaches. In particular, we found predictive adaptive filters to be particularly applicable to adaptive tutoring. One of the characteristics of adaptive predictive filters is that, when properyly configured, they are relatively impervious to noise [3]. That is, they can get at the underlying signal – in our case, the level of mastery of the learner in a course – with relatively few data points. This is an important distinction in online learning situations

B. Bergeron and A. Cline / An Adaptive Signal-Processing Approach to Online Adaptive Tutoring

63

in which decisions on what content to present must be based on perhaps two or at most three learner interactions with the system. Another advantage of using established adaptive filter technology is that their behavior is well documented, and the behavior is based on visible, definable parameters. This contrasts with a competing technology, neural networks, which can be used to recognize signals in a noisy environment. In addition to having hidden variables that can’t be examined explicitly, most neural networks require extensive training on exemplar data sets in order to be of value. The challenge in using adaptive signal processing as the basis for adaptive tutoring is in the initial configuration of the adaptive filter. As in adaptive filters used in communications, the filter constants should reflect the expected signal environment. That is, the filter constants should reflect the expected signal-to-noise ratio as well as the information content of the overall signal. A course with a handful of images and the same number of questions has low information content and requires the filter to adapt quickly to the learner’s performance level. As a result, the filter constants could result in wild oscillations about the true value of the learner’s performance. In contrast, a course with a hundred or more potential screens of information and that results in dozens of data points from learner interactions has a much higher information content. The adaptive filter constant values can be more conservative because there is more time to lock onto the learner’s level of performance. There are additional components to our adaptive signal processing approach not described in detail here. In particular, our approach makes use of an electronic competency recordTM (ECRTM) that serves as a long-term memory of learner experiences and encounters that could affect their competency in a variety of areas. These data are not limited to textual responses to online tests and content, but include motion sense and outcomes data, such as that generated by a learner manipulating surgical instruments, a mannequin, or field equipment. The ECRTM enables our adaptive algorithms to start with the most appropriate filter constants, even for brief interactions involving few additional data points. In addition, by enabling the values of the competency data to change as a function of time and other variables, the ECRTM provides a model of the learner that can be used by a variety of ancillary applications, from recertification of individuals to planning continuing medical education curricula for populations of learners. To demonstrate the commercial viability of our patent-pending, domainindependent approach to online adaptive tutoring, we are in the process of establishing a National Online Training Center through the University of Louisville’s Division of Protective Medicine. The center will enable military personnel, health care providers, and first responders to access online training content beneficial to their specific mission and/or continuing education needs.

Acknowledgement The work described here, as well as the establishment of a National Online Training Center, is a result of an ongoing Phase III grant Standards-based Intelligent Tutoring, Adaptive Nuclear, a Phase II grant, Biological & Chemical Multimedia Collaboratory, and a Phase I grant, Knowledge Warehouse Infrastructure for Standards-Based Intelligent Tutoring from the USAMRMC and administered by TATRC.

64

B. Bergeron and A. Cline / An Adaptive Signal-Processing Approach to Online Adaptive Tutoring

References [1] B.P. Bergeron, Augmented assessment for augmented reality, Stud Health Technol Inform 119 (2005), 49-51. Published by IOS Press. [2] B.P. Bergeron, Learning and retention in adaptive serious games, Stud Health Technol Inform 132 (2008), 26-30. Published by IOS Press. [3] P. Zarchan and H. Musoff, Fundamentals of Kalman filtering, AIAA, Virginia, 2004.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-65

65

Comparison of a Disposable Bougie versus a Newly Designed Malleable Bougie in the Intubation of a Difficult Manikin Airway Ben H. BOEDEKER, MDa,b,1, Mary BERNHAGEN, BSa, David J. MILLER, PhDa, and W. Bosseau MURRAY, MDc a Dept. of Anesthesiology, University of Nebraska Medical Center, Omaha, NE b Research Service, Omaha VA Medical Center, Omaha, NE c Clinical Simulation Center, Penn State College of Medicine, Hershey PA

Abstract. The endotracheal bougie is used for difficult intubations when only a minimal glottic view is obtained. Standard bougies are designed for use during direct, line-of-sight viewing of the glottic opening. With videolaryngoscopy, intubators “see around the corner”, thus requiring a bougie which can be shaped to follow a significant curve. A malleable bougie with an imbedded internal wire was created to enable intubators to shape the curve to best fit a difficult airway. This pilot study examined the malleable bougie compared to the SunMed™ bougie in a simulated difficult airway intubation using video laryngoscopy. Keywords. Bougie, intubation, video laryngoscope

Background Intubation bougies are frequently used to facilitate tracheal intubation when the view of the glottic opening is restricted. The bougie is easier to insert through the glottis than an endotracheal tube [1-5] as it is thinner and obscures the view less; it should be able to lift up the epiglottis; and the tip curls up behind the epiglottis. When using a traditional bougie as an aide to intubation, the working end often does not retain the shape necessary to successfully intubate a difficult airway. To eliminate the difficulty, a novel bougie was developed with a 2.87 mm diameter malleable filament that allows the user to shape and retain a curve to aid in intubation. The filament is built into 12 inches of the total 30 inch-long bougie. This extra length (compared to the SunMed™ bougie) enables the user to intubate without retracting the bougie from the trachea. This investigation compares the SunMed™ and malleable bougies in a simulated difficult airway intubation. Tools and Methods Following IRB approval, a Laerdal Difficult Airway TrainerTM (Laerdal Medical Corporation, Gatesville, TX) was set to the difficult airway setting by tongue inflation. 1 Corresponding Author: Ben H. Boedeker, MD, Professor of Anesthesiology, University of Nebraska Medical Center, Director, Center for Advanced Technology & Telemedicine, 984455 Nebraska Medical Center, Omaha, Nebraska 68198-4455, U.S.A.; E-mail: [email protected]

66

B.H. Boedeker et al. / A Disposable Bougie Versus a Newly Designed Malleable Bougie

Experienced laryngoscopists from UNMC were timed while performing a video laryngoscopy and inserting the SunMed™ (15Fr x 70cm Coude tip, Sun Med, Largo, FL) or malleable bougie through the glottic opening. The video laryngoscope used was the Storz C-MACTM (Karl Storz Endoscopy, Tuttlingen, Germany) fitted with a number 3 video MacIntosh blade. Selection of malleable or SunMed™ bougie was by random order. Non-parametric data (e.g. categorical data such as the success/non-success of intubation and airway view grading [1&2 = good view vs. 3&4= poor view]) was analyzed with a two-tailed Fisher’s Exact Probability test. An unpaired t-test was used to analyze differences in the average intubation times. Values are reported as “mean ± standard deviation.” A value of p < 0.05 was considered significant.

Results The 21 study participants included staff anesthesiologists, residents and CRNAs. In difficult airway intubation with the SunMed™ bougie, the average recorded CormackLehane (CL) airway grade was 2.67 ± 0.57 (n=21). With the malleable bougie, the average CL grade was 2.60 ± 0.59 (n=20) (p=1.00; no significant difference). The average intubation time was 35.6 ± 22.99 seconds (n=16) with the SunMed™ bougie; and 34.6 ± 26.39 seconds (malleable) (n=20). The success rates for intubation was 76.2% (SunMed™ bougie) (n=16) and 95.2% (malleable) (n=20). The difference in the success rates of intubation in respect to the different bougies is interesting; however, due to the small number of study subjects, this difference is not statistically significant (p = 0.18).

Discussion With the advent of video laryngoscopy, it is possible to view a wider field than can be seen by direct eyesight (Figure 1).

Figure 1. Visualization of the airway using the videolaryngoscope

Standard bougies are designed to function in a straight line of sight or, with a preplaced curve, be inserted behind an epiglottis which is obscuring the view of the vocal

B.H. Boedeker et al. / A Disposable Bougie Versus a Newly Designed Malleable Bougie

67

cords. Standard bougies may not hold a curve formed by the laryngoscopist. Such a shape might be needed to accomplish insertion into the glottic opening which may involve a significant curve. The standard bougie typically does not hold a curve, as it slowly unbends (straightens). This lack of curvature may make it especially difficult to follow the desired path to achieve an intubation [6]. The Boedeker Bougie (Figure 2) has a stiffening wire inserted in the distal half of its length. This allows the device to be shaped and hold any curve to which it is bent by the laryngoscopist. The flexible proximal shaft facilitates placement of the endotracheal tube by the operator and facilitates its dual use as a tube exchanger.

Figure 2. The malleable Boedeker Bougie shaped to the curvature of the laryngoscope blade.

Conclusions Participants using the Boedeker (malleable) bougie had a higher success rate in intubation of the difficult manikin compared to the SunMed™ bougie. When performing video laryngoscopy, the angle of bend required in the bougie is usually greater than the angle required for direct laryngoscopy. Attaining and maintaining the required angle of bend for intubation was shown by this pilot study to improve the success rate of intubation when using the Boedeker (malleable) bougie. The authors conclude that a larger trial is needed to determine if this difference in success rates is significant and to further evaluate the utility of the Boedeker Bougie in the clinical setting.

References [1]

[2] [3] [4] [5] [6]

R. Komatsu, K. Kamata, I. Hoshi, D. Sessler, M. Ozaki. Airway Scope and gum elastic bougie with Macintosch laryngoscope for tracheal intubation in patients with simulated restricted neck mobility. Br J Anaesthesia 101 (2008), 863-869. WTA Fox, S. Harris, N.J. Kennedy. Prevalence of difficult intubation in a bariatric population using the beach chair position. Anaesthesia 63 (2008), 1339-1342. X. Combes, M. Dumerat, G. Dhonneur. Emergency gum elastic bougie-assisted tracheal intubation in four patients with upper airway distortion. Can J Anesth 51 (2004),1022-1024. M.K, Arora, K. Karamchandani, A. Trikha. Use of a gum elastic bougie to facilitate blind nasotracheal intubation in children: a series of three cases. Anaesthesia 61 (2006), 291-294. J. Rich. Successful blind digital intubation with bougie introducer in a patient with an unexpected difficult airway. Proc Bayl Univ Med Center) 21 (2008), 397-399. L. Jayaraman, N. Sethi, A. Kumar, J. Sood. Modification of the gum elastic bougie. Anesthesia & Analgesia 103 (2006), 1336-1337.

68

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-68

Improving Fiberoptic Intubation with a Novel Tongue Retraction Device Ben H. BOEDEKER, MDa,b,1 Mary BERNHAGEN, BSa, David J. MILLER, PhDa, Thomas A. NICHOLAS IV, MDa, Andrew LINNAUS, BSa, and W.B. MURRAY, MDc a Dept. of Anesthesiology, University of Nebraska Medical Center, Omaha, NE b Research Service, Omaha VA Medical Center, Omaha, NE c Clinical Simulation Center, Penn State College of Medicine, Hershey, PA

Abstract. This study examined the utility of a novel tongue retractor created with a wider working blade and a more ergonomic curve to provide jaw lift and tongue management with one hand during intubation. Anesthesia providers participated in simulated intubation of a difficult manikin using the novel tongue retractor with the Bonfils video fiberscope. Results show that the tongue retractor improved placement success and was well received by the study participants. Keywords. Fiberscope, intubation, tongue retractor

Background Prior studies have demonstrated the usefulness of the Bonfils intubating fiberscope during routine as well as difficult intubation [1-2]. When performing a rigid or flexible fiberoptic intubation, however, several individuals are needed: one to provide jaw lift, one to manage the patient’s tongue and one to maneuver the fiberscope and endotracheal tube. Our research team has developed a novel tongue retractor with a wider working blade and a more ergonomic curve. This pilot study describes the efficacy of a novel tongue retractor (TR) designed to provide jaw lift and tongue management with one hand and its use with the Bonfils video fiberscope in difficult airway intubations.

Methods & Materials Following institutional ethical approval, 22 anesthesia providers (anesthesia attending physicians, residents and Certified Registered Nurse Anesthetists at the University of Nebraska Medical Center and Omaha VAMC, Omaha, NE) participated in simulated intubation of a difficult manikin airway. The participants first completed a preexperience questionnaire defining their level of training and specialty and assessing any prior experience with fiberoptic intubation. During the trial, the tongue on a Laerdal Difficult Airway Trainer (Laerdal Medical Corp, Gatesville, TX) was inflated to

1

Corresponding Author: Ben H. Boedeker, MD, PhD, Professor, Anesthesiology, University of Nebraska Medical Center, Director, Center for Advanced Technology& Telemedicine, 984455 Nebraska Medical Center, Omaha, NE 68198-4455, U.S.A.: E-mail: [email protected]

B.H. Boedeker et al. / Improving Fiberoptic Intubation with a Novel Tongue Retraction Device

69

simulate a difficult airway. The anesthesia providers performed timed intubations using the Bonfils video fiberscope (Karl Storz Endoscopy, Tuttlingen, Germany) with and without the novel tongue retractor (Figure 1).

Figure 1. CAD model of novel tongue retractor.

Selection of the Bonfils fiberscope plus/minus the tongue retractor was by random order. Data collected included the time to intubation, the Cormack and Lehane (CL) airway grade, the number of intubation attempts, the success or failure of the intubation and whether or not cricoid pressure was requested by the intubator. Following their intubation experiences, the providers completed a questionnaire qualifying their experience with, and value of, the novel tongue blade retractor, and whether or not they would use the novel tongue retractor for other procedures. Statistics-a paired t-test was used to calculate the comparison of the times of intubation. A Fisher’s Exact Test was used to calculate the significance of success/failure rate and rate of cricoid pressure requests. Because of the small sample size in this pilot study, an adjusted proportion and standard deviation were used to calculate the remaining population statistics. The values are reported as “mean ± standard deviation.” A p-value < 0.05 was considered significant.

Results Participants in this pilot study consisted of one Student Registered Nurse Anesthetist, 5 Certified Registered Nurse Anesthetists and 16 MDs, all of whom were anesthesia practitioners. Their levels of fiberoptic intubation experience ranged from no intubations to 50 or more. (One subject had no experience; six had experience in < 20 awake intubations; fifteen had experience in 20 or more awake intubations. Figure 2 shows the percent success of intubation rates and requests for cricoid pressure.

70

B.H. Boedeker et al. / Improving Fiberoptic Intubation with a Novel Tongue Retraction Device

Figure 2. Success of intubation rates and rate of requests for cricoid pressure during intubation attempts

Table 1 shows the average CL airway scores, number of intubation attempts and median times to intubation. Results from the questionnaire reveal that 68% (13/19) of the respondents indicated that the tongue blade assisted in the intubation with the Bonfils; 93% (13/14) agreed that they would benefit from having a tongue blade for other procedure types as well. This difference is significant (p < 0.03) when compared to a random result of 50%. Table 1. Average airway scores, number of intubation attempts and median times to intubation using the Bonfils Fiberscope with or without the novel tongue retractor Device Bonfils Bonfils w/ blade

Average Airway Score 1.67 ± 1.02

# of intubation attempts 1.09 ± 0.29

1.45 ± 0.80

1.05 ± 0.21

Median intubation time 20 sec (range 8-40) n=15 22 sec (range 5-88) n=19

Discussion The small sample size created a large standard deviation among the times to intubation. This is likely due to the varied experience of the operators with both the instruments and simulator. With this data, we can conclude that the tongue retractor appears to improve or maintain the quality of an intubation attempt (in respect to airway score, cricoid pressure requirement, intubation time, number of attempts, and placement success). The data indicate that the combination of the fiberscope and retractor offer a superior intubation experience to currently available best practices. The tongue retractor was well received and most of the participants indicated that they would welcome this device should it become clinically available in the future.

References [1] [2]

B. Bein, M. Yan, P. Tonner, J. Scholz, M. Steinfath, V. Dorges. Tracheal intubation using the Bonfils intubation fibrescope after failed direct laryngoscopy. Anaesthesia 59 (2004), 1207-1209. M. Halligan P. Charters. A clinical evaluation of the Bonfils Intubation Fibrescope. Anaesthesia 58 (2003), 1087-1091.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-71

71

Combined Intubation Training (Simulated and Human) for 4th Year Medical Students: The Center for Advanced Technology and Telemedicine Airway Training Program Ben H. BOEDEKER, MDa,b,1 , Mary BERNHAGEN, BSa, Thomas A. NICHOLAS IV, MDa , and W. Bosseau MURRAY, MDc a Dept. of Anesthesiology, University of Nebraska Medical Center, Omaha, NE b Research Service, Omaha VA Medical Center, Omaha, NE c Clinical Simulation Center, Pennsylvania State College of Medicine, Hershey, PA

Abstract. The video laryngoscope is a useful tool in intubation training as it allows both the trainer and the student to share the same view of the airway during the intubation process. In this study, the Center for Advanced Technology and Telemedicine’s airway training program employed videolaryngoscopy (VL) in teaching both simulated (manikin) and human intubation. The videolaryngoscope statistically improved the glottic view in both the standard and difficult manikin airways when compared to that with standard (direct) laryngoscopy. The success rate in simulated difficult airway intubation was significantly improved using VL. With human intubation training, there was statistically significant improvement in airway views using VL and a 97.5% intubation success rate. The enhanced view of the videolaryngoscope in airway intubation facilitates the learning process in performing both simulated and human intubation, making it a powerful tool in intubation training. Keywords. Intubation, video laryngoscopy, direct laryngoscopy

Background Prior studies have shown that video laryngoscopy (VL) is ideal for teaching orotracheal intubation, allowing the trainer and student to share the same airway view [1-2]. In our study (CATT airway training program), we attempted to demonstrate improved learning with a VL system in manikins as well as in humans.

Tools and Methods Following IRB approval, fourth year medical students (University of Nebraska School of Medicine) participated in basic intubation training - both simulated (n=56) and

1

Corresponding Author: Ben H. Boedeker, MD, Ph.D., Professor, Department of Anesthesiology, University of Nebraska Medical Center; Director, Center for Advanced Technology and Telemedicine, 984455 Nebraska Medical Center, Omaha, NE, 68198-4455, U.S.A.; E-mail: [email protected].

72

B.H. Boedeker et al. / Combined Intubation Training for 4th Year Medical Students

human (n=40). Students completed an online intubation course and test plus a pretraining questionnaire prior to the manikin and human intubation training. Manikin Intubation Training-A Laerdal Difficult Airway Trainer™ was intubated using the Storz Medi Pack Mobile Imaging System TM with a number 3 video MacIntosh blade. Students attempted 4 intubations: (1) on a standard airway with direct view (DV) of the glottis (direct laryngoscopy); (2) on a standard airway with an indirect view (videolaryngoscopy (VL); (3) on a difficult airway with DV; and (4) on a difficult airway with VL. The glottic view on a modified Cormack-Lehane (CL)[3] scoring system and time of intubation were recorded. Human Intubation Training- Under direct supervision, the students performed endotracheal intubation using VL. The CL view at the time of insertion of the endotracheal tube was noted by the student. Before removal of the laryngoscope and with no movement from its position, after insertion of the endotracheal tube, the CL direct view was also recorded. Nonparametric data (e.g. the classification of the view of the vocal cords and success/failure of the intubation) was analyzed with a two-tailed Fisher’s Exact Test (p < 0.05 was considered significant).

Results Manikin Intubation Training Performance Data- The average airway view grade recorded for the standard airway using DV was 2.62 ± 0.7 (Mean ± SD; n=56). Trainees performing VL on the standard airway manikin reported a mean CL score of 1.30 ± 0.6 (n=56). The indirect view gave a significantly greater proportion of Grade I & II views than the direct view in the standard airway model (P < 0.0001; comparison of grades 1& 2 vs. 3&4). The average CL grade for the difficult airway model with DV was 3.94 ± 0.2 which was significantly higher (worse view) (p < 0.0001) than the indirect view on the difficult airway model (average CL visual airway score 2.82 ± 0.610), leading to a significantly (p < 0.001) greater proportion (28.8 %) of Grade 1 & 2 views (VL) compared to the proportion of direct Grade 1 & 2 views in the difficult airway model (0 %).

Figure 1. Manikin intubation training: comparison of airway view scores in standard and difficult manikin airways using direct and video laryngoscopy

B.H. Boedeker et al. / Combined Intubation Training for 4th Year Medical Students

73

Human Intubation Performance Data-Trainees performed video laryngoscopy on patients with a 97.5% (39/40) success rate. The average CL view grade using VL was significantly lower (1.58 ± 0.68; n=40) than using DV (2.63 ± 0.75; n=38) (p < 0.0001).

Figure 2. Human intubation training: comparison of airway view scores in a human with direct and indirect visualization

Discussion and Conclusion The VL glottic views of a standard and difficult manikin airway were statistically improved as compared to direct laryngoscopy. A markedly improved intubation success rate of 67.8 % was achieved in the difficult manikin airway with VL versus 12.7% with direct laryngoscopy. In the human intubation training, improvement in the glottic views with VL from the direct view was statistically significant (Figure 1). Most importantly, we identified an intubation success rate of 97.5% (39/40) with the video laryngoscope. We recommend videolaryngoscopy be used as the preferred method of teaching intubation.

References [1] [2] [3]

B.H. Boedeker, B.W. Berg, M. Bernhagen, W.B. Murray. Direct versus indirect Laryngoscopic visualization in human endotracheal intubation: a tool for virtual anesthesia practice and teleanesthesiology. Stud Health Tech Inform 132 (2008), 31-36. Published by IOS Press. B.H. Boedeker, S. Hoffman, W.B. Murray. Endotracheal intubation training using virtual images: learning with the mobile telementoring intubating video laryngoscope. Stud Health Tech Inform 125 (2007), 49-54. Published by IOS Press. R.S. Cormack, J. Lehane. Difficult tracheal intubation in obstetrics. Anaesthesia 39 (1984), 1105-11.

74

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-74

Battlefield Tracheal Intubation Training Using Virtual Simulation: A Multi Center Operational Assessment of Video Laryngoscope Technology Ben H. BOEDEKER, MDa,b,1 , Kirsten A. BOEDEKERb, Mary A. BERNHAGENb, David J. MILLER, PhDa,b, and Lt. Col. Timothy LACY, MDc a University of Nebraska Medical Center, Omaha, NE b Omaha VA Medical Center, Omaha, NE c U.S. Air Force Medical Support Agency, Falls Church, VA

Abstract. Airway management is an essential skill in providing care in trauma situations. The video laryngoscope is a tool which offers improvement in teaching airway management skills and in managing airways of trauma patients on the far forward battlefield. An Operational Assessment (OA) of videolaryngoscope technology for medical training and airway management was conducted by the Center for Advanced Technology and Telemedicine (at the University of Nebraska Medical Center, Omaha, NE) for the US Air Force Modernization Command to validate this technology in the provision of Out of OR airway management and airway management training in military simulation centers. The value for both the training and performance of intubations was highly rated and the majority of respondents indicated interest in having a video laryngoscope in their facility. Keywords. Videolaryngoscope, laryngoscopy, intubation

Background Airway management is the cornerstone of medical support for the trauma patient. This is reflected in the fact that when resuscitating a patient using the “A,B,C’s of Trauma Management,” the first two sections of the algorithm (Airway and Breathing) relate to airway management. On the far forward battlefield, it is estimated that improved airway management could decrease deaths by 10% [1,2]. Video laryngoscopy technology offers a method to improve both the teaching of airway management and far forward battlefield airway care. An Operational Assessment (OA) of video laryngoscopy technology for medical training and airway management was conducted by the Center for Advanced Technology and Telemedicine (CATT) at the University of Nebraska Medical Center for the US Air Force Modernization Command. The purpose of the validation was to assess the perceived value of video laryngoscopy for teaching airway management in military simulation 1

Corresponding Author: Ben H. Boedeker, MD, PhD, Professor, Department of Anesthesiology, University of Nebraska Medical Center; Director, Center for Advanced Technology and Telemedicine, 984455 Nebraska Medical Center, Omaha, NE, 68198-4455; U.S.A.; E-mail:[email protected]

B.H. Boedeker et al. / Battleﬁeld Tracheal Intubation Training Using Virtual Simulation

75

centers and whether or not the technology could offer advantages for patient care in the out of OR airway management setting and far forward battlefield.

Methods Video laryngoscope systems were deployed at the following sites: – Medical Simulation Branch, Wilford Hall Air Force Medical Center, Lackland AFB, TX – Readiness Training Center, 55th Medical Group, Offutt AFB, Omaha, NE – Nebraska National Guard Training Center, Lincoln, NE The video laryngoscope used was the STORZ Medi Pack Mobile Imaging System TM (KARL STORZ Endoscopy-America, Inc., Culver City, CA) fitted with a #3 video MacIntosh blade. Onsite didactic intubation training was conducted to train medical personnel in airway management using a previously validated standardized training program [3,4,5]. Following this training, the participants completed a de-identified, anonymous post-training questionnaire designed to measure the perceived value of the video laryngoscope and the airway training program (Table 1). Participants were also asked for general comments on the device & the training. Table 1. Questions provided in post-training questionnaire. Post-Training Questions 1. How valuable is the video laryngoscope for intubations? Response: (1-10; 10 = highest) 2. How would you rate the teaching value of the video laryngoscope? Response: (1-10; 10 = highest) 3. Would you like to have this device for airway management at your deployed location? (Yes/No) 4. Would you like to have the video laryngoscope deployed at more locations in your hospital? (Yes/No) 5. Was the airway training valuable to you? (Yes/No) 6. How could this videolaryngoscope be improved? Comments/Suggestions

Results A total of 78 deploying medical personnel were trained in video laryngoscopy and questioned after the training. The resulting data are shown in Figure 1.

Figure 1. Histogram of responses to Questions #1, 2 & 5. n1 = 78, n2 = 77, n5 = 77.

76

B.H. Boedeker et al. / Battleﬁeld Tracheal Intubation Training Using Virtual Simulation

On average, the value of the laryngoscope for intubations was rated at 9.46 ± 1.10 while the teaching value was rated at 9.51 ± 1.06. All 78 respondents expressed interest in having a video laryngoscopy system in their facility and 76 wanted a video laryngoscope deployed at more locations in their hospital (2 respondents abstained). On average, the value of the training was rated by participants as 9.21 ± 1.32. In all cases, statistics are reported as “Mean ± SD.” When questioned as to how the video laryngoscope could be improved, answers were generally positive, but were comprised of a range of responses (not listed for sake of space). Similarly, comments were positive and indicated that the device would be life-saving and make training easier.

Discussion & Conclusions This OA demonstrated that the majority of students participating in intubation training with the video laryngoscope perceived it as having high value. This novel new technology offers a method to improve both teaching airway management and patient care. Incorporation of airway management with the video laryngoscope into Operational Readiness Medical Training may provide a venue to decrease deaths during trauma caused by loss of a patent airway. The results of this study suggest that video laryngoscopy technology should be deployed to all levels of military medical care where airway management may be necessary.

References [1] [2] [3] [4] [5]

J.F. Kelly, A.E. Ritenour, D.F. McLaughlin, K.A. Bagg, A.N. Apodaca, C.T. Mallak, et al. Injury severity and causes of death from Operation Iraqi Freedom and Operation Enduring Freedom: 20032004 versus 2006, J Trauma 64 (2008), S21–S27. R.T. Gerhardt, R.A. De Lorenzo, J. Oliver, J.B. Holcomb, J.A. Pfaff. Out-of-hospital combat casualty care in the current war in Iraq. Ann Emerg Med 53 (2009), 169-174. B.H. Boedeker, B.W. Berg, M. Bernhagen, W.B. Murray. Direct versus indirect laryngoscopic visualization in human endotracheal intubation: a tool for virtual anesthesia practice and teleanesthesiology. Stud Health Technol Inform 132 (2008), 31-36. Published by IOS Press. B.H. Boedeker, S. Hoffman, W.B. Murray. Endotracheal intubation using virtual images: learning with the mobile telementoring intubating video laryngoscope. Stud Health Technol Inform 125 (2007), 4954. Published by IOS Press. B. Boedeker, W.B. Murray. Basic review of endotracheal intubation for providers at a mass casualty. Journal of Education in Perioperative Medicine 10 (2008), 1-30.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-77

77

Intubation Success Rates and Perceived User Satisfaction Using the Video Laryngoscope to Train Deploying Far Forward Combat Medical Personnel Ben H. BOEDEKER, MDa,b,1, Mary A. BARAK-BERNHAGEN, BSa,b, Kirsten A. BOEDEKERa,b, and W. Bosseau MURRAY, MDc a Dept. of Anesthesiology, University of Nebraska Medical Center, Omaha, NE b Research Service, Omaha VA Medical Center, Omaha, NE c Clinical Simulation Center, Penn State College of Medicine, Hershey, PA

Abstract. Studies show the video laryngoscope enhances intubation training by facilitating visualization of airway anatomy. We examined the performance and training of military healthcare providers in a brief intubation training course which included both direct and indirect (video) laryngoscopy. This training format with the video laryngoscope improved airway visualization and intubation performance, promoting increased trainee confidence levels for successful intubation. Webbased training paired with hands-on instruction with the video laryngoscope should be considered as a model for military basic airway management training. Keywords. Videolaryngoscope, intubation training, indirect laryngoscopy

Introduction Expertise in basic airway management is essential for all medical first responders. In combat, attempts to secure the airway of a trauma patient may frequently end in failure [1]. It is important to instruct all individuals involved in emergency airway management, whether they function in a hospital environment or on the battlefield [2, 3]. Studies have shown that the video laryngoscope (VL) enhances intubation training by facilitating visualization of airway anatomy. We examined the performance and preference of military healthcare providers in intubation training using a web-based pre-training course followed by a hands-on training experience in simulated standard and difficult airways using the video laryngoscope.

Methods This IRB approved study was performed at the 710th Medical Squadron training room, Offutt Air Force Base, NE. The 22 subjects included medics, nurses, and physicians (no 1

Corresponding Author: Ben H. Boedeker, MD, Ph.D., Professor, Department of Anesthesiology, University of Nebraska Medical Center, Director, Center for Advanced Technology and Telemedicine, 984455 Nebraska Medical Center , Omaha, NE, 68198-4455, U.S.A.; E-mail: [email protected].

78

B.H. Boedeker et al. / Intubation Success Rates and Perceived User Satisfaction

anesthesiologists) being deployed to Afghanistan. For the training, we used a Laerdal Difficult Airway TrainerTM (Laerdal Medical Corporation, Gatesville, TX) and the Storz Medi Pack Mobile Imaging SystemTM (Karl Storz Endoscopy America, Commerce City, CA) with a #3 video MacIntosh blade. Prior to training, participants completed an online review of basic intubation and a questionnaire documenting their intubation experience and confidence level in first attempt intubation. Students were instructed on basic intubation technique using direct and indirect laryngoscopy and practiced several intubation attempts on the standard airway manikin. Trainees performed a series of 4 intubations: standard or direct laryngoscopy (DV) (viewing direct line of sight) and indirect laryngoscopy (VL) (viewing the screen while using the video laryngoscope) using both standard and difficult airway settings. Subjects recorded the glottic view using the Cormack Lehane (CL) scoring system at the time they inserted the endotracheal tube through the vocal cords [2, 4]. Success/non-success of the intubation attempt was recorded. Students completed posttraining evaluation forms to measure their perceived value of the course and posttraining confidence level in intubating a normal airway on first attempt. Data measured on categorical and ordinal scales (success/non-success of intubation and airway view grading [1&2 vs. 3&4]) were analyzed with a two-tailed Fisher’s Exact Probability test. P < 0.05 was considered statistically significant [2].

Results Eighty-two % of the subjects had experienced less than 11 intubations. The average perceived pre-training confidence level in first attempt intubation of a normal airway manikin = 5.0 (SD = 2.6; n=20). The confidence grading scale was 1-10; 1 = not confident and 10 = very confident. Participants reported an average CL view score of 2.8 (SD = 0.8) with DV and a significantly better average score of 1.4 (SD = 0.5) with VL in the standard manikin. The difference in number of participants obtaining airway view grades 1&2 vs. grades 3&4 with direct vs. indirect laryngoscopy was statistically significant (p=0.0006). The average CL score using DV in the difficult airway manikin was 3.9 (SD = 0.426) compared to a significantly better score of 1.8 (SD = 0.8) for VL (p < 0.0001). Significantly fewer trainees (73%) successfully intubated a standard airway manikin under DV while 100% intubated successfully with VL (p = 0.021; two-tailed Fisher’s Exact Test). In intubation of the difficult airway manikin, significantly fewer (9%) trainees were successful using DV compared to 100% of students being successful when using VL (p < 0.0001). Students had an average perceived pre-training confidence level of 5.0 (SD = 2.6; n = 20) and a significantly improved perceived post-training confidence level of 8.3 (SD = 1.4; n = 18) (p < 0.0001). Eighty-six percent of the students preferred the video laryngoscope for intubation of a difficult airway, whereas 14% of students preferred a standard laryngoscope. Out of 18 participants, 17 (94%) students reportedly chose the video laryngoscope as their preferred method of intubation; one (6%) student preferred the standard laryngoscope. The post-training workshop evaluation showed that 95% of participants agreed that the basic intubation hands-on manikin training was valuable; 100 % would recommend this course to other health care workers; 94% agreed that the web-based training was

B.H. Boedeker et al. / Intubation Success Rates and Perceived User Satisfaction

79

valuable; and 78% agreed that they would like to participate in future similarly structured training sessions.

Discussion Using advanced video technology to “see around the corner,” especially during difficult intubations, may make intubation easier to perform and to learn [5]. In this study, we found CL views improved by an average of 1.4 for the standard manikin and 2.1 for the difficult manikin. This is clinically relevant as an improvement of 1 or 2 CL scores should essentially convert a difficult airway to an easy airway to intubate. Trainees achieved an increased rate of successful intubation using VL in both the standard and difficult airway settings. When using the difficult airway, only 9% of subjects were able to successfully intubate with DV compared to a 100% success rate when using VL. Post training, students reported much higher confidence in believing they could successfully intubate a patient with the majority of students selecting the video laryngoscope as their preferential intubating system. Students expressed a strong desire for future training using a model similar to this training session.

Conclusion This investigation has shown that intubation training with the video laryngoscope significantly improved the view of the glottic opening, enhanced intubation performance, and increased trainee confidence in intubation. The video laryngoscope is an important tool for intubation training as it allows for both trainer and trainee to share the view of the airway anatomy. As the student manipulates the blade and endotracheal tube within the airway, the trainer is able to mentor the students’ technique and visually monitor the students’ performance. The use of the video laryngoscope in the training of lesser trained, deploying medical personnel, and having these devices as standard military issue, could improve airway management and save lives in combat trauma situations.

References [1] [2]

[3] [4] [5]

R. Ben Abraham, R. Yaalom, Y. Kluger, M. Stein, A. Weinbroum, G. Paret. Problematic intubation in soldiers: are there predisposing factors? Mil Med, 165 (2000), 111-113. B.H. Boedeker, S. Hoffman, W.B. Murray. Endotracheal intubation training using virtual images: learning with the mobile telementoring intubating video laryngoscope. Stud Health Technol Inform 125 (2007), 49-54. Published by IOS Press. R.R. Gaiser. Teaching airway management skills. How and what to learn and teach. Crit Care Clin 16 (2000), 515-525. R.S. Cormack, J. Lehane. Difficult tracheal intubation in obstetrics. Anaesthesia 39 (1984), 1105-1111. M.B. Kaplan, D. Ward, C.A. Hagberg, G. Berci, M. Hagiike. Seeing is believing: the importance of video laryngoscopy in teaching and in managing the difficult airway. Surg Endosc 20 (Suppl 2) (2006), S479-483.

80

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-80

Field Use of the STORZ C-MAC™ Video Laryngoscope in Intubation Training with the Nebraska National Air Guard Ben H. BOEDEKER, MDa,b,1, Mary A. BERNHAGEN, BSa, David J. MILLER, PhDa, Nikola Miljkovicb, Gail M. Kupera and W. Bosseau MURRAY, MDc a Dept. of Anesthesiology, University of Nebraska Medical Center, Omaha, NE b Research Service, Omaha VA Medical Center, Omaha, NE c Clinical Simulation Center, Pennsylvania State College of Medicine, Hershey, PA

Abstract. Previous studies have shown that the videolaryngoscope is an excellent intubation training tool as it allows the student and trainer to share the same anatomical view of the airway. Use of this training tool is limited; however, as many times intubation training must take place outside the hospital environment (as in the training of military health care providers). In this environment, the device can prove to be large and cumbersome. This study examined the use of the Storz CMAC™, a compact video laryngoscope system, for intubation training in a simulated field hospital setting with the Nebraska National Air Guard. The study showed that the C-MACTM was well-received by the trainees and would be useful in a deployment or hospital setting.. Keywords. Video laryngoscope, intubation training, out-of-hospital

Background The video laryngoscope has proven to be an excellent training tool for intubation training as it allows both the instructor and student to share the same airway view during the intubation process. However, in intubation training out in the field, the video laryngoscope can be bulky and cumbersome. This study tests the feasibility of using a compact video laryngoscope system for intubation training in a simulated field hospital. The purpose was to see if the portability of the device allowed it to be a useful training tool outside the traditional OR or hospital setting which typically would not be an optimal teaching environment.

Tools and Methods Following IRB approval at the Department of Veterans Affairs Medical Center, Omaha, Nebraska, 21 Nebraska National Air Guard personnel were instructed in a basic manikin intubation training session at the Nebraska Air Guard Mass Casualty

1

Corresponding Author: Ben H. Boedeker, M.D., Ph.D., Professor, Department of Anesthesiology, University of Nebraska Medical Center; Director, Center for Advanced Technology and Telemedicine, 984455 Nebraska Medical Center, Omaha, NE, 68198-4455, U.S.A.; E-mail: [email protected]

B.H. Boedeker et al. / Field Use of the STORZ C-MACTM Video Laryngoscope

81

Training Exercises at Mead, Nebraska. Participants were instructed in the use of a video laryngoscope in intubating a Laerdal Difficult Airway Manikin™ (Laerdal Medical Corporation, Wappingers Falls, NY). Two different hardware combinations were used in this study. The first was a standard C-MACTM, which included the monitor and a #3 MacIntosh Video Laryngoscope blade (Figure 1). The second was a prototype system developed at the Center for Advanced Technology and Telemedicine (CATT) at the University of Nebraska Medical Center. This setup included a Panasonic Toughbook® (model CF-19, Intel® CoreTM 2 DUO CPU @ 1.2GHz, 1.91GB RAM) and a specially modified C-MAC blade with USB compatibility. The software for viewing the video images was developed by CATT (see Figures 2 & 3).

Figure 1. The STORZ CMAC™ video laryngoscope intubation system (photo courtesy of KARL STORZ Endoscopy-America, El Segundo, CA)

Figure 2. The Panasonic Toughbook® with CATT viewer software and STORZ C-MACTM video laryngoscope blade

Figure 3. Screenshot of viewing software designed by CATT to interface PC and USB C-MAC blade.

B.H. Boedeker et al. / Field Use of the STORZ C-MACTM Video Laryngoscope

82

Upon completion of the training, the subjects completed a post-training questionnaire to assess the use of the C-MAC™ videolaryngoscope in manikin airway intubation and their perceived value of the simulated intubation training session overall.

Results Table 1 summarizes the questions and responses of the post-training questionnaire. Table 1. Results of Post-Training Questionnaire Question How valuable is the video laryngoscope for intubations?

Response on a Likert Scale of 1-10 (where 1= not valuable; 10=very valuable) Average score = 8.9 (SD = 1.2; min=6; max=10; n=21)

How would you rate the teaching value of the video laryngoscope?

Average score = 9.2 (SD = 0.98; min=7; max=10; n=21)

Was the airway training valuable to you?

Average score = 8.9 (SD = 1.1; min=7; max=10; n=21)

Question Would you like to have this device for the airway management at your deployed location? Would you like to have the video laryngoscope deployed at more locations at your hospital?

Response: 1 = yes; 2 = no Average = 1.05 (SD = 0.22; min=1; max=2; n=21) Average = 1.13 (SD = 0.34; min=1; max=2; n=16)

Discussion and Conclusion The results of this study show that the trainees held a very favorable view of the use of the STORZ C-MAC™ video laryngoscope for intubation and as a training tool for teaching the intubation procedure. The majority of the respondents indicated they would like to have the device at their deployed location as well as in more locations at their prospective hospitals. Overall, the study participant gave high ratings for the teaching value of the intubation training course. The portable STORZ C-MAC™ is an effective tool for both intubation in the field and training in an unconventional out-ofhospital environment.

References [1]

[2]

B.H. Boedeker, B.W. Berg, M.A. Bernhagen, W.B. Murray. Direct versus indirect laryngoscopic visualization in human endotracheal intubation: a tool for virtual anesthesia practice and teleanesthesiology. Stud Health Tech Inform 132 (2008), 31-36. Published by IOS Press B.H. Boedeker, S. Hoffman, W.B. Murray. Endotracheal intubation training using virtual images: learning with the mobile telementoring intubating video laryngoscope. Stud Health Technol Inform 125 (2007), 49-54. Published by IOS Press.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-83

83

The Combined Use of Skype™ and the STORZ CMAC™ Video Laryngoscope in Field Intubation Training with the Nebraska National Air Guard Ben H. BOEDEKER, MDa,b,1, Mary BERNHAGEN, BSa, David J. MILLER, PhDa, Nikola MILJKOVICb, Gail M. KUPERa and W. Bosseau MURRAY, MDc a Dept. of Anesthesiology, University of Nebraska Medical Center, Omaha, NE b Research Service, Omaha VA Medical Center, Omaha, NE c Clinical Simulation Center, Penn State College of Medicine, Hershey, PA

Abstract. This study examined the feasibility of using Skype™ technology in basic manikin intubation instruction of Nebraska National Air Guard personnel at a Casualty Training Exercise. Results show that the Skype™ monitor provided clear sound and visualization of the airway view to the trainees and the combination of VoIP technology and videolaryngoscopy for intubation training was highly valued by study participants. Keywords. Voice over Internet Protocol, intubation training, video laryngoscope

Background Telemedicine projects, ranging from instruction and distance mentoring to performance of medical procedures have recently increased [1-2] and typically require a large amount of Internet bandwidth and expensive hardware to perform. In the past few years, VoIP (Voice over Internet Protocol) has become common and robust, using existing Internet infrastructure to transmit voice or video images over a distance. The purpose of this study was to analyze the feasibility of using a commercially available and inexpensive VoIP technology to provide training and medical care over a distance, specifically in the far-forward battlefield.

Tools and Methods Following IRB approval from the VA Medical Center, Omaha, NE, 12 Nebraska National Air Guard personnel (with prior intubation experience) were instructed in basic manikin intubation at the Nebraska Air Guard Mass Casualty Training Exercises

1 Corresponding Author: Ben Boedeker, MD, PhD, Professor, Department of Anesthesiology, University of Nebraska Medical Center; Director, Center for Advanced Technology and Telemedicine, 984455 Nebraska Medical Center, Omaha, NE, 68198-4455, U.S.A.; E-mail: [email protected]

84 B.H. Boedeker et al. / The Combined Use of SkypeTM and the STORZ CMACTM Video Laryngoscope

(Lincoln, NE). Subjects were instructed in the use of the STORZ CMAC™ video laryngoscope (KARL STORZ Endoscopy – America, Inc, El Segundo, CA) fitted with a number 3 video MacIntosh blade. Intubations were performed on a Laerdal Difficult Airway Trainer™ (Laerdal Medical Corporation, Wappingers Falls, NY) on the difficult airway setting. Each study participant was positioned at the manikin’s bedside with the C-MACTM connected to a Panasonic Toughbook® (model CF-19, Intel® CoreTM2 DUO CPU @ 1.2GHz, 1.91GB RAM with built in speakers) through a 3rd party video capture device. The Toughbook® was connected to the Internet with a broadband access card (3.6 MBps with a signal strength of -92dBm). The Toughbook® was also connected to a Logitech® QuickCam® Pro 9000 USB webcam (1600 x 1200 pixels, 30 frames per second and built-in microphone). The instructor, located in another room in the training complex, used a QuickCam® Pro 9000 interfaced with a Dell XPS M1330 (Intel® CoreTM2 DUO CPU @ 2.40GHz, 4.00GB RAM with built in speakers). Internet connectivity was obtained through a T-Mobile UM150VW USB broadband modem (Figures 1).

Figure 1. “Patient” hardware setup with (1) C-MACTM, (2) Toughbook® PC, (3) USB webcam and (4) manikin

During the training, the instructor would view the students both through the external web cam and Internet-connected C-MAC™ to see both the intubating technique and glottic opening view seen by the student. This type of training is similar to that used in previous studies [2], with the exception that, in this case, the instructor was not in physical proximity to the student. Both external and internal views are shown in Figures 2 & 3. The two computers, connected to the Internet, signed into separate SkypeTM accounts, allowing them to pass audio and video in real-time between the two computers. This allowed the instructor to guide the students through the intubation training from a distance. Following the session, the trainees completed a questionnaire to assess the usefulness of Skype™ and the CMAC™ in manikin airway intubation training and their perceived value of the simulated intubation training session overall.

Figure 2. View from C-MAC as shown throuh SkypeTM

B.H. Boedeker et al. / The Combined Use of SkypeTM and the STORZ CMACTM Video Laryngoscope 85

Figure 3. View from external camera showing trainee laryngoscopy technique through SkypeTM

Results (From post-training questionnaire) Responses on a Likert Scale of 1-10 where 1= not valuable; 10=very valuable): Question 1. How valuable is the video laryngoscope for intubations? Average score = 9.2 (SD=1.05; min=7; max=10; n=12); Question 2. How would you rate the teaching value of this SkypeTM training for the video laryngoscope? Average score = 9.5 (SD = 0.80; min=8; max=10; n=12); Question 5. Was this airway training valuable to you? Average score = 9.3 (SD=0.78; min=8; max=10; n=12). Responses where 1 = yes; 2 = no: Question 3. Could you see the laryngoscope view on the SkypeTM monitor clearly? Eleven out of 12 subjects (91%) indicated they could see the monitor clearly. Question 4. Could you hear the sound from the SkypeTM monitor clearly? 100% of the subjects indicated they could hear the sound clearly.

Conclusion The SkypeTM monitor provided both clear sound and visualization of the airway view to all but one of the intubation trainees. The combined use of the SkypeTM technology along with the STORZ CMAC™ video laryngoscope for intubation training was highly valued by the study participants. The study subjects awarded high ratings for the teaching value of the SkypeTM intubation training course. Based on our results, we conclude that the combination of using the portable STORZ CMAC™ and SkypeTM technology is an effective training tool for providing distance training of intubation in the field as well as in an out-of-hospital environment.

References [1] [2]

R.E. Link, P.G. Schulam, L.R. Kavoussi. Telesurgery, remote monitoring and assistance during laparoscopy. The Urologic Clinics of North America 28 (2001), 177-188. B.H. Boedeker, B.W. Berg, M.A. Bernhagen, W.B. Murray. Direct versus indirect laryngoscopic visualization in human endotracheal intubation: a tool for virtual anesthesia practice and teleanesthesiology. Stud Health Technol Inform 132 (2008), 31-36. Published by IOS Press.

86

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-86

Online Predictive Tools for Intervention in Mental Illness: The OPTIMI Project Cristina BOTELLA a,c,1, Inés MORAGREGAa , R. BAÑOS b,c, Azucena GARCÍA-PALACIOS a,c a Universidad Jaume I b Universidad de Valencia c Ciber de Fisiopatología de la Obesidad y Nutrición (CIBEROBN)

Abstract. Mental health care represents over a third of the cost of health care to all EU nations and in US is estimated to be around the 2’5% of the gross national product. It additionally results in further costs to the economy in lost productivity. Depression and Stress related disorders are the most common mental illnesses and the prevention of depression and suicide is one of the 5 central focus points in the European Pact for Mental Health and Well Being. While other mental illnesses may benefit in the long term, Depression and Stress will be the focal point mental illnesses mentioned in OPTIMI. Currently the main treatments for mental illness are pharmacological and evidence based Cognitive Behavioral Therapy (CBT). CBT comprises a set of therapist and patient processes whose format allows for the whole treatment process to be computerized and personalized, Computerised CBT (CCBT). OPTIMI will try to improve the state of the art by monitoring stress and poor coping behavior in high risk population, and by developing tools to perform prediction through early identification of the onset of depression. The main goal of OPTIMI is to improve CCBT programs in order to enhance both efficacy and therapeutic effectiveness. The presentation will outline the main goals the project is aiming and its clinical rationale. Keywords. Depression, Assessment, Prevention, Stress, Computerized Cognitive Behavior Therapy (CCBT), Biosensors, Personal Health Systems, Mobile Technologies, Internet.

1. Introduction Mental health problems cause important costs to the economy through the loss of productivity. The illnesses damage the family and friends networks and therefore markedly impacts at the societal level. For example, in England

and in the United States of America, direct treatment costs of mental disorders were estimated to be around 2.5% of the gross national product. Indirect treatment costs are two to six times higher. Furthermore, depression is the leading cause of disability and the 4th leading contributor to the global burden of disease in 2000. By the year 2020, depression is projected to reach 2nd place of the ranking of DALYs (Disability Adjusted Life Years) calculated for all ages and both sexes. Today, depression is 1

Corresponding Author: Cristina Botella Arbona. Universitat Jaume I. Edifici d’Investigació nº 2. Campus Riu Sec. Castelló de la Plana. 12071. Spain; E-mail: [email protected]

C. Botella et al. / Online Predictive Tools for Intervention in Mental Illness: The OPTIMI Project

87

already the 2nd cause of DALYs in the age category 15-44 years for both sexes combined. Mental health problems account for up to a third of all general practitioner (GPs) consultations in Europe (fact sheet on “Mental Health in the European Region”) It is widely accepted that GPs are unable to deal with Mental Disorders effectively for various reasons, often restricting their therapeutic interventions to the prescription of SSRI´s (selective serotonin reuptake inhibitors) as a catch all send you home solution. The mental illnesses referred to above primarily deal with mood disorders and a number of anxiety disorders usually brought about by stress. These worrying figures confirm that we need to develop new strategies to help those affected, and more importantly, given the near exponential increment in the number of sufferers (young, adolescent, adult, aged and repeat episodes), we need to develop better tools for identification of subjects at risk and to design effective preventing programs. Currently the main treatments for mental illness are pharmacological and evidence based Cognitive Behavioural Therapy. However little is being done to develop effective systems for prevention of the onset of the illnesses. Depression and anxiety are treatable illnesses with CBT as the treatment of choice. However the provision of mental health care is generally less than adequate in terms of accessibility and quality. CBT comprises a set of therapist and patient processes whose format allows for the whole treatment process to be computerized and personalized, Computerised CBT (CCBT). For example in the UK, the Netherlands, Sweden, Italy, Spain, as well as many centres in the USA and Canada there are now emerging several CCBT treatments that include the wide use of Internet and CD based solutions. These computerised treatments are actively being tested and in certain cases licensed for wide application to meet the growing need for treatment . So CCBT will provide a second flank, a good alternative for traditional CBT that is restricted due to a combination of the number of available therapists, with adequate quality training, as well as economics and physical accessibility. The extensive computerization of CBT at all levels will indeed enhance the CBT treatments and provide a powerful means for effective deployment in general mental health care provision. OPTIMI will try to augment the current CCBT state of the art approaches. OPTIMI is a European Project funded by the European Union's 7th Framework Programme Personal Health Systems - Mental Health. The coordination of the program is leading by Everis Spain SL and the OPTIMI web page is: http://optimiproject.eu/ OPTIMI is based on the hypothesis that the central issue in the onset of depression and stress related disorders is the individual’s ability to cope with stress on a psychological and a physiological level. OPTIMI will thus attempt to predict the onset of illness by monitoring mood states, coping behaviour and changes in stress-related physiological variables (e.g. heart rate, cortisol, sleep, etc.).

2. The OPTIMI Project 2.1. Objectives Depression is often associated with poor coping behaviour in the face of stress. Some individuals are extremely resilient but others find it difficult to cope. Based on these premises, OPTIMI has set itself two goals: first, the development of new tools to

88

C. Botella et al. / Online Predictive Tools for Intervention in Mental Illness: The OPTIMI Project

monitor coping behaviour in individuals exposed to high levels of stress; second, the development of online interventions to improve this behaviour and reduce the incidence of depression. The emphasis in this project is to develop tools that will lead to prevention and identification of illness in support of CBT and CCBT treatments. To achieve its first goal, OPTIMI will develop technology-based tools to monitor the physiological state and the cognitive, motor and verbal behaviour of high risk individuals over an extended period of time and to detect changes associated with stress, poor coping and depression. A series of “calibration trials” will test a broad range of technologies. These will include wearable EEG and ECG sensors to detect subjects’ physiological and cognitive state, accelerometers to characterize their physical activity, and voice analysis to detect signs of depression. These automated measurements will be complemented with electronic diaries, in which subjects report their own behaviours and the stressful situations to which they are exposed. All participants will be regularly assessed by a psychologist, who will use standardized instruments to detect stress, poor coping and depression. A few will also be asked to wear implanted devices levels of cortisol in the blood, an objective physiological correlate of stress. The project will use machine learning to identify patterns in the behavioral and physiological data that predict the findings from the psychologist and the cortisol measurements. The final OPTIMI monitoring system will consist of the subset of tools that proves useful for this purpose and acceptable to users. To achieve its second goal, OPTIMI will adapt two existing systems, Beating the Blues [6] and the ETIOBE system [7] already used to provide online CBT treatment for mental disorders. The project will test the treatment systems in “treatment trials” targeting individuals at high risk of exposure to chronic or acute stress. Examples include persons with personal responsibility for the long term care of elderly or disabled people, individuals (especially unemployed people) in situations of acute financial stress, workers in emergency services and students preparing for important examinations. Ongoing monitoring with the OPTIMI tools will make it possible to assess the effectiveness of the treatment and to optimize the treatment cycle. The OPTIMI objectives may be summarised as follows • To develop psychological and physiological monitoring technologies (EEG, ECG, Activity monitoring, Voice Analysis, Cortisol sampling, Electronic Diaries) with the potential to detect early signs of stress, poor coping and depression. We will design wearable sensors providing this measurements on a 24/7 basis and acceptable to users. • To develop a database and a data mining system making it possible to correlate these measurements with assessments by experienced therapists using gold standard diagnostic and cortisol sampling. • To conduct “calibration trials” in three countries, comparing data from sensors with results from regular assessments by trained therapists using gold standard diagnostic measures. • To analyze the results of the calibration trials with the data-mining system, identifying a set of measurements by the development of a rule based engine that will provide effective prediction of stress, poor coping and depression. • To integrate this engine and the sensor system with two existing online CBT therapy systems, making it possible to measure the effectiveness of treatment and to optimize the treatment cycle. • To develop a CCBT system for treatment and prevention of depression and test this integrated system in “treatment trials” in three countries, measuring

C. Botella et al. / Online Predictive Tools for Intervention in Mental Illness: The OPTIMI Project

89

the viability, user acceptability, and accuracy of the system as a tool to diagnose and prevent the onset of depression. 2.2. Sample Healthy people with no mental disorder diagnosis, but with life circumstances related with a high risk to develop depressive symptoms. In OPTIMI we will consider high risk individuals the following: carers of a disabled child or a sick elderly relative, persons who are unemployed and seeking benefits, and students with high pressure study and examinations. 2.3. Measures In order to identify stress coping behaviour patterns we will use behavioural sensors (physical activity and a self-diary) and biological sensors as ECG, Cortisol, Voice Patterns, and EEG. Regarding behavioural sensors, the physical activity will be measured by means of Sleep, Actigraphy and Activity Behaviour (e.g. sports, apathy). The self-diary includes self-information about sleep, mood, coping, stressful events, activities, general health and functioning. 2.4. Studies In a first step, a calibration trial will be performed in three sites, in China, Switzerland and Spain, in which high risk individuals will be chosen (mothers of disabled children, unemployed, final year students) and who will both use the sensor based system on a 24/7 basis over 4 weeks and who will during that time also accept to have fortnightly interviews by the therapist to determine a GOLD standard evolution of any mental condition. With all this information it will be possible to tune the knowledge based system. In a second step, a further intervention trial with high risk individuals students in the UK and unemployed in Spain will be conducted and determine if the enhanced CCBT system provides a preventative approach helping the individuals deal with their stress patterns. 2.5. Procedure This project will develop wearable and domestic appliances based on EEG, ECG, cortisol levels, voice analysis, physical activity analysis and a self reporting electronic diary. In order to identify precedents to the onset of mental illness, OPTIMI will first monitor stress, in high-risk individuals, on a daily basis. Then it will determine the ongoing effect of stress on the individual by studying behavior patterns and physiological variables over longer periods. Measurements will be taken in subjects’ natural environments, thus avoiding artificial laboratory situations. All self-report measurements will be filled online, once a day, in the evening. OPTIMI will try to identify changes in behaviour and certain key biological factors so as to determine how well the person is coping with stress. If the level of coping is below a certain level, we can begin to raise an alarm and then provide some sort of treatment to engage a better coping behaviour.

90

C. Botella et al. / Online Predictive Tools for Intervention in Mental Illness: The OPTIMI Project

3. Results This project is currently in progress, even though, we can show the most significant implementations. The model followed by the OPTIMI project is showed in Figure 1. At a first step, the system receives the input of the sensors; the determinants and key parameters would have been previously defined to get the first level algorithms, with scores calculation; at the second level the calibration of the system is done against the ‘Clinician Gold Standard’; at the third level, through statistical analysis, we get to the acquired knowledge in prediction of depression, providing results of current vulnerability.

Figure1. OPTIMI Model

Besides, we have developed the protocol and tools to carry out the calibration trials. We have already developed a series of self-anchored scales integrated in a Home PC platform which are represented in the Self-Diary. Self-reporting measures are the following: Mood, Coping, Stressful Events, General Activity and General Health. Several mock-ups for the PC application are showed in Figure 2. The data base will allow register and stores the daily information, the tools which will provide feedback for the clinician, as the Self-Diary and the Activity Report. The Activity Report provides us information about the daily activities of the subject and the satisfaction obtained. We also will obtain the location, the type of activity, the type of company and the satisfaction of each daily activity (See Figure 3). This tool provides 4 dimensions of information, thus, giving the contextual information (time) to obtain a correct event interpretation, showed by physiological sensors. The activities: physical leisure, non-physical leisure, studying/working and doing nothing. The social dimension: alone, family, friends, others. The contextual variables: sleep, work, at home and out of home. The degree of satisfaction: low, medium, high. These dimensions will provide a wide range of information about the patterns of activity, which could be very relevant in the study of depression. The information provided by the Activity Report will be correlated with the information provided by the sensors.

C. Botella et al. / Online Predictive Tools for Intervention in Mental Illness: The OPTIMI Project

91

This integrated information will be useful in order to provide a personalized feedback to the participant about the possible risk of the onset of depressive symptoms. This feedback can be used to change the risk behaviors in the treatment trials. The value of this tool, based in the ecological momentary assessment (EMA), is to allow the participant to report his social behavior and how much satisfaction is obtained, close in time to experience. It can sample many events, behaviors and time periods.

Figure 2. Self-Reporting Measures: Mood, Coping and Stress

Figure 3. Activity Report

4. Discussion As with most CCBT applications the treatment are static and not sensitive to personalization, since the only active element can be when the therapist comes online to ask questions and provide feedback. This is one of the limitations of CCBT as pointed out by Anderson In OPTIMI, we are trying to overpass this limitation: the current CCBT will be enhanced with feedback coming as a function of live ongoing events impacting the individual, modifying the content to suit the trial subjects and integrating its communications with the central prediction and diagnostic rule base of OPTIMI. In a preventative role, OPTIMI tools combined with CBT techniques provides advice to the person when the system can detect a combination of stress and a period of

92

C. Botella et al. / Online Predictive Tools for Intervention in Mental Illness: The OPTIMI Project

negative behavioural response. OPTIMI’s tools are meant to obtain relevant information in a constant and systematic way, apart from providing to the users a consistent feedback about their current situation regarding coping, mood and stress and the feedback provides by the system are core aspects in the management and prevention of depressive symptoms. The Activity Report, methodology based in EMA [8], aims to get accurate information about social interaction. This interaction can provides clues to predict more risk behaviour, since depression and anxiety can have symptoms that manifest in the course of social interactions. The other current limitations in CCBT that OPTIMI try to overpass are the dissemination and the amount and type of therapist support needed. OPTIMI will be ready to deliver to the users the appropriate support and relevant information to cope with problems. As far as we know, whilst there has been a lot of research using sensors to identify predictors for stress and depression, there is still a large gap in between this research and its use in treatment and clinical validation for that treatment. The main aim in OPTIMI is to fill that gap.

5. Acknowledgment This study is funded by the European Union's 7th Framework Programme Personal Health Systems - Mental Health - Collaborative Project - Small of medium-scale focused research project - Grant agreement n. 248544

6. References [1] [2] [3] [4] [5] [6]

[7]

[8]

The economic and social costs of mental health problems in 2009/10. (2010). Centre for Mental health. http://www.centreformentalhealth.org.uk/pdfs/Economic and_social_costs_2010.pdf WHO Iniciative Depression in Mental Health (2010). Geneva: WHO. http://www.who.int/mental_health/management/depression/definition/en/ World Health Organization. The ICD-10 Classification of Mental and Behavioural Disorders: Diagnostic Criteria for Research. Geneva, 1993: WHO. National Institute for Health and Clinical Excellence (2006). Computerized cognitive behaviour therapy for depression and anxiety. Review of Technology Appraisal No. 51. Available in www.nice.org.uk G. Andersson. Using the Internet to provide cognitive behaviour therapy. Behaviour Research and Therapy, 47, (2009)175-180 M. Alcañiz, C. Botella, R.M. Baños, I. Zaragoza, J. Guixeres. The intelligent e-therapy system: a new paradigm for telepsychology and cybertherapy. British Journal of Guidance and Counselling, 37: (2009) 287-296. K. Cavanagh, D. Shapiro, S.Van den Berg, S Swain, M. Barkham, & J. Proudfoot. The effectiveness of computerised cognitive-behavioural therapy in routine primary care. British Journal of Clinical Psychology, 4, (2006) 499-514. D.S Moskowitz, S.N Young. Ecological momentary assessment: what it is and why is a method of the future in clinical psychopharmacology. Journal of Psychiatry & Neuroscience, 31 (2006) 13-20.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-93

93

An Integrated Surgical Communication Network – SurgON Richard D. BUCHOLZ, M.D., F.A.C.S.1, Keith A. LAYCOCK, Ph.D., Leslie L. McDURMONT, B.S., and William R. MACNEIL, M.S. Division of Neurosurgery, Department of Surgery, Saint Louis University, Saint Louis, Missouri

Abstract. An integrated communication network, SurgON, has been developed to enable a surgeon to control multiple operating room systems and devices and monitor data streams from diverse sources via a single console. The system also enables the surgeon to grant access and control to remote observers and participants. A test configuration has been evaluated. Keywords. telemedicine, remote consultant, operating room, network, integration

Introduction The increasing use of automated and computer-controlled systems and devices in surgical procedures has resulted in problems arising from incompatibilities in the control and communication standards associated with each system. The surgeon is forced to interact frequently with multiple computer interfaces during a procedure to obtain updates and exert control over the devices required to perform the procedure. Further, transmission of information, and control of devices, frequently requires multiple hardware connections using proprietary connectors that lead to confusion and increased operative time to establish connectivity. There is an urgent need to integrate devices, in order to reduce the number of visualization and control displays, reduce time for establishing connections, and permit more effective control, both locally and using remote consultants. To reduce this complexity and provide the surgeon with more complete and precise control of operating room systems, our group has developed a prototype of an integrated surgical communication network in the form of SurgON (SURGical Operative Network). In addition to improving efficiency in the operating room, this network also allows the surgeon to grant remote access to consultants and observers at other locations via variable bandwidth networks [1]. A key aspect of this system is that the hardware standard employs conventional network addressing, and the software standard employs standard hypertext markup language, allowing remote consultants access with off-the-shelf computers.

1

Corresponding Author. Richard D. Bucholz, M.D., F.A.C.S., Division of Neurosurgery, Department of Surgery, Saint Louis University Health Sciences Center, 1320 S. Grand Boulevard, Saint Louis, MO 63104, United States; E-mail: [email protected]

94

R.D. Bucholz et al. / An Integrated Surgical Communication Network – SurgON

1. Methods In this system, each device acts as a web server, storing web pages that serve as its control panels. These pages are used locally for control via a touch screen display. In addition, each device has an Ethernet port or wireless access adapter, and an IP address is assigned upon attaching a device to the SurgON local area network. Once connected, the stored web page is transmitted as requested and the device becomes controllable locally or remotely (as permitted). Multiple devices can thus be selected and centrally controlled from a single console. The controller allows secure routing of data and control streams via a dynamic firewall to the hospital and beyond. Each device is assured of a specific bandwidth, and every type of surgical device is controllable locally, even in the event of complete network failure. Devices can be connected or removed with instant recognition, while data streams can be shared over the Internet using a graphic user interface. When a device is plugged in, it becomes available on the network through its web page. Alternatively, radio-frequency shielding in the OR will allow a wireless device to be recognized on being brought into the shielded room. The control page is transmitted to control devices when selected, and the device becomes controllable locally or remotely. The system uses a single cable or WiFi connection for all data and control streams. Data originating from the device, such as audio streams from electrophysiological monitors or video from operative cameras, is streamed in real time over this connection. Each device is equipped with a standard encoder that outputs data streams that can be decoded with standard web browsers. Time-consuming preoperative set-up and physical interconnection of devices is essentially eliminated through use of this single connection. Control of complex diagnostic equipment is simplified, and the surgeon’s own control preferences can be encoded and distributed in an automated fashion over many devices. During a procedure, display elements can be moved, re-sized, iconified or closed as desired so that only those inputs relevant to a particular stage are visible and the surgeon or remote user is not overwhelmed by superfluous data. The system can be programmed to make such display adjustments automatically over the course of a procedure.

2. Results The system has been evaluated successfully in configurations simultaneously integrating control of a Möller VM900 surgical microscope, a bipolar coagulator and a Tetrad ultrasound imaging system (Figure 1). Ease of use is assured because the control applets look exactly like the native hardware controls with which the user is already familiar. Our current test model uses an LCD touch screen on a wheeled stand as the local surgical control device, but wall- or boom-mounted plasma screens, head-mounted displays or tablet devices could be used. Every control panel and information stream can be shared outside the OR as required, and the local surgeon can also stipulate the type of access to each device, e.g., data sharing, joint control or autonomous control by a remote surgeon. To facilitate such remote access, dynamic firewall software has been developed which provides a graphic interface that allows sharing of information with a remote consultant equipped with nothing but a PC and web browser. Control of a device can also be granted to a remote consultant, allowing him, for example, to optimize the output of a radiographic device before commenting and interpreting the resultant images. There will be some latency associated with remote control of certain devices, though this will vary with the type of data being transmitted.

R.D. Bucholz et al. / An Integrated Surgical Communication Network – SurgON

95

Figure 1. SurgON LCD touch screen displaying video and controls for Möller surgical microscope, Tetrad ultrasound system and bipolar coagulator.

3. Conclusions SurgON has the potential to provide an optimum user interface for the surgeon, facilitating integration and control of multiple diverse systems in the OR, and providing secure access to data streams and control interfaces for remote observers. Though SurgON is still at the laboratory testing stage, the FDA has already approved a predicate (Stereotaxis Odyssey™) that controls devices via a USB-mouse interface. By comparison, SurgON offers a more versatile applet-based control interface, integrating all proprietary keyboards, sliders and other controls to serve as a complete “universal remote”. SurgON provides a shielded space manifested by a firewall-protected wired and wireless approach, and can be networked and displayed on request to all participating consultants, including those at remote sites. There is a protocol for instant recognition of devices entering the protected space, and the system permits sharing control of devices with remote sites. Standard Internet software is adequate for such networking, so almost any computer may be used as a remote controller. As the special software is contained in the SurgON devices themselves, the remote consultant requires only a web browser. While only a few vendors have so far developed SurgON-configured devices, the open standard makes such modifications relatively easy, and many medical devices already have wired or wireless remotes. The system can scale easily to be employed not only in an OR environment but also in an emergency department or Medevac transport to allow projection of medical expertise to wherever a patient is being managed. Finally, the integrated nature of the associated data streams permits rigorous documentation and archiving of all information generated within a continuum of care.

Reference [1]

H. Feussner. The operating room of the future: A view from Europe. Semin Laparosc Surg 10 (2003), 149-156.

96

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-96

Web-Accessible Interactive Software of 3D Anatomy Representing Pathophysiological Conditions to Enhance the Patient-Consent Process for Procedures D. BURKE1,2, X. ZHOU1,2, V. ROTTY2, V. KONCHADA3, Y. SHEN1,2, B. KONETY, MD1, R. SWEET, MD1,2 1 Urologic Surgery, Medical School, University of Minnesota, Minneapolis, MN 2 Center for Research in Education and Simulation Technologies (CREST), University of Minnesota, Minneapolis, MN 3 Computer Science and Engineering, University of Minnesota, Minneapolis, MN

Abstract. Conveying to a patient the exact physical nature of a disease or procedure can be difficult. By establishing an access website, and using existing 3D viewer software along with our expanding set of anatomical models, we can provide an interface to manipulate realistic, 3D models of common anatomical ailments, chosen from a database frequently updated at the request of the medical community. Physicians will be able to show patients exactly what their condition looks like internally, and explain in better detail how a procedure will be performed. Keywords. Anatomy, Virtual Reality, 3 Dimensional Models, Web-Based, Patient Education

Background Failure to ensure that patients have adequate comprehension of details regarding diagnosis and proposed treatment may result in poor decision-making, lack of informed consent, negative perceptions of the experience and avoidable health care costs. We explore the possibility of adding an interactive virtual reality (IVR) experience to standard informed consent. We hypothesize that using IVR for such a purpose offers many benefits. First, it may enhance knowledge of the procedure as well as their perioperative experience through a first-hand experience. This removes physical tours that would interfere with hospital's regular activities. Second, it may diminish anxiety via establishing comfort in a recognizable, controllable and repeatable environment. Third, it may improve patient’s confidence in the care they will receive. 2D diagrams and plastic models can only go so far before a finer level of detail is needed. The software we are developing allows for anatomic and pathophysiological models with infinite levels of detail in a 3D environment that anyone can easily understand. By allowing the users to request specific diseases/conditions be added, the database can grow with the needs of the community.

D. Burke et al. / Web-Accessible Interactive Software of 3D Anatomy

97

1. Methods Web-based 3D interface tool development has been rising, leaving several options for us to create a useable system that can handle higher-resolution models over the internet. We have done so by importing models into a WebGL environment [1]. By creating a simple point-and-click interface on screen, the user (health-care provider or patient) can easily manipulate the models and strip away layers to view any part of the anatomy at any angle. The anatomy will be pulled from a library of pre-made 3D models defined by their patho-physiological details. Our group has currently developed several anatomic models that can be easily modified to express any abnormalities that need to be produced related to urological conditions. These models are derived from real human MRI/CT, and constructed with careful scrutiny between existing documentation and consulting physicians. Delivery of this product is via website login, accessible from any internet connected computer. We have deployed this system in the Urologic Clinic at the University of Minnesota for pilot testing.

2. Results Our anatomic models are viewed in great detail in real-time speed. When imported into a WebGL interface, even the most complex models are able to be manipulated in realtime. The user-interface is designed and ready for pilot-testing. This Software delivers a ready-made set of common pathophysiological conditions and pilot data from clinicians and their patients to be presented. When offered the opportunity to not only view, first-hand, what the patient’s problem is anatomically, but also the various methods of treatment, they will feel more in control of their treatment options.

Figure 1. Selection of Left Sagittal Male Anatomy.

98

D. Burke et al. / Web-Accessible Interactive Software of 3D Anatomy

Figure 2. Close Up Male Bladder.

3. Discussions Currently there are similar web services that provide the medical community with access to anatomical models defined by location in the body. While these products are helpful, they are not designed with the patient in mind and do not allow for explanation of patho-physiological conditions, disease processes or treatment options. By providing an option that focuses directly on a specific area of the body, along with the various physical ailments that could reside there, a physician could quickly and clearly describe a problem and procedure. Our models can be made to incorporate a level of detail, allowing the user to ‘zoom’ from the macro into the cellular level. Further development could involve the addition of animated tools showing specific procedures in a step-bystep fashion. With all of the resources at our disposal, between graphics computation and medical knowledge, we stand to create a unique system of patient education that will replace stand-alone posters and models, and greatly improve doctor-patient interaction.

References [1] WebGL Scene Graph Library, 28 September 2010, available from http://www.scenejs.org/; Internet; accessed 28 September 2010

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-99

99

Fast Adaptation of Pre-Operative Patient Specific Models to Real-Time IntraOperative Volumetric Data Streams Bruce M. CAMERONa, Maryam E. RETTMANNa, David R. HOLMES IIIa, and Richard A. ROBBa1 a Mayo Clinic College of Medicine, Biomedical Imaging Resource, Rochester MN

Abstract. Image-guided catheter ablation therapy is becoming an increasingly popular treatment option for atrial fibrillation. Successful treatment relies on accurate guidance of the treatment catheter. Integration of high-resolution, preoperative data with electrophysiology data and positional data from tracked catheters improves targeting, but lacks the means to monitor changes in the atrial wall. Intra-operative ultrasound provides a method for imaging the atrial wall, but the real-time, dynamic nature of the data makes it difficult to seamlessly integrate with the static pre-operative patient-specific model. In this work, we propose a technique which uses a self-organizing map (SOM) for dynamically adapting a pre-operative model to surface patch data. The surface patch would be derived from a segmentation of the anatomy in a real-time, intra-operative ultrasound data stream. The method is demonstrated on two regular geometric shapes as well as data simulated from a real, patient computed tomography dataset.

Introduction Image-guided, catheter ablation therapy is becoming an increasingly popular treatment option for atrial fibrillation[1]. Visualization tools such as 2-D ultrasound, bi-plane fluoroscopy, and tracked catheter technologies provide real-time intra-operative data and are important guidance tools. High resolution, patient-specific anatomy – taken from pre-operative CT or MRI scans – may be into the ablation procedure[2, 3]. Currently, it is difficult to fuse pre-operative and intra-operative data effectively. During the procedure, a catheter is guided into the left atrium and radio frequency energy is used to create a pattern of burns intended to interrupt the electrical pathways of ectopic foci. Accurate guidance and targeting is critical to creating a lesion pattern that will eliminate fibrillation. We have developed a prototype system for image-guided cardiac ablation that integrates high-resolution, pre-operative, patient-specific models with intra-operative electrophysiological and positional data from real-time tracked catheters[4, 5], and have shown that incorporating pre-operative and intra-operative data into a single visualization improves guidance[6, 7]. In order to accommodate the dynamic nature of the heart, we are incorporating intra-operative ultrasound into our prototype system. This volumetric data stream allows real-time monitoring of the atrial wall, but the dynamic real-time nature of the 1

Corresponding Author

100

B.M. Cameron et al. / Fast Adaptation of Pre-Operative Patient Speciﬁc Models

data makes seamless integration with static models and imagery problematic. In this paper, we present a method for near-real time adaptation of a pre-acquired, patientspecific model to surface patch data. The surface patch would be derived from a segmentation of the anatomy in a real-time, intra-operative ultrasound data stream. The method is demonstrated on two regular geometric shapes as well as data simulated from a real, patient computed tomography dataset.

1. Methods & Materials The pre-operative patient-specific geometric model is adapted to the incoming point cloud data through the use of a self-organizing map (SOM). SOM is a form of artificial neural network capable of generating a smooth and continuous mapping between vector spaces. SOM is unique among neural networks in that it preserves the topology of the initial vector space and allows for cluster-based operations[8, 9]. The initial topology and starting position of the SOM’s nodes are determined by a previously constructed geometric model. For each vertex in the geometric model there is a corresponding node in the SOM and there is a one-to-one relationship between the topology of the geometric model and the neighborhood relationships within the SOM. The initial value for any one node is determined by the 3-space position of the geometric vertex the node is linked to. This gives us an n-dimensional grid (or map) of Euclidean vectors that we wish to adapt to m randomly distributed input vectors. Adaptation is an iterative, competitive process in which each input vector is presented to the map, a best-matching unit (BMU) is determined and then the BMU and the nodes within its neighborhood are moved toward the input vector. To speed up the search for the BMU, we use a pre-processing step to identify those nodes that are most likely to be involved in the adaptation[10]. The Euclidean distance between each member of this pre-defined subset of nodes and the current input vector is determined and the node with the shortest distance is the BMU. The BMU’s neighborhood is traditionally determined by a Gaussian function: మ

݂ሺ‫ݔ‬ǡ ‫ݕ‬ሻ ൌ

ሺ௫ିఓೣሻ ାሺ௬ିఓ೤ሻ ͳ ିሺ ଶఙ మ ݁ ʹߨߪ ଶ

మ

ሻ

Where σ is the neighborhood radius and μ x and μy are the SOM coordinates of the BMU. The neighborhood radius σ is determined by an initial value and decreases linearly over time. Neighboring points are those points with non-zero values. Rather than incur the computational penalties of directly computing a two dimensional Gaussian function we create, for each node in the SOM, a linked list of nodes sorted by connectivity. That is, if a given SOM node N is associated with vertex A and A is a member of the triangles ABC, ABD, and ACE, then the first nodes in the linked list for node N would be those nodes associated with vertices B, C, D, and E. The next set of nodes would be those associated with the vertices in those triangles that contain the vertices B, C, D, and E, but not A. The list can be built out further until either all nodes are accounted for, or some a-prior limit has been reached. By

B.M. Cameron et al. / Fast Adaptation of Pre-Operative Patient Speciﬁc Models

101

recursively parsing the list, we are able to efficiently determine the relevant neighborhood for any given BMU. Further, the radius becomes a simple limit on the depth of recursion and is easily decremented during each adaptive pass through the SOM.

2. Results The algorithm was implemented in C++ as an Inventor Engine under Linux using the Coin libraries. The algorithm used to identify those nodes most-likely to be involved in the adaptation was implemented as a set of generic C++ classes and is called outside of the Inventor scene graph. Preliminary results based on simple regular geometric shapes showed that the algorithm operates in linear time and is dependent on the number of points in the incoming point cloud. On a generic PC with a 2.8 Ghz Pentium 4 processor and an NVidia Quadro graphics card we were able to maintain frame rates of 20-30 frames per second and processing times were typically under 30 milliseconds. For large point cloud sets (>10,000 vertices) and large geometric models (>50,000 vertices), the processing times to produce the initial frame can exceed 30 seconds. However subsequent frames are produced at frame rates of 20-30 frames per second.

3. Discussion & Conclusions As illustrated in Figure 1, the SOM adaptation yields a surface that is visually smooth, continuous, and well behaved. Slight errors in registration between the point could and the geometric surface can be accommodated. However, as evident in Figure 2, large registration errors can result in some surface distortion.

Figure 1. SOM adaptation (in light grey) of simple regular surface (in dark grey) to a patch (in white). As seen from the side (on left) and from above (on right)

102

B.M. Cameron et al. / Fast Adaptation of Pre-Operative Patient Speciﬁc Models

Figure 2. SOM adaptation showing surface distortion due to large registration errors. Here, the point cloud (white) is not in registration with the original surface (dark grey). The SOM adapted surface (in light grey) shows characteristic distortion. Clockwise from upper left: front view, top view, side view, bottom view.

The transition between those vertices that are displaced by the SOM and those that remain in their original locations is a step function primarily controlled by the initial neighborhood radius. While a small initial radius reduces the time required to compute the adaptation, it also reduces the transition function to a square wave. The transition function can also be affected if there is a large disparity in the sampling frequencies of the initial geometric model and the incoming point cloud. This is illustrated in Figure 3 where a highly detailed pre-operative, patient-specific model is adapted to a sparsely sampled point cloud. The point cloud simulates data segmented from an intra-operative ultrasound image stream. Due to the competitive nature of the SOM, multiple vertices in the geometric model may be mapped to the same point in the point cloud. This phenomenon is most apparent at the transition between the displaced and stationary vertices where the transition takes on a “twisted” appearance. Our initial work shows that by using a pre-processing step to identify those nodes most likely to be displaced during adaptation and to build a map of neighborhood relationships, an SOM can be used to provide fast, near-real time adaptation of preoperative, patient specific models to point clouds which can be derived from intraoperative volumetric data streams. Additional work is needed to reduce the computational time required to perform the pre-processing steps as these steps can result in considerable frame lag when presented with densely sampled data.

B.M. Cameron et al. / Fast Adaptation of Pre-Operative Patient Speciﬁc Models

103

Figure 3. Patient-specific, pre-operative model of left atrium adapted to simulated intra-operative point cloud data. On left, model (dark grey) and point cloud (white). On right, model after adaptation; altered are shown in light grey.

Acknowledgements The authors would like to thank Mr. Jon Camp, Mr. Liu Jiquan, Dr. Douglas Packer, and the staff of the Biomedical Imaging Resource for their contributions to this work.

References [1]

Lin, D. and F.E. Marchlinski, Advances in ablation therapy for complex arrhythmias: atrial fibrillation and ventricular tachycardia. Curr Cardiol Rep, 2003. 5(5): p. 407-14. [2]. Mansour, M., G. Holmvang, and J. Ruskin, Role of imaging techniques in preparation for catheter ablation of atrial fibrillation. J Cardiovasc Electrophysiol, 2004. 15(9): p. 1107-8. [3] Reddy, V.Y., et al., Integration of cardiac magnetic resonance imaging with three-dimensional electroanatomic mapping to guide left ventricular catheter manipulation: feasibility in a porcine model of healed myocardial infarction. J Am Coll Cardiol, 2004. 44(11): p. 2202-13. [4] Rettmann, M.E., et al., An event-driven distributed processing architecture for image-guided cardiac ablation therapy. Comput Methods Programs Biomed, 2009. 95(2): p. 95-104. [5] Rettmann, M.E., et al., An integrated system for real-time image guided cardiac catheter ablation. Stud Health Technol Inform, 2006. 119: p. 455-60. [6] Rettmann ME, H.I.D., Dalegrave C, Johnson SB, Camp JJ, Cameron BM, Packer DL, Robb RA., Integration of patient-specific left atrial models for guidance in cardiac catheter ablation procedures. Medical Image Computing and Computer-Assisted Interventions (MICCAI) workshop on Image Guidance and Computer Assistance for Soft-Tissue Interventions, 2008 Sep. [7] Rettmann ME, H.I.D., Cameron BM, Robb RA., Interactive Visualization of Cardiac Anatomy in Catheter-Based Ablation Therapy. Medical Image Computing and Computer-Assisted Interventions (MICCAI) workshop on Interaction in Medical Image Analysis and Visualization, 2007 Nov.

104

[8]

B.M. Cameron et al. / Fast Adaptation of Pre-Operative Patient Speciﬁc Models

Kohonen, T., Self-organized formation of topologically correct feature maps. Biological Cybernetics, 2001. 43: p. 59-69. [9] Kaski, S., Data exploration using self-organizing maps. Acta Polytechnica Scandinavica, Mathematics, Computing and Management in Engineering Series No. 82, 1997: p. 57 pp. [10] Jiquan L., Rettmann M.E., Holmes D. R. III, Huilong D., Robb R. A., A Piecewise Patch-to-Model Matching Method for Image-guided Cardiac Catheter Ablation. Computerized Medical Imaging and Graphics, (submitted).

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-105

105

Realistic Visualization of Living Brain Tissue Llyr ap CENYDDa, Annette WALTERb, Nigel W. JOHNa,1, Marina BLOJ b and Nicholas PHILLIPSc a School of Computer Science, Bangor University, UK b Centre for Visual Computing, University of Bradford, UK c Department of Neurosurgery, Leeds General Infirmary, UK Abstract. This paper presents an advanced method of visualizing the surface appearance of living brain tissue. We have been granted access to the operating theatre during neurosurgical procedures to obtain colour data via calibrated photography of exposed brain tissue. The specular reflectivity of the brain’s surface is approximated by analyzing a gelatine layer applied to animal flesh. This provides data for a bidirectional reflectance distribution function (BRDF) that is then used the rendering process. Rendering is achieved in realtime by utilizing the GPU, and includes support for ambient occlusion, advanced texturing, sub surface scattering and specularity. Our goal is to investigate whether realistic visualizations of living anatomy can be produced and so provide added value to anatomy education. Keywords. Rendering, BRDF, Brain anatomy

Introduction Cadaver dissection is widely accepted as being the ‘gold standard’ for anatomy education. It provides the learner with knowledge of the shape and size of the organs; it gives them an appreciation of spatial relationships between organs; and also introduces students to death in a controlled manner. However, financial and ethical pressures have led to a decrease in the availability and usage of this approach [1]. This has led to anatomy today being taught in a variety of different ways, including prosections, problem-based learning scenarios, and computer graphics based systems such as those using data derived from the Visible Human and similar projects. All of these methods offer potential benefits to the learner. However, one weakness with computer generated anatomy models is that they are rendered as grey scale or use pseudo colour that at best is a gross approximation of the true appearance of healthy living organs. Even cadaveric specimens are very different in appearance from when they were still live tissue. Our hypothesis is that more realistic models of internal human anatomy will improve anatomy education, preparing the novice surgeon or nurse to more easily recognize anatomy the first time they are exposed to an operation on a real patient. This paper presents the results of a multi-disciplinary feasibility study that has been carried out to investigate the above hypothesis using the human brain as the target organ. We have been granted access to the operating theatre during neurosurgical procedures to obtain colour data from calibrated photography of exposed brain tissue. The specular reflectivity of the brain’s surface is approximated by analyzing a gelatine 1

Corresponding Author: Nigel W. John, University of Bangor, UK; E-mail: [email protected]

106

L. ap Cenydd et al. / Realistic Visualization of Living Brain Tissue

layer applied to animal flesh. This provides data for a bidirectional reflectance distribution function (BRDF) that is then used to help generate a realistic model of living brain tissue. The human skin has been extensively modelled using BRDF models e.g. [2], [3]. The only previous example that we have discovered for rendering internal anatomy with the use of a BRDF is within a virtual bronchoscopy application [4]. This solution exploits the particular restrictions of image acquisition from a bronchoscope, however, and is not applicable to recreating anatomy models from the wide range of viewing directions and illuminations required for general use.

1. Methods and Materials A BRDF is a function that defines how light is reflected at an opaque surface [5]. The parameter data for the BRDF can be recovered from calibrated photographs and light sources [6]. We use the Lafortune model [7] to render colour accurate computergenerated images: 9:

&(', )) = +- + ∑40123,4 5'2 )2 + '3 )3 6 + 17,4 '7 )7 8

(1)

where ρd, Cxy,i, Cz,i, and Ni are the fitted parameters (sum over i is the sum of each lobe, specular and retro-reflective in this case), (ux, uy, uz) and (vx, vy, vz) are the incident and excitant light vectors respectively.

Figure 1. Photograph of Exposed Brain Surface before and after colour shift calibration (image shown is after a non-calibrated print process and/or screen display).

1.1. Colour Data Colour samples of living brain tissue have been collected from photographs taken with patient consent during neurosurgery procedures at the Leeds General Infirmary. We used a Canon EOS 5D digital camera with a 50mm lens set on manual mode and with auto focus. We selected the ISO 200 setting under a colour temperature of 6500K. A small white reference card was held by the surgeon near to the region of interest (i.e. an area of exposed brain). This enables the images to be later recalibrated to compensate for the colour shift caused by the illumination and so allow the RGB values of brain tissue, blood vessels and cancer tissue to be calculated independently of the used light source. The pictures were acquired in raw format to avoid automatic recalibration by

L. ap Cenydd et al. / Realistic Visualization of Living Brain Tissue

107

the camera. Figure 1 is an example of the photographs taken, showing a small area of the exposed brain surface of the patient. 1.2. Reflectance During neurosurgery the brain is constantly drenched with sterilized water, which results in a large uniform specular reflection on its surface. There are several parameters that can affect this specular effect e.g. position of light sources and the camera, distance between camera and region of interest, and size of the region of interest. However, the time made available to us in the operating theatre to take photographs and the inherent restrictions of working in this environment meant it was not possible to collect enough data to fully investigate these parameters in vivo. To ensure we remained unobtrusive to the surgeon and posed no additional risk to the patient we were able to collect a maximum of five photographs using a single fixed light source position. An in vitro experiment was also designed using animal flesh (a lamb steak) as a possible brain substitute for generating the reflectance data. A layer of gelatine was applied to the surface of the meat (just adding water to the surface provided insufficient specularity as the water evaporated too quickly). This model does produce a slightly higher reflectance profile than in the in vivo scenario. However, a diffuse light source is required for a BRDF model and this can be used in this experiment. The strong light source used in the operating theatre focuses the radiation and so gives higher reflectance behaviour than is desirable for our purposes.

Figure 2. Measurements from two illumination positions at 75° and 65° from the surface normal. In each case the five camera positions are indicated by the coloured circles. The curves are fitted to the rgb pixel values of the region of interest, which represents the spatial surface reflectance. The distance from the centre shows the relative intensity of the pixel values. An ideal Lambertion reflectance profile would be represented by a radial curve (semi-circle). Location of specular components are indicated by an increase in intensity.

The Lafortune BRDF parameters in the in vitro experiment can be calculated based on images corresponding to two illuminated locations and five camera locations - see Figure 2. Observe that the peak of the specular reflection is shifted by 10 degrees; 15 degrees would be theoretically expected for a plane surface. The curvature of the gelatine layer at the region of interest is not known. The intensity for all RGB components, under both light positions, is similar but not identical after taking into account the shift of the intensity peak. This indicates an insignificant change of colour for the meat surface due to different light positions. To reach our aim of a realistic visualization of living brain tissue this small position error of the specular reflectance peak will not change the visual fidelity of the rendered surface. It can be ignored

108

L. ap Cenydd et al. / Realistic Visualization of Living Brain Tissue

because a shift of 5 degrees in reflectance is not discernable for a complex structure like the brain. 1.3. Rendering the Brain The brain surface mesh used in this project has been created in a 3D modelling package. A mesh segmented from volume data did not provide enough accuracy for depicting the folds (gyri) and grooves (sulci) that are abundant across the cerebral surface. Our brain rendering algorithm is implemented on the GPU using the GLSL 2.0 shading language, programmed in ATI's RenderMonkey development environment. In the first pass, diffuse lighting is calculated and a procedural detail texture is generated through a process of multi-texturing, using tile-able vein and corresponding bump-map textures. The brain is then rendered into an off-screen texture, where a series of separable blur passes are performed to produce irradiance textures, simulating subsurface scattering. Finally, in the last pass a final diffuse texture is produced by blending all generated irradiance textures, and the Lafortune BRDF model is used to calculate specular and retro reflection of light. The sum of diffuse, ambient occlusion map and specular gives the final pixel colour. Ambient Occlusion. Ambient occlusion adds realism to local reflection models by taking into account attenuation due to occlusion. It allows for better perception of the 3D shape of objects by enhancing lighting contrast. We pre-compute ambient occlusion maps to highlight the deep sulcus (fissures) of the brain's surface. Texturing the Brain Surface. The surface membrane of the human brain contains a network of blood vessels of various sizes and structure. Our algorithm automatically constructs the micro diffuse detail and topology of the brain's surface by sampling a series of high resolution vessel textures at different scales, colours and strengths. Each tileable vessel texture is created by modeling a representative section of the brain's surface from reference photographs, and baking the geometry into a bump map. The vessel textures are layered to construct a complex pseudo-anatomically correct topology. Each instance is parameterized with scale, colour and bump-strength constants. Creating the diffuse texture through parameterized multi-texturing allows for the generation of an extremely detailed surface diffuse texture and topology. Sub Surface Scattering. The human brain is translucent and so some light waves will penetrate the surface, be scattered, partially absorbed and then exit at a different location. The sub-surface scattering algorithm used in our current rendering pipeline is based on a technique introduced in [8], which takes advantage of the fact that the transport of light though skin and similarly transparent materials (like the brain) are local enough in nature that scattering can be approximated through a process of texturespace diffusion [9]. A diffusion profile describes the manner by which light scatters through the surface of a highly scattering translucent material. By calculating a diffusion patch for every region across a polygonal surface, the aggregate sum of overlapping patches gives a realistic translucent appearance. In texture-space diffusion, diffusion profiles are created by summing multiple Gaussian functions together to create a convolution filter describing how light spreads to neighbouring locations. An initial pass in our algorithm calculates the diffuse colour and per-pixel lighting across the brain surface in texture space, based on the generated detail texture's diffuse and normal map components respectively. This yields a diffuse map of the brain's surface, which is rendered to an off-screen texture using the texture coordinates as position - effectively unwrapping the 3D mesh onto a 2D plane. Our current

L. ap Cenydd et al. / Realistic Visualization of Living Brain Tissue

109

implementation represents diffusion profiles as a sum of up to six Gaussian functions, with each Gaussian weighted separately for red, green and blue wave-lengths. The weighted sum of the three bands for each Gaussian is normalized so that no colour shift occurs during convolution, as the base diffuse colour has already been deduced in the first pass. A multilayer diffusion profile is necessary for approximating scattering though brain tissue. Specularity. The Lafortune BRDF model is used to provide the specularity of the brain’s surface. From our measured data, for each waveband (RGB), and each incident light angle, there are parameters fitted for each component of this model, describing a two lobe representation of surface reflectance: • • •

Lambertian/Diffuse component. Specular-reflective xy and z components, plus exponent. Retro-reflective xy and z components, plus exponent.

The Final Colour. After calculating the diffuse lighting, irradiance textures and specular reflectance, the final pass acts to process and combine the components. The brain mesh is rendered in 3D using a standard vertex shader. The different blurred textures are linearly combined to compute the total diffuse light exiting the surface, with each texture being multiplied by its corresponding Gaussian weight and renormalized to ensure white diffuse light. The contribution of diffuse, specular and ambient occlusion maps are combined to give the final pixel colour.

2. Results Figure 3 demonstrates the effect of specular highlighting rendered with our Lafortune BRDF model. The results indicate that the system can visually reproduce the reflective properties of tissue from real world measurements. The brain is a very complicated surface with many folds and fissures, properties which make it very difficult to generate a map of the entire surface free of excessive distortion or overlap. Due to the prevalence of texture-space computation, we currently concentrate on rendering a representative section of the human brain. One potential solution to this problem is to use cerebellar flat mapping techniques [10], which use a circle packing algorithm to unfold the complicated gyri and sulci into a simpler map of the cortical surface. Implementing a variation of this technique would also facilitate rendering segmented meshes from volume data. The excessive curvature of the brain's topology also leads to distortion, as distances on the diffuse texture do not correspond directly to distances on the mesh itself. We also plan to compute stretch-correction textures to correct this issue.

Figure 3. The brain surface rendered without (left) and with (right) the in vitro experiment BRDF data applied (75 degrees, 5 camera locations; 60 degrees, 5 camera locations)

110

L. ap Cenydd et al. / Realistic Visualization of Living Brain Tissue

Sub-surface scattering through texture-space diffusion is a powerful technique that can achieve very realistic results. Figure 4 shows a section of the brain mesh rendered using our Lafortune BRDF model, combined with sub-surface scattering calculations from a 3¬Gaussian diffuse profile. While we currently use hand-crafted Gaussian variance and weighting values, we are in the process of fully parameterizing the system to facilitate further experimentation.

Figure 4. Left: Section of brain rendered with 3 Gaussian sub-surface scattering (8 passes). Right: Detail on brain's surface, created with one vessel texture at differing tile scales.

Figure 4 also gives an example of the surface detail generated by our texture generation algorithm, with a highly specular BRDF designed to replicate water on the surface. Generating a realistic diffuse texture is largely dependent on the quality and accuracy of the underlying vessel texture maps, which we continue to improve. Finally Figure 5 provides a pictorial comparison between a photograph of the exposed brain surface of a patient in the operating theatre with synthetically generated results of our approach. In the latter case, the diffuse colour has been obtained from the calibrated operating theatre photographs, and the reflectance parameters from the in vitro meat with gelatine layer BRDF (75 degrees, 5 camera locations; 60 degrees, 5 camera locations). A second BRDF is mixed in (Cxy = -1, Cz = 1; essentially Phong specularity, direct reflection) to simulate the sharp glistening from the water layer.

Figure 5. Photograph from operating theatre (left); Rendered brain surface (right)

3. Conclusions We have presented an advanced method of visualizing the surface appearance of living brain tissue. We have been granted access to the operating theatre during neurosurgical procedures to obtain colour data from calibrated photography of exposed brain tissue. The specular reflectivity of the brain’s surface is approximated by analyzing a gelatine

L. ap Cenydd et al. / Realistic Visualization of Living Brain Tissue

111

layer applied to animal flesh. This provides data for a bidirectional reflectance distribution function (BRDF) that is then used the rendering process. Rendering is achieved in realtime by utilizing the GPU, and includes support for ambient occlusion, advanced texturing, sub surface scattering and specularity. The constraints of acquiring all of the data required in an operating theatre environment have prevented a full BDRF model being calculated purely from patient data. However, the reflectance data obtained from lamb/gelatine material has proven to be an adequate alternative that provides specular reflectance that is close to that of living brain tissue. Other anatomy can also be depicted using the techniques described above and in many cases it will be easier to acquire the BRDF data – more surface area of organs is exposed in abdominal surgery, for example, and the organs have far smoother surfaces than the brain, and are not covered with a membrane. In the future we expect to create a library of realistic models of virtual organs that look vibrant and alive. We plan further improvements to the rendering pipeline. A future extension will allow the texture generation process to take into account the properties of the brain's topology when calculating the vein size, colour and opacity. One possible solution is to use the pre-computed ambient occlusion texture, which effectively maps the location and depth of fissures, where vessels tend to be more visible and numerous. Our clinical collaborators have commented favourably on the first results that we have obtained and presented in this paper. The final stage of the current project will be to evaluate our models with medical educators and students. We have repeated the experiment at Keele University School of Anatomy to obtain data from cadaveric brain specimens and so generate equivalent cadaveric brain models to act as a comparison. The validation study is currently under way and will be reported in a follow up paper.

References [1] MCLACHLAN J.C., BLIGH J., BRADLEY P., SEARLE J.: Teaching anatomy without cadavers. Medical Education 38, 4 (2004), 418–424 [2] DEBEVEC P., HAWKINS T., TCHOU C., DUIKER H.-P., SAROKIN W., SAGAR M.: Acquiring the reflectance field of a human face. In ACM SIGGRAPH ’00 Conference Proceedings (2000), 145–156 [3] DONNER C., JENSEN H.W.: A Spectral BSSRDF for Shading Human Skin. In Proc. Eurographics Symposium on Rendering, (2006) 409-427 [4] CHUNG A.J., DELIGIANNI F., SHAH P., WELLS A., YANG G-Z: Patient-specific bronchoscopy visualization through BRDF estimation and disocclusion correction. IEEE Trans on Medical Imaging 25, 4 (2006), 503 – 513 [5] NICODEMUS, F.: Directional reflectance and emissivity of an opaque surface. Applied Optics 4, 7 (1965), 767–775. [6] DEBEVEC P., REINHARD E., WARD G., PATTANAIK S.: High dynamic range imaging. In ACM SIGGRAPH 2004 Courses (2004) [7] LAFORTUNE E.P.F., FOO S-C., TORRANCE K.E., GREENBERG D.E.: Non-linear approximation of reflectance functions. In ACM SIGGRAPH 97 Conference Proceedings, (1997) 117-126 [8] BORSHUKOV, G., LEWIS, J. P: Realistic Human Face Rendering for The Matrix Reloaded. In ACM SIGGRAPH 2005 Courses (2005) [9] D’EON E., LUEBKE D.: Chapter 14. Advanced Techniques for Realistic Real-Time Skin Rendering. In GPU Gems 3 (Nguyen H., Editor). Addison-Wesley, 2007. [10] HURDAL M. K., LEE A., RATNANATHER T., NISHINO M., MILLER M., BOTTERTON K.: Investigating the Medial Prefrontal Cortex with Cortical Flat Mappings. NeuroImage, 19, 2 (2003)

112

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-112

A Virtual Surgical Environment for Rehearsal of Tympanomastoidectomy Sonny CHAN a , Peter LI b , Dong Hoon LEE b , J. Kenneth SALISBURY a and Nikolas H. BLEVINS b,1 a Department of Computer Science, Stanford University b Department of Otolaryngology, Stanford University Abstract. This article presents a virtual surgical environment whose purpose is to assist the surgeon in preparation for individual cases. The system constructs interactive anatomical models from patient-speciﬁc, multi-modal preoperative image data, and incorporates new methods for visually and haptically rendering the volumetric data. Evaluation of the system’s ability to replicate temporal bone dissections for tympanomastoidectomy, using intraoperative video of the same patients as guides, showed strong correlations between virtual and intraoperative anatomy. The result is a portable and cost-effective tool that may prove highly beneﬁcial for the purposes of surgical planning and rehearsal. Keywords. Surgical simulation, surgical rehearsal, haptic rendering, volume rendering, patient-specifc models, temporal bone surgery

1. Introduction Given the current availability of high-resolution three-dimensional medical imaging, surgeons commonly have access to multimodal anatomic data prior to undertaking a surgical procedure. Imaging studies such as computed tomography (CT) and magnetic resonance imaging (MRI) can offer accurate information on tissue composition and geometry, and are often used together given their complementary strengths. However, even after structures are identiﬁed on imaging, the surgeon must be able synthesize these data into a conceptual model that can predict what will be encountered intraoperatively. To achieve this, the surgeon often needs to take a number of steps in preparing for surgery: 1. Mentally co-register volumetric data from different modalities, so that the studies can be combined effectively to take advantage of the best aspects of each. 2. Formulate an integrated 3-D representation of the patient from the studies, so that anatomic relationships are understood from a variety of potential viewpoints. 3. Create a mental image that predicts how surgical manipulation and removal of tissues will affect subsequent access and exposure. Accomplishing these steps can be a challenge, especially if the studies are examined as sequential two-dimensional slices, as is the current practice. Despite the use of multi1 Corresponding Author: Chief, Division of Otology/Neurotology, Department of Otolaryngology – Head and Neck Surgery, Stanford University, 801 Welch Road, Stanford, CA; E-mail: [email protected].

S. Chan et al. / A Virtual Surgical Environment for Rehearsal of Tympanomastoidectomy

113

planar reconstructions, critical spatial relationships need to be inferred rather than seen directly as occurs in actual surgery. We have developed a virtual surgical environment intended to facilitate these steps, and optimize the beneﬁts of available imaging. Our goal has been to create an environment that can relatively quickly incorporate routine clinical studies, enabling real-time interactive preoperative assessment. Our approach thus far has focused on procedures involving the resection of cholesteatomas (skin cysts) from the middle ear and mastoid, collectively known as tympanomastoidectomy. Such procedures involve the removal of portions of the temporal bone to gain access to these cysts, which are commonly associated with chronic ear infections. The ability to experiment with varied approaches may prove beneﬁcial to the outcome of the procedure. Traditional imaging of the temporal bone prior to tympanomastoidectomy involves the use of high-resolution CT. This demonstrates bone and air quite well, and therefore shows other adjacent structures by the absence of either of these. It does not, however, differentiate between various types of soft tissue, such as scar or ﬂuid, from the more surgically relevant cholesteatoma. Recently, diffusion-weighted MRI sequences have been used successfully to aid in the identiﬁcation and localization of cholesteatoma [5]. We obtained MR imaging on a series of patients with chronic ear infections, and integrated its speciﬁc identiﬁcation of cholesteatoma into the bony framework from the CT images. A number of virtual environments for simulation of temporal bone surgery have been developed [2]. Wiet et al. describe dissection of a virtual temporal bone derived from CT images of a cadaveric specimen in an early work demonstrating a surgery simulator incorporating both visual and haptic feedback [12]. Agus et al. present a similar surgical training system where volumetric object models directly derived from CT and MRI data can be dissected [1]. They simulate a virtual surgical drill using a realistic, physically based model for haptic force feedback and bone tissue removal. Morris et al. present a comprehensive simulation environment for both training and performance evaluation of bone surgery [7]. They discuss techniques for automated evaluation and feedback, allowing the possibility of using the simulator for surgical skills assessment. In a recent study, Tolsdorff and colleagues have proposed the use of individual patient models derived from CT image data for virtual mastoid surgery [11], though the study was conducted using a training-oriented simulator [8], and only allowed for the import of bone tissue. The majority of surgical simulation work to date has focused on surgical education, training, and assessment using standardized libraries of anatomic models. In contrast, the system described here is intended as a step towards surgical rehearsal, with which a surgeon can prepare for an upcoming case by practicing dissections on a virtual representation of the patient’s speciﬁc anatomy. To do this, it makes direct use of multi-modal preoperative imaging data with minimal need for preprocessing. It also incorporates new and efﬁcient methods to render multiple volumetric data sets visually and haptically to enable interaction with the virtual anatomy in a manner familiar to surgeons.

2. Mathods & Materials Preoperative image data from a growing library of 8 patients, each of whom was a candidate for tympanomastoidectomy, was collected for evaluation. Imaging for each patient consisted of a clinical CT scan of the temporal bone (Figure 1a) and two MR images: a T2-weighted FIESTA sequence and a diffusion-weighted PROPELLER sequence (Fig-

114

S. Chan et al. / A Virtual Surgical Environment for Rehearsal of Tympanomastoidectomy

(a)

(b)

(c)

Figure 1. A preoperative axial clinical CT scan of the temporal bone (a). Soft tissue is seen ﬁlling the middle ear (arrow). The corresponding slice of the MR PROPELLER sequence shows a hyperintense region indicative of cholesteatoma (b). A close-up of the cholesteatoma registered and superimposed on the CT (c).

ure 1b). Conventional CT and MR imaging cannot easily identify cholesteatoma within the temporal bone, but diffusion-weighted MR imaging shows potential as the modality of choice for this purpose (Figure 1c). These images contain complementary information, and are used collectively to create a virtual model of the patient’s anatomy. 2.1. Data Processing Registration of the temporal bone CT and the MRI sequences was performed using Amira 5.3 (Visage Imaging Inc., San Diego, CA). The DICOM images were imported into Amira and registered in three dimensions using the automated registration tool with a mutual information metric and the conjugate gradient optimization algorithm. Anatomic structures of interest were extracted from the different imaging datasets using Amira’s computer-assisted segmentation tools. The CT images were used to segment the semicircular canals, facial nerve and ossicles. The FIESTA image sequence was used to segment the carotid artery and sigmoid sinus, and the PROPELLER image sequence was used to segment cholesteatoma. Segmentations were exported both as label volumes and as Wavefront object meshes to be used in our virtual surgical environment. Data processing, including segmentation of all vital structures and smoothing of the resulting models, took approximately two hours for each patient. Not all processing steps are necessary in every case though: a clinical CT can be used directly by our virtual surgical environment for visualization and dissection of bone without any preprocessing. 2.2. Volume Rendering Our system uses three-dimensional preoperative image data as the principal representation of the virtual patient. As this model consists primarily of volumetric data, it is natural to adopt a real-time volume rendering approach for data visualization within the simulation. We have developed a multi-modal volume rendering method based on a GPUaccelerated ray casting approach [9] that is capable of simultaneously displaying the different forms of data supplied to the virtual surgical environment. We simultaneously display a clinical CT volume (with both isosurface and direct volume rendering), a segmentation volume, and a dissection mask volume in a single rendering pass. Each data volume is represented as a separate 3D texture on the graphics processor, but is co-located in texture coordinate space with the primary (CT) volume. This allows the rendering algorithm to perform ray casting in lock step through all data

S. Chan et al. / A Virtual Surgical Environment for Rehearsal of Tympanomastoidectomy

(a)

(b)

115

(c)

Figure 2. A slice of a combined and dilated segmentation volume showing the cholesteatoma, cochlea and semi-circular canals, and carotid artery in different colors over the CT scan (a), and the low-pass ﬁltered union of the structures for volumetric isosurface rendering (b). Note the cholesteatoma in (a) is a result of dilation from an adjacent slice, and is absent in (b). The black area in a close-up of a mask volume shows the smooth edge of a region removed by a spherical burr (c).

volumes simultaneously. A ray is cast for each pixel in the rendered image from the viewpoint in virtual space through the volume data, accumulating color and opacity information that determines its ﬁnal appearance. A shader program samples all volumes along the ray at a regular spatial interval, taking advantage of the highly parallel computational architecture of modern GPUs to achieve interactive visualization. The preoperative clinical CT serves as the primary volume, and is represented at its native resolution as a 16-bit single-channel texture in video memory. The rendering algorithm traverses the ray until it ﬁrst encounters a sample value greater than the isosurface value. Several interval bisection steps [3] are then performed to reﬁne the rayisosurface intersection point, and the surface is shaded with conﬁgurable material properties. If the surface is partially or fully transparent, the ray is continued, accumulating color and opacity through the interior of the volume in a front-to-back composition using a pre-integrated color transfer function [9], until the exiting isosurface is found. Thus, the appearance of the primary volume is controlled by an isosurface value (in CT Hounsﬁeld Units) with material properties and a user-deﬁned transfer function that maps Hounsﬁeld Units to optical properties, both of which can be speciﬁed interactively. In our data, we have found that an isosurface provides a good indication of the tissue-air boundary, and direct volume rendering generates a realistic visualization of the bone tissue (Figure 3c). A segmentation or label ﬁeld for an anatomical structure consists of a binary volume which indicates inclusion of contained voxels into the structure. We apply a preprocessing step that combines all non-overlapping label volumes into a two-channel, 8-bit luminance-alpha texture to be rendered volumetrically (Figure 2a, 2b). The alpha channel contains a union of the segmentations, and is low-pass ﬁltered to facilitate isosurface rendering without stepping artifacts [4]. The luminance channel contains a unique index for each constituent structure that maps to a set of material properties, and a dilation operation is performed on the ﬁnal image to ensure that texture sampling retrieves the correct index. The ray-casting algorithm samples this volume in lock step with the primary volume, and if the sample is found to lie within a segmented structure, material properties are retrieved from an auxiliary texture to perform shading (Figure 3b). Finally, the mask volume, which controls visibility of the model, is an 8-bit, singlechannel representation that can represent smooth surfaces to a sub-voxel resolution (Figure 2c). During ray casting, both the primary and segmentation volumes are modulated

116

S. Chan et al. / A Virtual Surgical Environment for Rehearsal of Tympanomastoidectomy

(a)

(b)

(c)

Figure 3. Temporal bone anatomy, including the sigmoid sinus, semicircular canals, facial nerve, ossicles, carotid artery, and a cholesteatoma lesion rendered as polygonal meshes (a), the larger structures rendered as volumetric isosurfaces with a semi-transparent bone surface in front (b), and direct volume rendering of the bone in a full composite, ready for surgery (c).

by the mask value. Thus, editing of the volume data can be accomplished by attenuation or zeroing of the mask volume. In addition to the volumetric data, our virtual surgical environment uses polygonal models to represent surgical instruments and certain segmented structures. Anatomical structures with details near to or ﬁner than the native voxel resolution of the data may be better represented as polygonal meshes (Figure 3a). Examples of these structures include the facial nerve and the ossicles. However, the rendering of polygonal meshes is not affected by changes in the mask volume described earlier, and thus any structure that permits dissection should be represented as part of the segmentation volume. 2.3. Haptic Rendering The primary representation of the patient’s anatomy in our virtual surgical environment is in volumetric form, and thus a method for haptic interaction with volume data is required. McNeely et al. proposed a popular algorithm for haptic rendering of volumetric geometry [6], and several variants have been described for use in surgical simulators [1,7,8]. However, these methods do not prevent tool-tissue interpenetrations in the simulation environment, and can suffer from the virtual instrument passing completely through a thin layer of tissue (Figure 4a), especially when using commercially available haptic devices with lower force output and limited stiffness capabilities. Proxy-based rendering algorithms constrain the virtual instrument to the surface of the geometry, preventing pop-through problems (Figure 4b). Salisbury & Tarr have described an algorithm for proxy-based haptic rendering of implicit surfaces [10] that can readily be adapted to render isosurface geometry embedded within volumetric data. A volume can be treated as a discrete sampling of a scalar ﬁeld, and can be sampled at arbitrary positions through interpolation. Rather than evaluating an analytical gradient, the surface normal can be estimated using a central difference. The primary limitation here is that a tool can only be modeled as a single point for interaction with the volume. A virtual surgical drill allows dissection of the anatomy in our virtual surgical environment. We model the spherical burr of the drill by extending the proxy-based, point interaction algorithm to incorporate elements of the method described by [7]. The burr is discretized into individual points in three-dimensional space that occupy the volume of the sphere. During interaction, the volume is sampled at these points, and any point found to lie within the surface exerts a ﬁxed amount of force toward the center of the

S. Chan et al. / A Virtual Surgical Environment for Rehearsal of Tympanomastoidectomy

117

F F F

(a)

(b)

(c)

Figure 4. With a point sampling algorithm, the contact force can be in the wrong direction when the instrument is pushed into a thin object (a). A proxy-based algorithm can constrain an interaction point to the surface (b). We combine these algorithms to prevent the virtual instrument from popping through thin layers of bone (c).

burr. In addition, we treat the center of the burr as a proxy point, so that if the haptic device moves in a way that this point penetrates into an object, the proxy-based algorithm constrains the center of the virtual instrument to lie on the surface of the object. A virtual spring exerts additional force proportional to the displacement between the device position and the proxy point. The end result is that superﬁcial interaction forces are computed primarily from point sampling, whereas the virtual spring force dominates during haptic exploration involving deep tool-tissue interpenetration (Figure 4c). Tissue resection is modeled by attenuating or removing voxels from the mask volume in a manner similar to that described in [8]. When the virtual drill is on, an antialiased voxelization [4] of the spherical burr is subtracted from the mask volume, preserving a smooth, accurate cut surface (Figure 2c). This technique allows modeling of tissue dissection at a sub-voxel resolution, and prevents the jagged or stepped appearance that normally results from voxel-resolution modiﬁcation of volume data. 3. Results & Conclusions Images from the use of our virtual surgical environment on data from two selected patients are shown in Figure 5. We have been able to replicate salient anatomic detail in the virtual environment as compared to the video images taken during actual tympanomastoidectomy. The geometry from the CT dataset yields a subjectively accurate representation of the bony contours seen during surgery. Similarly, the cholesteatoma volume derived from PROPELLER MR imaging is accurately placed within the bone, and presents a realistic representation of what the otologic surgeon will encounter in the patient. By rendering the bone transparent, other segmented vital structures can be seen in their familiar relative locations. Our preliminary subjective experience suggests that our virtual surgical environment can offer an accurate and interactive representation of patient-speciﬁc anatomy. Our system represents a step towards the use of a virtual environment to prepare for tympanomastoid surgery. It enables the relatively rapid integration of multi-modal imaging datasets, direct volume rendering, and a means of manipulating preoperative clinical data in a surgically relevant manner. We anticipate that the methods described can be generalized to a variety of surgical procedures. Clearly tools such as this require objective validation to ensure that they can beneﬁt a surgeon in preparing for an operative case. We intend to carry out such studies in the future as our system becomes further reﬁned. We are encouraged by the assumption that the more a surgeon is familiar with working in and around speciﬁc anatomy, the more he or she is likely to be effective. Offering surgeons such an opportunity holds great potential.

118

S. Chan et al. / A Virtual Surgical Environment for Rehearsal of Tympanomastoidectomy

(a)

(b)

(c)

(d)

Figure 5. An intraoperative video capture during right tympanomastoidectomy (a) and the corresponding image from a virtual dissection (b) using that patient’s preoperative imaging data. The cholesteatoma (white) was automatically segmented from MR imaging, and has been exposed following removal of overlying bone. Images (c) and (d) demonstrate similar images from a different patient during left tympanomastoidectomy. The size and extent of cholesteatoma in each case was accurately predicted and superimposed onto the bone.

References [1] [2] [3] [4] [5]

[6] [7] [8] [9] [10] [11] [12]

M. Agus, A. Giachetti, E. Gobbetti, G. Zanetti, and A. Zorcolo, Real-time haptic and visual simulation of bone dissection, Presence 12 (2003), 110–122. M.P. Fried, J.I. Uribe, and B. Sadoughi, The role of virtual reality in surgical training in otorhinolaryngology, Current Opinion in Otolaryngology & Head and Neck Surgery 15 (2007), 163–169. M. Hadwiger, C. Sigg, H. Scharsach, K. Buhler, and M. Gross, Real-time ray-casting and advanced shading of discrete isosurfaces, Computer Graphics Forum 24 (2005), 303–312. S. Lakare and A. Kaufmany, Anti-aliased volume extraction, Proceedings of the Symposium on Data Visualization (2003), 113–122. P. Lehmann, G. Saliou, C. Brochart, C. Page, B. Deschepper, J.N. Vallée, and H. Deramond, 3T MR imaging of postoperative recurrent middle ear cholesteatomas, American Journal of Neuroradiology 30 (2009), 423–427. W.A. McNeely, K.D. Puterbaugh, and J.J. Troy, Six degree-of-freedom haptic rendering using voxel sampling, Proceedings of SIGGRAPH (1999), 401–408. D. Morris, C. Sewell, F. Barbagli, K. Salisbury, N.H. Blevins, and S. Girod, Visuohaptic simulation of bone surgery for training and evaluation, IEEE Computer Graphics and Applications 26 (2006), 48–57. B. Pﬂesser, A. Petersik, U. Tiede, K.H. Höhne, and R. Leuwer, Volume cutting for virtual petrous bone surgery, Computer Aided Surgery 7 (2002), 74–83. S. Roettger, S. Guthe, D. Weiskopf, T. Ertl, and W. Straßer, Smart hardware-accelerated volume rendering, Proceedings of the Symposium on Data Visualization (2003), 231–238. K. Salisbury and C. Tarr, Haptic rendering of surfaces deﬁned by implicit functions, ASME Dynamic Systems and Control Division (1997), 61–67. B. Tolsdorff, A. Petersik, B. Pﬂesser, a. Pommert, U. Tiede, R. Leuwer, and K.H. Höhne, Individual models for virtual bone drilling in mastoid surgery, Computer Aided Surgery 14 (2009), 21–27. G.J. Wiet, D. Stredney, D. Sessanna, J.A. Bryan, D.B. Welling, and P. Schmalbrock, Virtual temporal bone dissection: An interactive surgical simulator, Otolaryngology–Head & Neck Surgery, 127 (2002), 79–83.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-119

119

Acquisition of Technical Skills in Ultrasound-Guided Regional Anesthesia Using a High-Fidelity Simulator Jeffrey JH CHEUNGa, Ewen W CHENa, Yaseen AL-ALLAQa, Nasim NIKRAVANa, Colin JL MCCARTNEYa, Adam DUBROWSKIb, Imad T AWADa a Department of Anesthesia, Sunnybrook Health Sciences Centre b Sick Kids Learning Institute, University of Toronto

Abstract. Despite the increasing popularity of ultrasound-guided regional anesthesia (UGRA), structured training programs during residency are often lacking. The lack of a regional block area, lack of expertise, and lack of structured training programs have limited hands-on experience in residency programs. However, these constraints may be circumvented through the use of simulation. This observational study looked at the use of a high-fidelity simulator for training novice undergraduate students UGRA techniques. Despite some improvement in the second trial with the simulator, the ability to maintain visualization of their needle (p<0.05), align needle with probe (p<0.05), and angle their needle approach (p<0.05), as well as reduce needle passes (p<0.05) did not improve. The results show students had difficulty learning skills requiring more coordination and fine motor control. Keywords. Simulation, ultrasound, motor learning, feedback, education, regional anesthesia

Introduction UGRA has evolved over the last 6 years in facilitating the performance and improving the safety of peripheral nerve blocks [1]. It allows real-time identification of relevant anatomy and needle position, shortens time to perform, reduces the risk of systemic toxicity and increases the success of peripheral nerve block techniques [2-4]. Resident didactic training and competency in UGRA has been mandated recently by local, national and international societies.[5] Learning this new skill could be difficult and time consuming for most novice residents. Furthermore, limited exposure and training of UGRA, time constraints, as well as the ethical issues of practicing on patients, has generated a new interest of educators to shift training to the skills labs and simulation centers [6,7]. Novices’ attention resources are often maximized in the clinical setting, such as the operating room because of concurrent engagement in technical performance, clinical judgment, comprehension of instruction, and additional learning. Because their attention capacity may be exceeded, their ability to engage in monitoring errors and complications may be limited. Providing prior technical skill training may release some of the attention resources related to technical performance. The emergence of a multitude of simulation based training approaches in communications and team training suggest that similar principles may apply in surgery

120

J.J.H. Cheung et al. / Acquisition of Technical Skills in Ultrasound-Guided Regional Anesthesia

and anesthesia [8]. In a stress free environment, trainees can practice on bench models in a laboratory environment to acquire the basic ultrasound skills necessary for regional anesthesia application. This study aimed to observe and assess a high-fidelity simulator developed for learning UGRA. Specific learning goals and tasks were determined by a group of experts through the Delphi technique. The amount of feedback and the number of needle passes required to complete the simulation were used to determine where students encountered the most difficulty with the simulation.

1. Materials and Methods Upon Institutional Ethics Board approval, 26 undergraduate students from the University of Toronto were consented for this study. All students received a brief 15minute introductory lecture on the basics of ultrasound physics, imaging nerves, and how to visualize the needle when performing peripheral nerve blocks. Immediately following this lecture, under the supervision of an expert in UGRA, all participants were asked to perform several nerve block skill specific tasks under ultrasound guidance (38 mm broadband 13-6 MHz linear array ultrasound probe, SonoSite Inc. Bothell, WA) on a high-fidelity model (trial 1). The expert instructor was allowed to provide feedback and answer questions when students were having difficulties completing tasks. After students received their initial hands-on trial on the simulator (trial 1), they then received a more in-depth didactic teaching session given by an anesthesiologist expert in UGRA. The teaching involved a 30-minute power point presentation demonstrating physics of ultrasound, how the machine operates, principles of how to image nerves, and how to align the probe and the needle to visualize the needle tip. Again the students were given a second training session (trial 2) on the high-fidelity simulator. Students all received instruction and feedback from the same instructor for both simulator sessions. Figure 1 illustrates an example of captured video during a student trial on the high-fidelity model. 1.1. Simulator The high-fidelity model consisted of a pork shoulder with a superficial and deep beef tendon (¼ to ½ inches in diameter) inserted to simulate nerves when imaged with the ultrasound. The tasks and learning goals of this high-fidelity simulator were developed through the use of the Delphi technique by a group of experts in UGRA within the institution. The high-fidelity simulator was broken down into 4 tasks: 1) Locating the superficial and deep targets in a transverse axis. 2) Using ultrasound-guidance to hit the superficial target. 3) Using ultrasound-guidance to hit the deep target. 4) Imaging the superficial nerve in a longitudinal axis. The established learning goals focused on performing/learning UGRA in a proper and safe manner and included: 1) Locating Targets.

J.J.H. Cheung et al. / Acquisition of Technical Skills in Ultrasound-Guided Regional Anesthesia

2) 3) 4) 5) 6)

121

Centering targets on the screen Aligning needle with probe Proper angling of needle approach to hit target Keeping visualization of the needle Imaging the targets in a longitudinal axis

As well, the number of needle passes required to complete the tasks was deemed an important clinical measurement to be looked at. A needle pass was defined as the movement of the needle just below or out of the surface of the model to be reinserted.

Figure 1. Video recording screenshot of a high-fidelity simulator training session (left – ultrasound monitor; right – student and instructor). A – Visualized needle; B – superficial nerve; C – deep nerve.

1.2. Analysis and Statistics Each student’s simulator session was video recorded, looking only at hand motions and the ultrasound machine monitor to ensure anonymity and blinding. The videos were randomized and scored by observing the number of times verbal or physical feedback were provided to students, what learning goals the feedback related to, and how many needle passes were required to hit both the superficial and deep targets. Verbal feedback included any comments aside from the scripted commands from the supervising instructor to perform one of the 4 tasks. Physical feedback included events where the instructor physically gestured (pointing or touching target) or took control of the needle and or probe to aid in any one of the learning goals. Descriptive statistics were performed to summarize the overall data; a two-tailed ANOVA was completed to look at learning across trials. All analysis was performed with SPSS Statistics Version 17.0 (SPSS, Inc., Chicago, IL, USA); significance was set at p<0.05.

2. Results There was a significant learning effect detected across the two trials looking at the amount of feedback as a measurement of learning. Far less feedback was required for students to complete the simulation in trial 2 compared with trial 1 (p<0.001). Table 1 provides a summary of both trial 1 and trial 2 feedback and needle pass counts. Figure 2 provides a visual summation of feedback of both verbal and physical in trials 1 and 2.

122

J.J.H. Cheung et al. / Acquisition of Technical Skills in Ultrasound-Guided Regional Anesthesia

Table 1. Summary chart of verbal and physical feedback, and needle passes across trials 1 and 2. Task

Trial 1 Verbal Physical Feedback Feedback

Trial 2 Verbal Feedback

Physical Feedback

1) Locate Targets

93

26

28

8

2) Centre Targets on Screen

18

4

12

1

3) Align Needle with Probe

46

45

20

13

4) Angle of Approach

52

28

23

8

5) Keep Needle Visualized

70

13

46

16

6) Image Longitudinal Axis

34

14

7

1

No. Of Needle Passes

122

98

Figure 2. Cumulative frequency distribution of both verbal and physical feedback categorized by learning goals.

Students initially required the most feedback in locating the target structures. Combining both verbal and physical feedback, the total decrease in the frequency of feedback for locating the targets, aligning needle and probe, angling the needle, and imaging the longitudinal axis of the targets decreased by 70%, 64%, 61%, and 83% respectively. Looking at the individual learning goals of the simulator, according to verbal feedback counts, students significantly improved in locating the targets (p<0.01), aligning needle and probe (p<0.01), angling their needle approach (p<0.01), and in imaging the longitudinal axis of the targets (p<0.01). The physical feedback data as well shows a significant decrease in feedback in trial 2 for locating the targets (p<0.05), aligning needle and probe (p<0.005), angling their needle approach (p<0.05), and in imaging the longitudinal axis of the targets (p<0.001). No significant difference was

J.J.H. Cheung et al. / Acquisition of Technical Skills in Ultrasound-Guided Regional Anesthesia

123

found for the number of needle passes required for students to complete the simulation between trials 1 and 2. These data are summarized in Figures 3 and 4.

Figure 3. The average amount of verbal feedback students (n=26) received across trial 1 and 2 as well as the number of needle passes required to complete the simulation. The error bars represent the standard error of the mean, * indicates a statistically significant difference (p<0.05) between trials.

Figure 4. The average amount of physical feedback students (n=26) received across trial 1 and 2. The error bars represent the standard error of the mean, * indicates a statistically significant difference (p<0.05) between trials.

124

J.J.H. Cheung et al. / Acquisition of Technical Skills in Ultrasound-Guided Regional Anesthesia

3. Discussion The results suggest that novice students learning UGRA skills face the most difficulty with locating targets within the high-fidelity model, keeping the needle visualized, aligning needle with probe, and adjusting the angle of needle approach. There was a significant improvement seen from the 1st trial to the 2nd but not in all tasks. There was no significant difference between feedback requirements for centering targets and visualizing the needle according to both verbal and physical feedback. The lack of significant improvement in centering of the targets may be contributed to the fact that the students did not encounter this error at a very high frequency, even on their first trial with the model. This learning goal itself is more procedural in nature and almost intuitive. However, the lack of significant improvement in maintaining visualization of the needle, aligning needle and probe, and adjusting the angle of need approach, is likely due to these task requiring higher levels of skill and coordination that of a higher degree of complexity. The same can be said for the number of needle passes, which also did not decrease significantly across the two trials. In conclusion, the results demonstrate the different skills required for UGRA and a difference in their learning curves. Skills such as learning to properly image targets and center them can be learned with relative ease. However, there are more intricate skills such as visualizing the needle that may require more time and focus to develop, both on the educator’s side as well as the student’s. Though students may be able to become proficient in some tasks after a single exposure, other tasks may require more focus and practice in order to see improvement. Training programs and instruction should perhaps focus on the development of fine motor coordination for these tasks. The use of a simulator may benefit learning in terms of allowing sufficient time for trainees a competent level of motor coordination before practice in an actual clinical setting. Further work should be done on the learning curves of these particular motor skills in UGRA.

References [1] P. Marhofer, V.W. Chan, Ultrasound-guided regional anesthesia: current concepts and future trends, Anesth Analg 104 (2007), 1265-1269, tables of contents. [2] R. Brull R, M. Lupu, A. Perlas, V.W. Chan, C.J. McCartney, Compared with dual nerve stimulation, ultrasound guidance shortens the time for infraclavicular block performance, Can J Anaesth 56 (2009), 812-818. [3] B.D. Sites, M.L. Beach, B.C. Spence, et al., Ultrasound guidance improves the success rate of a perivascular axillary plexus block, Acta Anaesthesiol Scand 50 (2006), 678-684. [4] H. Willschke, P. Marhofer, A. Bosenberg et al., Ultrasonography for ilioinguinal/iliohypogastric nerve blocks in children, Br J Anaesth 95 (2005), 226-230. [5] B.D. Sites, V.W. Chan, J.M. Neal et al., The American Society of Regional Anesthesia and Pain Medicine and the European Society Of Regional Anaesthesia and Pain Therapy Joint Committee recommendations for education and training in ultrasound-guided regional anesthesia, Reg Anesth Pain Med 34 (2009), 40-46. [6] O. Grottke, A. Ntouba, S. Ullrich et al., Virtual reality-based simulator for training in regional anaesthesia, Br J Anaesth 103 (2009), 594-600. [7] Z. Friedman, N. Siddiqui, R. Katznelson, et al., Clinical impact of epidural anesthesia simulation on short- and long-term learning curve: High- versus low-fidelity model training, Reg Anesth Pain Med 34 (2009), 229-232. [8] A.G. Gallagher, E.M. Ritter, H. Champion et al., Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training, Ann Surg 241 (2005), 364-372.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-125

125

MeRiTS: Simulation-Based Training for Healthcare Professionals a

David CHODOSa,1, Eleni STROULIAa and Sharla KING b Department of Computing Science, University of Alberta, Canada b Educational Psychology, University of Alberta, Canada

Abstract. Simulation-based training has been used in numerous settings for procedural training. In this research, we focus on a method of simulation-based procedural skills training that uses virtual worlds. This method, implemented in our MeRiTS software system, models procedures using executable workflows, which are enacted by the trainee in a virtual world. The workflows may be defined by educators, or demonstrated by experts and then extracted from system logs. To demonstrate the utility of the system, we have created a scenario for training EMTs in patient rescue and transition procedures. We have pilot tested this scenario with students at a polytechnical institute, and will be conducting more rigorous testing with a range of students and institutions in the near future. Keywords. Virtual world, training, simulation

Introduction Simulation, in various forms, has long been used for training professionals in numerous disciplines, motivated by pedagogical theories promoting learning-by-doing. In the context of healthcare, this has taken forms such as mannequin-based training, standardized-patient scenarios, and desktop software. In this research, we are investigating the potential of virtual worlds (VWs), currently working with Second Life 2 , for training interprofessional healthcare teams in complex, in terms of the procedures they involve and the communication they require, scenarios. To this end, we have developed MeRiTS, a software system for modeling complex procedures so that they can be enacted in VWs. In this system, to simulate a scenario, one has (a) to design and model in the VW the relevant elements of the simulated physical environment, and (b) to specify a set of workflows and constraints, representing the behaviors of the dynamic artifacts in the world and the capabilities of the various health professionals participating in the scenario. Once the scenario is thus modeled, health professionals can log in, assume one of the modeled roles and participate in the scenario situation, bringing to bear their knowledge of proper clinical procedures and communication protocols. As they do so, their actions send messages to a workflow engine that keeps track of the in-world activities, based on the scenario models. Comparing the actual behavior against the various models, the engine can 1

Department of Computing Science, 221 Athabasca Hall, University of Alberta, Edmonton, Alberta, Canada, T6E 2J5; E-mail: [email protected] 2 http://www.secondlife.com

126

D. Chodos et al. / MeRiTS: Simulation-Based Training for Healthcare Professionals

recognize conformance and divergence from prescriptive and desired behaviors, on the basis of which it can generate feedback to the participants. The dynamic object and professional behaviour workflows are informally modeled by domain experts and software designers, in a collaborative fashion, using a visual tool. These visual models can be then “compiled” in a straightforward manner into WSDL/BPEL specifications. This approach, however, is less than ideal when the envisioned workflows are not well understood and formally specified; in some cases, the experts intuitively know what to do but cannot necessarily specify it. Through a more recent extension to MERiTS, we are exploring the potential of “workflow modeling by example”. In this situation, a domain expert can demonstrate the procedure by enacting it in the VW through her avatar; MERiTS records the demonstrator’s actions, and based on the (possibly multiple) recorded trace(s), the system can induce a workflow model. The constraints, meanwhile, are specified in terms of the modeled behaviours, using a web-based interface. This workflow-induction process relies on a general model of actions in VWs. This model plays three roles. First, it makes explicit the conceptual model of the activities that can be enacted in a VW. Second, it provides a layer of abstraction that can enable the integration of multiple VWs: as long as the actions in this model can be implemented in a given VW, then the modeled workflows can be also enacted in this world. Third, from an implementation point of view, it specifies the language in terms of which the recorded in-world traces are parsed, towards inducing workflow specifications. In this paper, we place our work on MERiTS in the context of related work (Section “Related Work”); we review in some detail the design of the MERiTS system (Section “MERiTS”); we discuss an initial pilot study we conducted at a local polytechnical institute to evaluate the system (Section “Results”); and we conclude with some early lessons learned and plans for the future (Section “Conclusions”).

1. Related Work The subject of using VWs for education and training has received an increasing amount of attention from the academic community in recent years, as VWs have become established in both mainstream culture and in educational institutions. Second Health3, a project at Imperial College, London, simulates several key points of care in a proposed model for the British healthcare system, including a hospital and a clinic. The project includes a number of virtual representations of healthcare-related objects, such as hospital beds, IV poles and crash carts, and allows the user to take actions such as checking the patient’s pulse and listening to the patient’s breathing. However, there is no intentional training component to the project. Rather, its goal is to provide stakeholders with an interactive view of a proposed healthcare model. Vergara [1] developed a virtual 3D multi-user environment to teach medical students about hematomas, by interacting with a virtual character, “Mr. Toma”. Several rigorous studies of the system's effectiveness have demonstrated that it is equally effective as conventional, paper-and-pencil education methods, and offers additional advantages, including the chance to collaborate with geographically dispersed students, and an increased sense of immersion when using the MUVE system. 3

http://secondhealth.wordpress.com/ Accessed Sep. 9, 2010

D. Chodos et al. / MeRiTS: Simulation-Based Training for Healthcare Professionals

127

Pulse!! [2] offers a Virtual Clinical Learning Lab, i.e., “an interactive virtual environment simulating operational health-care facilities, procedures and systems”. While the system was initially focused on a naval hospital setting, it has since been tested in the Yale School of Medicine and Johns Hopkins School of Medicine, and has licensed the technology to BreakAway Ltd., for commercial development. The Canadian Border Simulation project is another very good example of a practical and successful VW-based training program [3]. This project uses a virtual border crossing to augment the in-class role-playing experience that students receive, resulting in better grades and increased enthusiasm for the course. However, it should be noted that the virtual objects were created specifically for the border-crossing context, and are not extendable to other situations.

2. MeRiTS Informed by this research, we have created MeRiTS (Mixed Reality Training System), a training system that uses a VW client, along with a workflow engine, to teach students procedural and communication skills, by enacting various roles in complex multi-person scenarios. The process of using MeRiTS is described in the following paragraphs. At a high level, the scenario that is to be simulated is defined through a set of BPEL workflows, augmented by logical constraints. The workflows specify the behaviour of the interactive objects in the scenario, and thus also define the capabilities of the scenario participants. Complementing the workflows are constraints that define both what is possible (feasibility) and what is recommended (advisibility), with respect to the actions defined by the workflows. The workflows and constraints offer the student passive, non-restrictive guidance as they go through the scenario. That is, the students receive feedback but, as long as they do not violate feasibility constraints, they are not restricted in the actions (either correct or incorrect) that they can take. The student, meanwhile, interacts with the workflow through a VW client, thus experiencing the scenario in an immersive, collaborative environment. Figure 1 shows the principal components of the MeRiTS architecture. During the execution of the scenario, the user interacts with the interactive objects, which send messages to the workflow engine. The engine, in turn, processes the message and responds with a message conveying the results of the action, as per the interactive object’s workflow. At the same time, as the user ’s avatar moves and interacts with in-world entities and other avatars, it generates low-level action logging messages, which provide a detailed record of the user’s in-world actions. At a level of abstraction below that of the workflows and constraints, we would like to be able to understand the user’s actions in a VW-independent manner. Thus, in order to ensure that the user’s actions are comprehensible across a range of VW implementations, we have defined a model for parsing the user’s in-world actions, called the Avatar Capabilities Model (ACM). The structure and content of the model draws from work by Schank, et al., [4] on codifying behavior in terms of scripts and plans, while its non-verbal portion is based on work by Mehrabian [5]. The model is equally influenced by our experience in working in several VW environments [6], and the kinds of behaviors that we have observed. The model is presented in the following text box, and described more fully in the paragraphs that follow.

128

D. Chodos et al. / MeRiTS: Simulation-Based Training for Healthcare Professionals

Figure 1: MeRiTS Architecture

Action = <(Movement | Communication | Sensing | Object), Output> Movement = Move | Sit Move = Sit = Communication = Speak | Write | Non-Verbal Speak = Write = Non-Verbal = Sensing = Object = Create | Hold |Transfer | Take | Interact Create = Hold = Transfer = Take = Interact = Output = Message to Web Application Of course, for this model to be useful, the actions it describes must be executable by a character in a VW. By way of an example, Table 1 presents each action in the model, and the method of executing that action in Second Life. At the lowest level of abstraction, we need to record and parse the user’s actions in the VW, at a level of granularity appropriate for the ACM described previously. In order to create a complete record of the user’s in-world actions, we have employed some auxiliary tools to record, parse and analyze the user’s actions. The first of these tools is a wearable, in-world logger that captures all of the user’s actions, and saves them as entries in a database. However, these entries must be parsed before they can be used by other components of the system, for several reasons. Most critically, the data in movement entries must be converted from absolute locations (that is, {x,y,z} coordinates) to conceptual locations (e.g., “at the accident scene”). To deal with these issues, we have developed a second tool: a parser that analyzes the stored entries, extracts the relevant information, and stores it in a parsed action log. The process of an action being processed by the workflow, recorded by the logging tool, and parsed into an ACM action is shown in Figure 2.

D. Chodos et al. / MeRiTS: Simulation-Based Training for Healthcare Professionals

129

Table 1: Implementation of Actions in Second Life Action Move Speak Write Non-Verbal Sense (sight) Sense (hearing) Sense (other) Create Hold Transfer Take Interact

Implementation in Second Life Avatar movement, using arrow keys Text and voice chat, instant messaging Scripted objects that display text chat messages on an object’s surface Built-in and custom animated gestures Shift gaze using mouse, camera controls (zoom, rotate camera angle) An object which plays an audio file An object describing a sensation Create an item from one’s inventory Hold an item that one owns Give an item to another user Take an item into one’s inventory An object which uses dialog boxes

The parsed log, which contains compact, generalized representations of the actions taken by the user, is used as input to a pattern-discovery algorithm, thus allowing the user to identify patterns of behavior. Specifically, the action log (or, more precisely, set of logs) is analyzed by our implementation of the Apriori frequent-item set algorithm. The algorithm identifies the subsequences of actions that occur frequently among the action sequences. These common action sequences may be then analyzed for a variety of purposes, such as identifying differences between an expert’s execution of a process and that of a novice. Perhaps most interesting, however, is the potential for a more intuitive method of process definition. In cases where a process cannot be easily defined, an expert can demonstrate it several times in- world, the recorded traces can be parsed, and the pattern recognition algorithm employed to elicit the key actions in the process. Thus, the process can be defined implicitly through the action log, rather than relying on a pre-existing process definition.

Figure 2: Action Processing and Logging Tools

130

D. Chodos et al. / MeRiTS: Simulation-Based Training for Healthcare Professionals

3. Results Using the MeRiTS system, we implemented a training scenario for training emergency medical technicians (EMT) in victim treatment and transfer processes. To develop this scenario, we have created many interactive entities in Second Life, such as an ambulance, medical equipment, and the victim. Figure 3 shows a screenshot of the accident rescue scenario. We have also developed workflows that define the victim’s behavior, the functionality of relevant medical equipment, and the scenario itself. We are evaluating the EMT training scenario in several ways. As a first step, we conducted a pilot study with four first-year students at the Northern Alberta Institute of Technology (NAIT) in March 2010. The pilot was conducted with help from two emergency doctors and the associate chair of the paramedic program at NAIT. Students and instructors were given time in-world, before the pilot, in order to familiarize themselves with the VW interface. They were also provided with instructions on basic functionality. During the pilot itself, the students went through the scenario in pairs – a lead and an assistant. The scenario required the students to treat a victim at an accident scene, transfer the victim to an ambulance, and then bring them to a hospital, where they had a “hand-off” conversation with an ER doctor. This process, which was expected to take 5-10 minutes, was digitally recorded and supervised by an instructor from the paramedic program. Each student was given pre and post-study questionnaires, which elicited basic participant information, and their opinions on the quality and educational value of the scenario, respectively. The instructors were also given a questionnaire that elicited their impressions of the pilot. The pilot was, on the whole, quite successful, and the students were able to both have a meaningful experience with the simulation, and provide valuable feedback. The students felt immersed in the simulation, and were emotionally invested in the experience. Also, the experience of rescuing a victim, and then reporting the result of the rescue operation to an unfamiliar third party, was instructive and challenging.

Figure 3: Accident Scene in Second Life

D. Chodos et al. / MeRiTS: Simulation-Based Training for Healthcare Professionals

131

In addition to these outcomes, there were some unexpected benefits. One was the interaction between paramedic students and doctors. This interaction would have been difficult to arrange in person, but was facilitated by the commonly accessible virtual space. In this case, the students were in a computer lab at NAIT, one doctor was in a research lab at the University of Alberta, and the second doctor was at home. Another unexpected benefit for the students was the “physical interaction” with doctors in the ER. The experience of sharing this space was one that, again, would have been difficult to get in the real world, and was quite helpful for the students. There were also several shortcomings of the system identified through the pilot. One was that students were able to get into situations that were not covered by the model, and which caused the objects to behave in unexpected and unrealistic ways. Another challenge was the usability and intuitiveness of the interface, particularly with regards to the interactive objects. Although the students had an opportunity to practice before the pilot, many of them did not make use of this, and thus struggled in rescuing the victim, and in the handoff conversation. Although this issue could have been mitigated by improved training before the pilot session, it also indicates a need for improvement in the way that students interact with the objects in the simulation. Overall, both students and doctors felt that the system could improve the delivery of health education, and assist students’ understanding of interprofessional relationships and patient care. Also, one doctor noted that healthcare education is becoming more centralized, and this system could be used to deliver training in a standardized, accessible way, which was quite important for students in rural locations. We plan on conducting further studies in the coming months, with participants from several partner institutions, to rigorously analyze the educational benefits of the system. For example, integrating first- and second-year paramedic students at NAIT, and 4th year medical students into the same simulation session. This will allow us to rigorously analyze the educational benefits of the system, and provide a broad range of students a chance to try the MeRiTS-based scenario.

Acknowledgements We would like to thank colleagues from health education and medicine, particularly Mike Carbonaro, Andrew Reid, Ken Brisbin, and Mark McKenzie. Support for this research has been provided by Natural Sciences and Engineering Research Council (NSERC) and the Informatics Circle of Research Excellence (iCORE).

References [1] V., Vergara, T. Caudell, T. Goldsmith, Panaiotis, D. Alverson. Knowledge-driven Design of Virtual Patient Simulations, Innovate: Journal of Online Education, 5 (2008). [2] C.L. Mcdonald. The Pulse!! Collaboration: Academe & Industry, Building Trust. Presentation given at the 17th Medicine Meets Virtual Reality Conference, January 19-22, Long Beach, CA, (2010). [3] K. Hudson, K. deGast-Kennedy. Canadian Border Simulation at Loyalist College, Journal of Virtual World Research, 2, 1 (2009). [4] R. Schank, R.P. Abelson, R.P. Scripts, Plans, Goals and Understanding; An inquiry into human knowledge structures, Lawrence Erlbaum, New Jersey, 1977. [5] A. Mehrabian. Nonverbal Communication. Aldine-Atherton, Chicago, 1972. [6] D. Chodos, E. Stroulia, P. Kuras, M. Carbonaro, S. King. MERITS Training System: Using Virtual Worlds for Simulation-based Training, CSEDU 2010, April 7-10, Valencia, Spain, (2010), 54-61.

132

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-132

A Framework for Treatment of Autism Using Affective Computing Seong Youb CHUNGa and Hyun Joong YOONb,1 Department of Mechanical Engineering, Chungju National University, Republic of Korea b Faculty of Mechanical and Automotive Engineering, Catholic University of Daegu, Republic of Korea a

Abstract. It is known that as many as 1 in 91 children are diagnosed with autistic spectrum disorder. Since the children with autism usually do not express their own emotional status, it is needed to develop a novel technology to sense their emotional status and give proper psychological treatment. This article presents a framework of the treatment system for children with autism using affective computing technologies. Keywords. Affective computing, emotion sensing, autism, autistic spectrum disorder

Introduction The children with autism, which is also called as a pervasive developmental disorder, show problems in social relationships and interactions such as difficulty in verbal expression-understanding, limited interests in activities or play, and difficulty in mixing with other children [2]. The well known effective and relevant treatments of autism in children include educational programs, reinforcement of mutual communication, diet control, auditory training, behavior modification, medication, music therapy, physical therapy, occupational therapy, sensory integration, and vision therapy. Finding the most relevant treatment for a certain child with autism, however, is not easy since the autistic children need personalized treatment according to the types and the severity of symptoms. Recent affective computing technology, focused on emotional interactions between humans and computers, can provide one of the most promising treatment methods for the children with autism, since it enables the emotion recognition through bio-sensors and it can provide the children with effective virtual training scenarios using virtual affective agent and haptic technologies. Thus, this article presents a framework of the treatment system for children with autism using the affective computing technologies.

1

Corresponding Author.

S.Y. Chung and H.J. Yoon / A Framework for Treatment of Autism Using Affective Computing

133

1. A Framework for Treatment of Autism Figure 1 shows the framework for the treatment system of children with autism using bio-sensors and affective computing technologies. Immerged in virtual treatment environment with the head mount display, a child with autism controls his or her avatar using haptic devices. Agents in virtual environments give the composition of the stimulus, which includes physical stimulus stimulated through the haptic devices, to the virtual avatar according to the treatment scenario to cause a particular emotional situation. The stimulus is determined based on the affective model to express emotion generation process of human. The emotional status of the child is recognized through the series of processes of measuring bio-signals such EEG (electroencephalogram), ECG (electrocardiogram), BVP (blood volume pulse), respiration, SC (skin conductance), EMG (electromyogram), and SKT (skin temperature), noise filtering, feature extraction, and pattern matching. The treatment system aids a clinician to diagnose whether the child is emotionally disturbed by presenting the statistical analysis results compared with expected normal emotional states. The proposed treatment system for the child with autism consists of three components. •

Virtual interactive environment component. The virtual interactive environment component is intended to implement the treatment scenario for the avatar that interacts with virtual agents in a hypothetical situation. As each virtual agent has its unique affective model, the treatment scenario changes dynamically according to the interaction between the avatar and the virtual agents. This dynamic change helps the child with autism improve interaction with people in real world.

•

Bio-sensing component. The bio-sensing component consists of hardware for sensing bio-signals such as EEG and ECG, and software for signal processing. The emotional status expressed by the user is recognized through the biosensing component.

•

Virtual environment interface component. The virtual environment interface component is intended to enhance the immersive realization in virtual environment. It includes hardware such as HMD (head mounted display) and haptic interface device, and control software.

In Figure 1, a video tape records the non-bio-signals such as the voice, behavior, and facial expression of the child with autism during the treatment and is reported to the clinician. It is used as an auxiliary mean to diagnose the treatment.

2. Discussion This article presents a new affective computing system framework for treatment of children with autism. The proposed method provides the treatment environment for autism by combining virtual reality technology, which has already been applied in the field of psychotherapy [1], with the emotion recognition through bio-sensing and virtual agent modeling technologies. Through this framework, objective diagnosis and continuous treatment in one’s daily life for autism will be possible if the bio-sensing system becomes compact and the emotion recognition is reliable.

134

S.Y. Chung and H.J. Yoon / A Framework for Treatment of Autism Using Affective Computing

Figure 1. The system framework for treatment of autism using affective computing.

References [1] A.A. Rizzo, D. Klimchuk, R. Mitura, T. Bowerly, G.B. Buckwalter, K. Kerns, K. Randall, R. Adams, P. Finn, I. Tarnanas, C. Sirbu, T.H. Ollendick, and S.C. Yeh, A virtual reality scenario for all seasons: The virtual classroom, Proceedings of the 11th International Conference on Human Computer Interaction, Las Vegas, USA, July 22-27, 2005. [2] B.D. Zager, Autism Spectrum Disorders Identification, Education, and Treatment, Mahwah, N.J. : Lawrence Erlbaum Associates, 1999.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-135

135

Modification of Commercial Force Feedback Hardware for Needle Insertion Simulation Timothy R COLES a, b,1, Nigel W. JOHN a, Giuseppe SOFIAb, Derek A. GOULD c, Darwin G CALDWELL b a Bangor University b Istituto Italiano di Tecnologia c Royal Liverpool University NHS Trust

Abstract. A SensAble Omni force feedback device has been modified to increase the face validity of a needle insertion simulation. The new end effector uses a real needle hub and shortened needle shaft in place of the Omni’s pre-fitted pen shaped end effector. This modification facilitates correct procedural training through the simulation of co-located visual and haptic cues in an augmented reality approach to simulation. The development of the new end effector is described and a pictorial guide to its manufacture and the fitting process is provided. Initial results from face validation studies bode well for the fidelity of this low cost device. Keywords. Needle Insertion, Simulation, Haptic, Hardware, Modification.

Introduction One of the most common tasks in many medical procedures is to insert a needle into a patient. There have been examples of purpose built haptics devices for needle puncture, such as the CathSim for intravascular catheterisation [1]. A detailed review can be found in [2] and [3]. However, most solutions use commercial off-the-shelf force feedback devices such as those in the SensAble Technologies (Woburn, USA) Phantom range. Here, the default stylus tool held by the user replaces the real needle within the simulation. Although this can be an acceptable solution, the face validity is low. We have built an augmented reality (AR) simulation to train palpation and needle puncture of the femoral artery [4]. These are the first steps in many Interventional Radiology (IR) procedures. The simulation co-locates visual and haptic feedback through the use of an AR video see-through visualisation, whilst requiring no headwear to be worn [5]. The simulation also offers shadowing of the user’s real hand in the virtual world for increased depth perception, textured deformable tissue and visually realistic cloth. Realistic haptic feedback is provided throughout the palpation; tactile feedback via a hydraulic end effector and force feedback via a modified Novint (Albuquerque, USA) Falcon force feedback device. This feedback is based on in-vivo measured force and tactile data. 1

Corresponding Author: Timothy R Coles, Istituto Italiano di Tecnologia, 16163, Genova, Italy; E-mail: [email protected].

136

T.R. Coles et al. / Modiﬁcation of Commercial Force Feedback Hardware

Figure 1. A) Omni force feedback device. The cover of the second armature has been removed to reveal the two bearings and washer assembly through which the Y shaped arm is inserted. The top encoder must be detached and the arm slid out with careful application of force. B) New end effector wrist, designed in Pro Engineer and printed on a 3D plastic printer. Single armed structure gives good camera visibility in AR application. Off the shelf potentiometer replaces the Omnis third encoder. C) The new wrist end effector is slid up into the arm and re-wired. Final encoder is to be placed back into the device. D) A visual comparison of original and modified Omni devices. E) A real needle can be inserted into the new end effectors holster and secured by tightening three small screws. Real needle hub appears in the augmented reality visualisation.

As the practitioner locates the simulated femoral artery, a virtual needle, represented physically by a force feedback device with a 6 degree of freedom (DOF) end effector, is picked up and inserted into the virtual patient at the desired location / orientation, whilst feeling simulated force feedback. SensAble’s Omni force feedback device that provides 3DOF force feedback and 6DOF sensing was chosen to simulate this task due to its adequate force feedback capabilities and low cost. However the Omni’s standard stylus end effector neither looks like a real needle hub during the augmented reality visualisation, nor does it provide the correct tactile cues as it is grasped between the trainee’s fingertips, providing low face validity during simulation. The Omni’s force feedback end effector has been modified to overcome this limitation.

1. Device Design For simple integration into existing and new applications and inter-changeability between modified and non-modified Omni devices, the kinematics of the new end

T.R. Coles et al. / Modiﬁcation of Commercial Force Feedback Hardware

137

effector must match those of the existing device. Ideally, a real needle will be used to provided realistic tactile feedback and enable a wire to be passed through its centre. The device’s linkages must not obscure the AR cameras line of sight to the user’s hand during normal usage. The end effector has been re-designed from its wrist to end effector tip. This section of the device controls the end effectors un-actuated roll, pitch and yaw with the remaining 3DOF providing the force from within the devices base. To perform this modification, two covers on the Omnis second arm must first be removed to reveal the shaft of the rotating Y shaped wrist Fig.1(A). The wires must now be cut to allow the wrist to be removed from the arm by sliding (with force) the arm out through two bearings which hold the rotating Y shape structure. The replacement wrist Fig.1(B) has been designed in Pro Engineer and manufactured on a 3D plastic printer. Two of the three original potentiometers within the device can be easily removed without damaging the original device’s structure. The third, contained within the section fitted with a microphone style jack was not broken open so as not to permanently damage the device. In place of this potentiometer an off-the-shelf replacement has been used Fig.1(B). Fitting the new end effector is performed by reversing the removal process.

2. Results and Conclusions A modified end effector has been designed, manufactured and fitted to three Omni force feedback devices Fig.1(D,E). Preliminary feedback from interventional radiology practitioners is extremely positive and the device has been integrated into the described Seldinger training simulation [4] for a full simulator evaluation. The tactile fidelity inherent with using a real needle hub through which the penetration is conveyed is expected to heighten a user’s presence whilst they rehearse the needle insertion. Although this a modification marginally increases the net cost of the force feedback device, this is thought to be outweighed by the increased fidelity of the simulation. A simple guide is presented here to allow others to carry out such a modification. In conclusion, initial results from face validation studies bode well for the fidelity of this low cost force feedback device. Full end user feedback will be given in the poster presentation.

References [1] Ursino, M., Tasto, J.L., Nguyen, B.H., Cunningham, R., Merril, G.L. “Cathsim: An Intravascular Catheterization Simulator on a PC,” Stud Health Technol Inform 62 (1999). pp 360-366. Published by IOS Press. [2] T. Coles, D. Meglan, and N. John, “The Role of Haptics in Medical Training Simulators: A Survey of the State-of-the-Art,” Haptics, IEEE Transactions on, (2010). Available in advance of print DOI: 10.1109/TOH.2010.19 [3] T.R. Coles, and N.W. John, “The Effectiveness of Commercial Haptic Devices for Use in Virtual Needle Insertion Training Simulations,” in Advances in Computer-Human Interactions, 2010. ACHI '10. Third International Conferences on, (2010), pp. 148-153. [4] T.R. Coles, D.A. Gould, N.W. John and D.G. Caldwell, “Integrating Haptics with Augmented Reality in a Femoral Palpation and Needle Insertion Training Simulation”, IEEE Transactions on Haptics. In submission. [5] T.R. Coles, N.W. John, D.A. Gould, and D.G. Caldwell, “Virtual Femoral Palpation Simulation for Interventional Radiology Training,” in Theory and Practice of Computer Graphics (2010), pp 123-126.

138

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-138

Visualization of Pelvic Floor Reflex and Voluntary Contractions Christos E CONSTANTINOUa1, Daniel KORENBLUMa, Bertha CHENb Spinal Cord Injury Center, Palo Alto Veterans Administration, Palo Alto, CA b Gynecology and Obstetrics, Stanford University Medical School, Stanford, CA a

Abstract. Visualization of the geometric deformation and associated displacement patterns of tubular abdominal organs to mechanical stimuli provides a quantitative measure that is useful in modeling their elastic properties. The origin of the stimulus may be the result of direct and voluntary muscle contraction or in response to a triggered reflex activity. Using trans-perineal 2D ultrasound imaging we examined the characteristics of deformation and displacement of these organs in response to voluntary activity, contraction, straining, and fast reflex responses to stimuli such as coughing. The relative time sequence in movement was examined by serially segmenting the outline of these structures and mapping their temporal characteristics. Keywords. Bio-imaging, Ultrasound, Segmentation, Image analysis.

Introduction Anatomically, taken as a group, Pelvic Floor Muscles (PFM) contributes to a variety of functions ranging from the mechanical support of abdominal contents against gravity to conception, delivery, continence and sexual function. Consequently their response varies according to the purpose demanded and can be voluntary or triggered by involuntary reflex reactions [1, 2]. Visualization of the deformation and displacement patterns of the organs contained by the PFM to mechanical stimuli provides a quantitative way to determine their elastic properties and the basis for simulation and modeling of a large number of multiple functional characteristics [3]. The effectiveness of PFM in fulfilling these roles depend greatly on the biomechanical characteristics of the various structures contained therein, parity, aging and hormonal status. Using ultrasound imaging, sequences of complex movements can be captured segmented and analyzed quantitatively to examine the characteristics of deformation and displacement produced [4]. Reflex induced activity is invariably fast and the motion of different organs takes place within seconds rendering biomechanical analysis challenging [5]. Using imaging we examined the kinematic characteristics of the bladder, urethra and rectum in response to coughing, levator muscle contraction and straining. The relative time sequence in movement was examined by serially segmenting the outline of these structures frame-by-frame, mapping the direction and path of the multiple organs. A 1

Corresponding author: Christos E Constantinou, Spinal Cord Injury Center, Palo Alto Veterans Administration, 640/128 94305, USA. E-mail [email protected]

C.E. Constantinou et al. / Visualization of Pelvic Floor Reﬂex and Voluntary Contractions

139

novel 3D representation of the temporal and spatial distribution of closure is presented and an associated Webb page illustrates the dynamic characteristics of the observations.

1. Methods Subjects for this study were asymptomatic volunteers recruited specifically for evaluation of their pelvic floor function following a protocol approved by the local IRB panel. Two dimensional ultrasound imaging was done using a GE at 7MHZ system with a curvilinear probe held at the perineum. Digitization and storage of video sequences was initiated for a period of 3sec duration at rest for each of the following three actions: [a] Voluntary contraction of PFM, [b] straining, [c] coughing. During these actions digitized video was stored on disk. Audio signals were also simultaneously recorded and used to identify details of activity. Video segments were edited to select appropriate quality of data off line and stored for analysis. Analysis was done by identifying and segmenting the outline of the bladder, urethra, symphysis pubis and AnoRectal Junction (ARJ). Displacement curves were subsequently computed between ARJ and urethra. These curves measure the change in the distance between two anatomical structures relative to the resting state. Each curve, representing an anatomical structure, can be compared to another curve along each point of its length where the opposing structure exists on a direct line of contact, allowing the transmission of mechanical forces between the two structures. Each line connecting the two structures defines a pair of points, one on each structure. The set of all pairs of points is used in constructing the displacement curves. To identify these pairs of opposing points along each curve, we defined a central axis dividing the ARJ and the urethra. The first step in computing this axis was to convert the ultrasound image into a mask using thresholding and morphological image processing operations. The right-edge of this mask defined the input to the fitting function which computed a 2nd-order polynomial fit. The displacements between the ARJ and the urethra were computed along the points of this central axis for which there were opposing pairs of points on the segmented ARJ and urethral lines joined by a line perpendicular to this axis. For this patient, we were able to use a single parameter to define this curve; in general, we will need two parameters for more complex anatomies. Segmentations of the anterior and posterior surface of the urethra were also done and the Urethral Diameter Profile was computed. Analogously the Vaginal Diameter Profile VDP was constructed and represented so that compression or expansion can be illustrated with respect to time. For this patient’s anatomy, a one-dimensional function y (x) was sufficient to define the urethral axis, using a straight-line fit of the segmentation at each frame. The distances between the structures along the axis were computed. We ignored points on the segmentations that did not occur as pairs, connecting the two structures by a line perpendicular to the axis. In general, we will need two dimensional, parameterized curves (x(s), y(s)) to define the central axes for more complex anatomies. Data from the

140

C.E. Constantinou et al. / Visualization of Pelvic Floor Reﬂex and Voluntary Contractions

UDP and VDP was subsequently computed to derive the strain variation within these two structures relative to the stimulus. Animated illustrations of the biomechanical parameters can be viewed in the authors Web page http://danielkorenblum.com/cconst 2. Results Illustrated in Figure 1 are ultrasound images showing the principal abdominal organs subject to displacement to pelvic floor contraction and Valsalva. Frame by frame segmentation shows the temporal sequence of displacement of ARJ, bladder, and urethra. To identify the timing of each frame, the segmentation lines were color coded, showing the comparative displacements of bladder and posterior rectal surface and the corresponding variation within the urethra.

Figure 1. Ultrasound video frame of a continent patient performing a contraction and a Valsalva; segmentation curves are superimposed in the foreground with colors varying over time, and measured displacements are shown as colored regions behind these. Dark blue lines in the background, running top-tobottom, depict the central axes computed for the vagina and the urethra. Vaginal displacements were measured between the ARJ and the urethra. Urethral displacements were measured between the dorsal and ventral sides of the urethra.

On the basis of these measurements a 3D representation was constructed to dynamically show the Vaginal and Urethral and Profile. Figure 2 shows the characteristics of the two profiles in response to a voluntary contraction while the specific video sequence of the process is given by: http://danielkorenblum.com/cconst/contraction

C.E. Constantinou et al. / Visualization of Pelvic Floor Reﬂex and Voluntary Contractions

141

Figure 2 Visualization of vaginal (top) and urethral (bottom) displacements vs. time occurring over a period of approx. 5.0 sec. The measured displacement of the segmentations defining the structures was plotted as a surface with changes in width along the central axis shown in centimeters on the vertical scale. The distance along the central axis where the displacement was measured is shown on the ordinate and time is shown on the abscissa. Blue corresponds to compression and red corresponds to stretching. Zero change in width is shown as a transparent gray plane. The surfaces show that the urethra was compressed along its entire length, with the amount of stretching increasing near the bladder while the vagina was stretched. The graded scale of diameter change, Δ w, is illustrated where red corresponds to compression and blue stretching.

Compared to the process of actively recruiting a voluntary pelvic floor contraction, the abdominal organs also respond passively to voluntary straining. To capture the differences between these two actions on the profiles, the corresponding schema given above for contraction was constructed and is shown by Figure 3.

Figure 3 Visualization of vaginal and urethral displacements vs. time occurring over a period of approx. 5.0 sec during a Valsalva. The measured displacement of the segmentations defining the structures was made using the schema of Figure 2. Surfaces of urethra was stretched along its entire length for the duration of the Valsalva.

142

C.E. Constantinou et al. / Visualization of Pelvic Floor Reﬂex and Voluntary Contractions

The video sequence of the process can be found in http://danielkorenblum.com/cconst/contraction The results so far demonstrated the response of the displacement of the vagina and urethra consequent to voluntary actions. The transient effect of the cough-induced response results in an increase magnitude of urethral compression while the response of the vaginal structures is more time dependent by comparison and stretching is minimized. Figure 4 and the video sequence http://danielkorenblum.com/cconst/cough show the corresponding segmented sequence.

Figure 4 Reflex response due to a cough. Visualization of vaginal and urethral displacements vs. time occurring over a period of approx. 2.5 sec resulting from a cough. The measured displacement of the segmentations defining the structures was using the same schema as Figure 2. The surfaces show that the urethra was highly compressed along its entire length. Maximum amplitude of compression can be seen at the base of the bladder.

3. Conclusions Using this approach it was possible to identify the nature of the, UDP and VDP, reflecting the biomechanical influence of PFM in the kinematic response of the urethra, bladder using ultrasound imaging. Visualizations of the temporal sequence of urethral closure were generated on the basis of the active reflex reaction of the anatomical displacements such as coughing as well as the passive response to voluntarily initiated actions such as straining and contractions. Currently the results apply on the normal response of asymptomatic subjects. Consequently their response varies according to the purpose demanded and can be voluntary or triggered by reflex reactions [2]. In this presentation identification made of the biomechanical factors involved in the kinematic response of some of the major contained structures, bladder, urethra and rectum using ultrasound imaging. The advantage of this visualization approach is to examine in slow motion the effect of active reflex reactions on the anatomical displacements such as coughing as well as the passive response to voluntarily initiated actions such as straining contractions.

C.E. Constantinou et al. / Visualization of Pelvic Floor Reﬂex and Voluntary Contractions

143

The application of such visualizations, done under control conditions afforded by the use of asymptomatic volunteers, can be used to lay the groundwork for more extensive modeling and virtual reality constructs where parameter such as tissue properties can be used [7]. Furthermore in such a model the impact of virtual surgery using new material or configurations of supporting mechanisms such as meshes can be explored. In the future the results generated from such visualizations will focus primarily on the normal response of asymptomatic subjects and some the differences in subjects with urinary incontinence. The influence of posture in considering the results will be demonstrated in terms of new parameters developed specifically for these studies. Distinction will be made between the visualization of pelvic floor dynamics measured using imaging and the vaginal force measurements using a probe. Controversies surrounding the strengths and weaknesses of each type of measurement will be illustrated using video presentations.

Acknowledgements Study supported by NIH grant R01 EB006170.

References [1]

[2] [3]

[4]

[5] [6]

[7]

Haylen BT, de Ridder D, Freeman RM, Swift SE, Berghmans B, Lee J, Monga A, Petri E, Rizk DE, Sand PK, Schaer GN. Intern Urogynecological Association (IUGA)/International Continence Society (ICS) joint report on the terminology for female pelvic floor dysfunction. Int Urogyn J Pelvic Floor Dysfunction. 21(1):5-26 2010. Constantinou CE. Dynamics of Female Pelvic Floor Function Using Urodynamics, Ultrasound and Magnetic Resonance Imaging. Eur J Obstet. Gynecology 144S: 159–165, 2009. Rahmanian S, Jones R, Peng Q, Constantinou CE: Visualization of Biomechanical Properties of Female Pelvic Floor Function Using Video Motion Tracking of Ultrasound Imaging. In: TECHNOLOGY AND INFORMATICS, 132:390-395, 2008. Jones R, Peng Q, Stokes M, Humphrey VF, Constantinou CE. The Mechanisms of Pelvic Floor Muscle (PFM) Function and the Effect on the Pelvic Floor and Urethra during a Cough. Europ J Urol 57:11011110, 2010. Constantinou CE, Govan DE. Spatial distribution and timing of transmitted and reflexly generated urethral pressures in the healthy female. J. Urol 127:964-969, 1982. Shishido K, Peng Q, Jones R, Omata S, Constantinou CE: Influence of Pelvic Floor Muscle Contraction on the Profile of Vaginal Closure of Continent and Stress Urinary Incontinent Women J Urol 179:191722, 2008. Hasegawa S, Yoshida Y, Wei D, Omata S, Chen B and Constantinou CE. Simulation of vaginal wall biomechanical properties from pelvic floor closure forces map MMVR18, 2011.

144

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-144

Mixed Virtual Reality Simulation -Taking Endoscopic Simulation One Step Further O. COURTEILLEa,1, L. FELLÄNDER-TSAIb, L. HEDMANb,c, A. KJELLINb, L. ENOCHSSONb, G. LINDGRENb and U. FORSa a Dept LIME, Karolinska Institutet, Stockholm, Sweden b Department CLINTEC, Center for Advanced Medical Simulation, Karolinska Institutet and Karolinska University Hospital, Stockholm, Sweden c Department of Psychology, Umeå University, Sweden

Abstract. This pilot study aimed to assess medical students’ appraisals of a “mixed” virtual reality simulation for endoscopic surgery (with a virtual patient case in addition to a virtual colonoscopy) as well as the impact of this simulation set-up on students’ performance. Findings indicate that virtual patients can enhance contextualization of simulated endoscopy and thus facilitate an authentic learning environment, which is important in order to increase motivation. Keywords. Mixed Reality, Virtual Patient, Surgical Simulation, Contextualized learning

Introduction Acquisition of endoscopic skills could be optimized in a highly realistic, engaging and immersive VR simulator environment. In fact, empirical studies have already shown that simulation technology and virtual patients support more individualized, situated and contextualized learning paradigms [1, 2, 3]. Research has also shown that the learning curve can be substantially enhanced and shortened for motivated trainees [4]. However, little attention has been focused on exploring the potential benefits of delivering combined simulation methods during surgical training sessions [5].

1. Objectives The main objective of this study was to assess medical students’ appraisals of a “mixed” virtual reality simulation for endoscopic surgery by exploring the potential benefits of this contextualized learning experience (like engagement and motivation). A secondary aim was to assess the impact of this simulation set-up on students’ performance.

1

Corresponding Author. Dr .Olivier Courteille, Karolinska Institutet, Dept of Learning, Informatics, Management & Ethics (LIME), 171 77 Stockholm, Sweden. Email: [email protected]

O. Courteille et al. / Mixed Virtual Reality Simulation

145

2. Methods and Materials 37 fourth-year medical students participated in a self-controlled study design with two combined simulation methods (Virtual Patient (VP) case and Virtual Colonoscopy simulator). Participants filled in an exit questionnaire about their appraisals (perception, motivation and attitudes) of the mixed virtual reality simulations (Fig 1). The impact of the sequence order of the combined simulations was also measured. Psychometric scales were used to assess self-efficacy before the simulation session 1 -to control for possible bias in student performance- and engagement immediately after each simulation session. Control Group (n=19)

Experimental Group (n=18)

Self-Efficacy Questionnaire

Self-Efficacy Questionnaire

Session 1 Virtual Colonoscopy

Session 1 Virtual patient Psychometric Measurements

Session 2 Virtual patient

Session 2 Virtual Colonoscop Psychometric Measurements

Exit Questionnaire

Figure 1. Design of the study

3. Results Analysis of inter-group homogeneity for controlling self-efficacy confirmed that no bias on student performance could be observed. As can been seen in Table 1, the participants’ overall opinion about the added pedagogical value and experienced usefulness of the mixed simulations was overwhelmingly positive (97%). To a very large extent, the mixed reality simulation was perceived as an enriched, contextualized and beneficial learning experience. Overall, the synergy between the two simulations was perceived as effective for the experimental group and a strong wish for the control group. A significant increase of engagement (p=0.008) was measured for females in the experimental group (i.e. VP training prior to endoscopic simulation) with potential positive learning effect as a result. Correspondingly, the reversed order led them to a significant decrease of engagement. Finally, a trend for positive effects on time to accomplish the endoscopic task was also observed for the VP-trained group.

146

O. Courteille et al. / Mixed Virtual Reality Simulation Table 1. Summary of the pedagogical evaluation

Survey Questions What is your opinion about this teaching method? How did you experience this [combined] way of solving clinical problems? Do you think these simulation methods can complement each other?

Group

Positive Answer

Negative Answer

Don’t Know

EG

17

1

0

CG

19

0

0

EG

17

1

0

CG

18

1

0

EG

18

0

0 1

CG

18

0

Mean

Mean

Mean

Values All

Males

Females

a) EG only: How did the synergy work between the virtual patient and the endoscopy simulator?

3.8

4.2

3.6

1.2

from ”not at all” to ”very well”

b) CG only: Had you preferred having some kind of synergy between the endoscopy simulator and the virtual patient?*

3.7

3.8

3.5

1.1

from ”not at all” to ”yes, very well”

Synergy between VP and VC

SD

Scale Min-Max 1-5

1-5

EG: Experimental group (n=18), CG: Control group (n=19) * In the VP session, the colonoscopy test was not performed by the trainee but was merely delivered as a text-based result

4. Conclusion The students experienced the whole session as an authentic, innovative and effective blended learning experience with a taste of augmented reality. They also expressed overall a wish for an increase use of similar individualized and contextualized learning environments for surgical novices. In conclusion, this study indicates that VPs might enhance contextualization of simulated endoscopy and presumably will facilitate an authentic learning environment, which is important in order to increase motivation.

5. Acknowledgment This study was supported by research grants from Karolinska Institutet.

References [1] Quinn, C. N. Engaging Learning: Designing E-Learning Simulation Games. Pfeiffer, San Francisco, 2005. [2] Cook D & Triola M (2009). Virtual patients: a critical literature review and proposed next steps. Medical Education, 43 (2009), 303-311. [3] Rosen K, McBride J & Drake R.The use of simulation in medical education to enhance students’ understanding of basic sciences. Medical Teacher 31(2009), 842–846. [4] Schlickum MK, Hedman L, Enochsson L, Kjellin A, Felländer-Tsai L. Systematic Video Game Training in Surgical Novices Improves Performance in Virtual Reality Endoscopic Surgical Simulators: A Prospective Randomized Study. World J Surg 33 (2009), 2360-2367. [5] Ellaway R, Topps D, Lachapelle K & Cooperstock J. Integrating Simulation Devices and Systems. Medicine Meets Virtual Reality 17 (2009), 88-90.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-147

147

A Serious Game for Off-Pump Coronary Artery Bypass Surgery Procedure Training Brent COWANa, Hamed SABRIa, Bill KAPRALOSa,1, Fuad MOUSSAb, Sayra CRISTANCHOc, and Adam DUBROWSKId,e a

Faculty of Business and Information Technology, and Health Education Technology Research Unit, University of Ontario Institute of Technology. Canada. b Division of Cardiac & Vascular Surgery, Schulich Heart Centre, Sunnybrook Health Sciences Centre. Canada. c Department of Surgery and Centre for Education Research & Innovation, Schulich School of Medicine & Dentistry, University of Western Ontario. Canada, d The Hospital for Sick Children Learning Institute. Canada. e Department of Pediatrics, Faculty of Medicine, and The Wilson Centre, University of Toronto. Canada.

Abstract.We have begun development of an interactive, multi-player serious game for the purpose of training cardiac surgeons, fellows, and residents the series of steps comprising the Off-Pump Coronary Artery Bypass grafting (OPCAB) surgical procedure. It is hypothesized that by learning the OPCAB procedure in a “first-person-shooter gaming environment”, trainees will have a much better understanding of the procedure than by traditional learning modalities. The serious game will allow for simulation parameters related to levels of fidelity to be easily adjusted so that the effect of fidelity on knowledge transfer can be examined. Keywords.Off-pump coronary artery bypass surgery, serious games, virtual simulation, game-based learning, fidelity, levels of realism, knowledge transfer.

Introduction The Off-pump Coronary Artery Bypass grafting (OPCAB) cardiac surgical procedure allows for a blocked coronary artery to be bypassed with a healthy artery or vein from another part of the body while the heart is beating in contrast to traditional coronary artery bypass grafting (CABG) which requires the use of a cardiopulmonary pump. Eliminating the cardiopulmonary pump may lead to a reduction in the number of postoperative complications. Despite the benefits associated with OPCAB and more specifically, its non-reliance on the cardiopulmonary pump, the OPCAB procedure itself is complex and technically challenging. It has been suggested that “appropriate training” be provided before operating on patients [1]. However, the nuances, techniques, problem solving, and trouble shooting that surrounds this procedure are primarily acquired in the operating room and this leads to increased resource consumption (e.g., monetary, faculty time, etc.). We have developed a prototype of an interactive, multi-player serious game for the purpose of training a series of steps comprising the OPCAB procedure. The goal is for the user/trainee to successfully complete the procedure, focusing on the cognitive and

148

B. Cowan et al. / A Serious Game for OPCAB Surgery Procedure Training

decision making processes related to the order in which steps are performed and on the tools required to perform each step, as opposed to the technical aspects, while minimizing the time to complete the procedure and maximizing the score (points are either added or taken away based on the trainee's actions). We take a situated learning approach [2], focusing on high fidelity and more specifically, rendering of very accurate graphical models, and auditory cues. The serious game is being developed as a research platform that will allow levels of graphical and auditory fidelity to be easily modified so that the effect of fidelity on knowledge transfer can be examined.

1. Background Virtual simulation offers a viable substitute to practice in an actual operating room, giving residents the opportunity to train until they reach a specific competency level. Although virtual reality-based technologies have been incorporated in the teaching and training curricula of a large number of professions across various industries (including surgery) for several decades, the rising popularity of video games has seen a recent push towards the application of video game-based technologies to teaching and learning. Serious games, or games that are used for training, advertising, simulation, or education, inherently support experiential learning by providing students with concrete experiences and active experimentation [3]. They leverage the advances made in the video game realm along with the growing popularity of video games, particularly with today’s generation of students/learners, to overcome some of the problems and limitations associated with traditional teaching methods [5]. Fidelity denotes the degree of similarity between the training situation and the operational situation which is simulated [3]. Transfer can be defined as the application of knowledge, skills, and attitudes acquired during training to the environment in which they are normally used [4]. With respect to virtual learning environments and serious games, fidelity and transfer are interconnected. More specifically, how much fidelity is actually needed to maximize transfer? This question is actually very important and may have a number of implications particularly when considering that “perfect” fidelity appears to be impossible to achieve, at least with our current technology and striving to reach full fidelity can also lead to increased development costs [6]. Given the expense and technical limitations associated with striving for high fidelity virtual environments, an important question is just how much fidelity is required to maximize transfer? However, the topic of fidelity remains a vague topic with some generic suggestions such as: simulators in later learning stages require a higher fidelity than in the initial stages of learning [3,6]. Furthermore, as described by Visschedijk [6], most of these generic suggestions have arisen from research primarily conducted with using (physical) flight simulators that focused on exactly replicating theactual (physical) equipment used in the real environment and on psycho-motor or procedural based skills which does differ from 3D serious games used for the training of cognitive skills like tactical decision making.

2. Overview The OPCAB serious game is being developed to easily alter various simulation parameters, including the fidelity of the auditory and visual cues, in order to test the

B. Cowan et al. / A Serious Game for OPCAB Surgery Procedure Training

149

effect of fidelity on knowledge transfer and retention. More specifically, it will allow us to examine i) do “better”, higher fidelity visual (graphical) and sound cues lead to greater transfer of knowledge? and how do changes in audio-visual simulation fidelity affect students’/trainees’ perceptions and their ability to perform specific tasks?, ii) what effect does fidelity have with respect to level of experience of the trainee or in other words?, is there a relationship between fidelity and trainee level, and iii) how do multi-cues (graphical and auditory cues in particular) interact and affect the transfer of knowledge? Testing of these factors will include user-based experiments using the OPCAB serious game. Figure 1 provides an example of how levels of visual fidelity could be altered. More specifically, the original scene is shown in Figure 1(a). Figures 1(b) and 1(c) illustrate lower quality examples that were obtained by processing the original scene with a “cartoon shader”.

Figure 1.Varying visual fidelity. (a) Original. (b) and (c) Lower quality versions.

3. Discussion Through the development of the OPCAB serious game within a “first-person-shooter gaming environment”, we anticipate that trainees will have a better understanding of the cognitive components of the procedure than by traditional learning modalities. In addition, by manipulating fidelity, we expect to contribute to the understanding of how specific simulation parameters affect knowledge transfer. Acknowledgments. The financial support of the Natural Sciences and Engineering Research Council of Canada (NSERC) in the form of a Post-Graduate Scholarship to Brent Cowan and individual Discovery Grants to Bill Kapralos and Adam Dubrowski is gratefully acknowledged.

References [1] J. I. Fann, A. D. Caffarelli, G. Georgette, S. K. Howard, D. M. Gaba, P. Youngblood, R. S. Mitchell, and T. A. Burdon. Improvement in coronary anastomosis with cardiac surgery simulation. Journal of Thoracic and Cardiovascular Surgery, 136 (2008), 1486–1491. [2] B. Dalgarno, and M. J. W. Lee. What are the learning affordances of 3D virtual environments? British Journal of Educational Technology, 41 (2010), 10–32. [3] R. T. Hays, and M. J. Singer. Simulation fidelity in training system design. New York: Springer, 1989. [4] K. D. Squire, K. Game-based learning: An emerging paradigm for learning. Performance Improvement Quarterly, 21 (2008), 7-36. [5] P. M. Muchinsky. Psychology Applied to Work. WA. USA. Hypergraphic Press, 1999. [6] G. C. Visschedijk. The issue of fidelity: What is needed in 3D military serious games? Master’s Thesis. University of Twente, Faculty of Behavioural Sciences, April 2010.

150

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-150

Progressive Simulation-Based Program for Training Cardiac Surgery-Related Skills Sayra CRISTANCHOa,1, Fuad MOUSSAb, Alex MONCLOUc, Camilo MONCAYOc, Claudia RUEDAc, and Adam DUBROWSKId,e a Department of Surgery and Centre for Education Research & Innovation, Faculty of Medicine, University of Western Ontario. London, Canada. b Division of Cardiac & Vascular Surgery, Schulich Heart Centre, Sunnybrook Health Sciences Centre. Toronto, Canada. c Faculty of Electronics Engineering, Pontificia Bolivariana University, Bucaramanga, Colombia. d The Hospital for Sick Children Learning Institute. Toronto, Ontario Canada. e Department of Pediatrics, Faculty of Medicine, and The Wilson Centre, University of Toronto. Toronto, Canada.

Abstract. Off Pump Coronary Artery Bypass (OPCAB) surgery is a strategy for revascularizing diseased coronary arteries without cardiopulmonary bypass. The complete operation can be deconstructed into individual tasks and subtasks that are ideal for creating simulation modules. Recently, we have developed a modular mechanical beating-heart OPCAB simulator for use in learner-centered training. In the present study, we describe the design of a progressive, simulationaugmented training program for OPCAB surgery. In particular, we a) define needdriven education and training goals, b) create simulation scenarios with progressive difficulty to specifically address these goals, and c) design corresponding assessment tools for both formative and summative purposes. Keywords. Simulation, cardiac surgery, curriculum

Introduction The apprenticeship model has been successful in training surgeons, particularly in the field of cardiac surgery. Such in-theatre training in the current era creates an ethical dilemma for the teacher dividing his/her responsibilities to deliver the highest quality of care to the patient and to train the future generation of surgeons. The advent of simulation has helped solve this dilemma. Indeed, it has been shown that simulation training prior to clinical exposure is beneficial for novice trainees [1]. In contrast to traditional opportunistic clinical teaching, simulation-based programs allow trainees to be progressively challenged in a systematic, learner-centered and patient-focused fashion. Although simulation has been suggested to be effective in other domains [2], it has not been systematically tested in cardiac surgical education. Off Pump Coronary Artery Bypass (OPCAB) surgery is a strategy for revascularizing diseased coronary arteries without cardiopulmonary bypass. The complete operation can be 1

Corresponding Author.

S. Cristancho et al. / Progressive Simulation-Based Program

151

deconstructed into individual tasks and subtasks that are ideal for creating simulation modules. The standard curriculum design activity is a cyclical process composed of 6 steps [3]: Conducting a general needs assessment, developing a rationale to target needs, designing goals and objectives, selecting the educational and evaluation strategies, implementing those strategies, and program evaluation/feedback. In developing simulation-augmented surgical training programs, designers face several issues related to identifying a guiding framework to meet the objectives of the programs. These issues may include: • The lack of an objective needs assessment: At present, the methodology used to design and implement simulation-augmented training programs has reflected the designer’s clinical experience. This may increase the risk of either de-contextualizing or over-contextualizing the skills to be trained [4]. • The lack of a systematic design methodology: A systematic design methodology must make use of the needs assessment to define appropriate training objectives and simulation resources. • The lack of structured assessments of performance: Such assessments should allow comparisons of trainee cohorts in order to revise and optimize the program. • The lack of a research-centered evaluation: Research-centered mechanisms should be built into training programs a priori to systematically evaluate, validate and augment their effectiveness with respect to educational value and relevance. The primary goal of this work is to implement a new methodology developed by our research team called ‘Aim–FineTune–FollowThrough’ to design a progressive, simulation-augmented training program for OPCAB surgery. More specifically, we aim to a) define needs-driven education and training goals, b) create simulation scenarios with progressive difficulty to address these goals, and c) design corresponding assessment tools for both formative and summative purposes.

1. Tools and Methods 1.1. The Curriculum The curriculum was based on the ‘Aim-FineTune-FollowThrough’ methodology [5]: At the ‘Aim’ phase, two stages are implemented: In the design definition stage, the procedure to be trained is selected. In the present context, a procedure is understood as a complex clinical activity that can be decomposed into simpler building blocks that include tasks, subtasks, and individual skills. In the design mapping stage, task analysis methods and diagrammatic tools are used to model the components of the procedure into workflow representations. At this stage, we used a new modeling tool (Motor and Cognitive Modeling Diagram – MCMD) [6], to map the performance of a group of experts. The MCMD will serve as a template for the simulation design. At the ‘FineTune’ phase, the verification stage uses MCMDs of other experts in order to validate (content) the workflow diagrams. Ten new cases were videotaped in the operating room followed by post-operative interviews with the experts. Finally, the ‘FollowThrough’ phase involves two additional stages. During the implementation stage, the simulation scenarios are designed and developed. The main

152

S. Cristancho et al. / Progressive Simulation-Based Program

simulator element used in the training program consists of a ‘beating-heart’ simulator, which physically reproduces the movement and rhythm of the heart, and allows trainees to perform replicated vascular anastomoses (Figure 1). The notion of progression is taken into account and therefore, the task difficulty of the simulations is adjusted for each training level according to the Challenge Point Framework [7]. At the validation stage, the effectiveness of the training delivered by the program is measured using standard evaluation models, namely the Kirkpatrick Program Evaluation Model [6]. We are currently conducting assessments at the first and second levels of the Kirkpatrick model for the OPCAB curriculum by running training courses with cardiac surgery residents.

Figure 1. Mechanical ‘Beating Heart’ Simulator for the OPCAB Training Program.

1.2. The Simulator Featuring a collaborative North-South partnership between medical and academic institutions from Canada and Colombia, the Faculty of Electronics Engineering from the Pontificia Bolivariana University in Bucaramanga, Colombia led the process of developing a physical beating-heart simulator for the OPCAB training program. The Beating Heart-Simulator consists of an artificial silicone heart model that beats and uses an electronic system to (1) control the beating of the heart, (2) simulate cardiac rhythm and hemodynamic disturbances, and (3) sense the amount of manipulation of the heart during a simulated scenario. The simulator is powered by an air compressor and an electronics system that automatically controls a set of solenoids. Given this arrangement, it is possible to create two independent movements; one to simulate the upper cardiac chambers (atria) and the other to simulate the lower cardiac chambers (ventricles). The result is recreation of realistic, energetic and dynamic cardiac movements. In terms of surgical skill development, this simulator allows training in distal coronary artery bypass anastomoses, both on the static and the beating heart. During

S. Cristancho et al. / Progressive Simulation-Based Program

153

positioning for distal anastomoses (enucleation), the system is capable of electronically sensing the degree of manipulation exerted on the heart and responds accordingly. This way, it is possible to specify two forms of performance-based assessment; (1) Computer-based monitoring that requires an expert to program specific scenarios (e.g: cardiac ischemia, ventricular fibrillation, asystole, tachycardia and sinus bradycardia). If the trainee performs an erratic or dangerous maneuver, the simulator responds by generating a cardiac disturbance that the trainee must overcome, (2) Expert-based monitoring where a surgical expert performs either an on-line or off-line (through video) assessment of the trainee’s performance. The physical component of the simulator consists of a fiberglass torso with anatomic proportions. The exterior of the heart is constructed with silicone and finished with water-based paints such as aniline. The internal mechanism responsible for mimicking cardiac movements is comprised of one rubber pump inside another providing decreased compliance. The pumps are filled with small plastic balloons that approximate the weight of the heart and allow for quick changes in the heart movement by limiting the amount of air required to fill the cavities. In addition, interchangeable arteries were simulated using 5mm Penrose Drains.

2. Results A simulation-based curriculum was designed for OPCAB surgery, which is composed of five simulation stations designed following the MCMD levels2 [6] and arranged in order of difficulty according to the Challenge Point Framework [7] (Table 1). At the skill level, the trainees perform the suturing step for a standard vascular anastomosis using a suture board equipped with 5mm silastic tubes (Penrose drains, Dacron; DuPont, Wilmington, Del) and 6-0 polypropylene suture. The first “subtask” involved performing a distal anastomosis on the static heart simulating the left internal mammary artery to the left anterior descending artery bypass. The second subtask consisted of performing the same simulated anastomosis on the beating-heart model. The final subtask consisted of performing a simulated anastomosis to a distal vessel on the enucleated, beating heart model, replicating a bypass graft to an obtuse marginal artery. Special consideration was given to changing vital signs during the enucleation stage and the required interaction with the anesthesiologist at the task level. At the procedure level, a complete distal anastomosis during a crisis scenario should be performed as before but now placed in an operating room context. The crisis management and the interaction between the surgical team and anesthesiologists allow for assessing teamwork and communication skills. For the assessment component of the curriculum, we have designed a set of checklists (one per simulation station) that comprise between 12 and 36 assessment items. We are currently performing a content validation study using a Delphi technique with a panel of 11 expert cardiac surgeons from various hospitals in Toronto and the US.

2 Skill level: One technical activity with no cognitive load; Subtask level: One technical activity with low cognitive load; Task level: More than two technical activities with medium cognitive load; Procedure level: More than two technical activities with high cognitive load

154

S. Cristancho et al. / Progressive Simulation-Based Program Table 1. Five-stage Training Program for the OPCAB Procedure

Training Level Skill level

Training Station 1. Suture of anastomosis

Subtask level

2. Distal anastomosis on still heart

Task level

3. Distal anastomosis on beating heart

4. Distal anastomosis on beating and enucleated heart

Procedure level

5. Distal anastomosis on beating and enucleated heart including crisis management

Objectives

Simulator

• Perform the suturing step for a standard vascular anastomosis

Penrose drains (5mm plastic tube model using prosthetic graft and either 6-0 or 7-0 polypropylene suture) Plastic model of a heart, plastic tube model for vascular anastomosis (5mm)

• Perform the complete anastomosis on the static heart between a graft and the left anterior descending artery (LAD) • Perform a complete anastomosis on the beating heart between the left internal mammary artery (LIMA) and the left anterior descending artery (LAD) • Perform enucleation • Perform complete anastomosis on the enucleated beating heart between the saphenous vein graft (SVG) and the Obtuse Marginal (OM) artery • Perform enucleation • Prepare for anastomosis on the enucleated beating heart between the saphenous vein graft (SVG) and the Obtuse Marginal (OM) artery • Demonstrate appropriate interaction with the anesthesia team

6-0 polypropylene suture Mechanical model for beating heart surgery, plastic tube model for vascular anastomosis (5mm)

Mechanical model for beating heart surgery, plastic tube model for vascular anastomosis (5mm)

In-situ simulation: The same equipment for station 5 will be transferred to the operating room and a crisis scenario will be implemented.

Assessment • Water-tight anastomosis • Symmetry • Efficiency of motion • Application of octoplus stabilizer. • Arteriotomy: Round and full thickness • Lie of anastomosis: parallel to aorta • Suturing • Exposure • Application of octoplus stabilizer • Application of silastic. • Arteriotomy & Proper shunt. • Suturing. • Patient positioning • Enucleation of heart. • Application of octoplus stabilizer • Application of silastic. • Arteriotomy & Proper shunt. • Suturing. • Enucleation of heart. • Application of octoplus stabilizer • Application of silastic. • Team work and communication.

3. Conclusions We have developed a simulation training program for the complex OPCAB procedure that includes the following general features: (1) modular-design approach to isolate task training and to provide distributed simulation; (2) immersion with the clinical

S. Cristancho et al. / Progressive Simulation-Based Program

155

environment; and (3) hybridization by combining technical and cognitive skills with interdisciplinary interactions. Our goal is to employ custom-designed and modular simulation scenarios so that the same learning resources could be easily adapted for developing curricula for other cardiovascular procedures. In an effort to expand the types of skills practiced with the proposed OPCAB curriculum future work will involve developing a virtual model that will consist of an interactive, multi-player prototype serious game. The goal will be for the user/trainee to successfully complete the OPCAB operation, focusing on the order in which steps are performed and on the tools required to perform each step, while minimizing the time to complete the procedure and maximizing the score. Once physical and virtual simulators are integrated, we will test the educational effectiveness of the curriculum by implementing randomized-controlled trials to evaluate various types of simulation-based learning conditions.

Acknowledgments The authors would like to acknowledge the contributions of the industrial design undergraduate student Jennifer Monclou, and professor Juan Carlos Villamizar from the BISEMIC research group at the Pontificia Bolivariana University, Bucaramanga, Colombia who assisted during the design and implementation of the beating heart simulator.

References [1] Brydges R, et. al. Developing criteria for proficiency-based training of surgical technical skills using simulation: changes in performances as a function of training year. J Am Coll Surg. 2008 Feb;206(2):205-11. [2] Reznick, R; MacRae, H. Teaching Surgical Skills - Changes in the Wind. N Engl J Med. 2006; 355:2664-9. [3] Kern DE, Thomas PA, Howard DM, Bass EB. Curriculum Development for Medical Education: A SixStep Approach. The Johns Hopkins University Press, 1998. [4] Kneebone RL. Practice, rehearsal, and performance: an approach for simulation-based surgical and procedure training. JAMA. 2009, 23;302(12):1336-8. [5] Cristancho SM, et. al. A Framework-based Approach to Designing Simulation-Augmented Surgical Education and Training Programs. Am J Surg. (submitted). [6] Cristancho SM, et al. Assessing cognitive & motor performance in minimally invasive surgery (MIS) for training & tool design. Stud Health Technol Inform. 2006;119:108-13. [7] Guadagnoli MA, Lee TD. Challenge point: a framework for conceptualizing the effects of various practice conditions in motor learning. J Mot Behav 2004;36(2):212-224

156

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-156

MSMS Software for VR Simulations of Neural Prostheses and Patient Training and Rehabilitation Rahman DAVOODI and Gerald E. LOEB Department of Biomedical Engineering, University of Southern California

Abstract. In the increasingly complex prosthetic limbs for upper extremity amputees, more mechanical degrees of freedom are combined with various neural commands to produce versatile human-like movements. Development, testing, and fitting of such neural prosthetic systems and training patients to control them effectively are complex processes that cannot be performed efficiently or safely by ad hoc and trial-and-error approaches. We have developed a software tool known as MSMS to enable researchers and engineers to simulate the movement of these neural prostheses and evaluate their performance before they are built and to train the patients in virtual simulation environments to operate their prostheses before receiving them. Further, MSMS facilitates development of interactive virtual reality applications for training, rehabilitation, and treatment of patients suffering from movement disorders. Keywords. Myoelectric Prostheses, Modeling and Simulation, Virtual Reality, Patient Training and Rehabilitation

Introduction Computer simulations have become an integral part of the design and development and safe operation of complex systems such as airplanes. In the development stage, different mechanical designs are prototyped virtually and simulated under various operating conditions to optimize performance. And once the airplane is built, its computer simulations are used in the safe environment of the flight simulator to train the pilots to fly the plane and practice emergency procedures that would be impractical with the real airplane. Complex multi-degrees of freedom (DOF) prostheses for upper limb amputees and their operation by neural or myoelectric commands of amputee patients are similarly complex and unintuitive and can benefit from computer simulations in all stages development. Computer aided design software such as SolidWorks (SolidWorks Corp., USA) is already used to virtually prototype the mechanical design of the prosthetic limbs. These tools help the engineers iterate different design ideas and improve the mechanical design before actual manufacturing. But what remain unknown are the actual performance of the prosthetic limb under various operating conditions and the ability of the patients to operate it successfully. These are important design questions and answers to them can and should influence the design of the prosthetic limbs. Currently, to find out how the prosthetic limb actually performs or whether the patient can learn to operate it successfully, the engineers have to wait until a physical prototype is built and

R. Davoodi and G.E. Loeb / MSMS Software for VR Simulations of Neural Prostheses

157

delivered to the patient, when it may be frustrating and even dangerous to identify mistakes and too costly to fix them. Therefore, virtual prototyping of the prosthetic limbs must include not only its mechanical design but also its performance under various operating conditions and evaluation of the ability of the patients to operate them successfully and effectively. In addition, these tools must enable the patients to use the safe virtual environments to practice and learn to operate their prostheses before actually receiving them. These tools are especially important for the new neural prosthesis systems where the combination of a multi-DOF prosthesis with multichannel neural commands forms a complex system with non-intuitive behavior that is likely to change as the patient gains experience. Development of neural prostheses to restore movement to the paralyzed limbs faces similar challenges. Patients must produce voluntary command signals to drive the electrical stimulation of a large number of paralyzed muscles, each with highly nonlinear behavior, to move the multi-DOF arm or leg to perform useful functions such as reaching, grasping, and walking. A virtual simulation environment can benefit the development of such complex systems and subsequent training of the patients to operate them effectively. A number of software tools have been developed over the years to enable virtual prototyping of neural prostheses but none of them support the full cycle of neural prostheses development and delivery. Mechanical design and simulation software such as ADAMS (MSC Software Corp., USA) and Working Model (Design Simulation Technologies, Inc., USA) have been used to simulate neural prostheses for amputees and paralyzed patients. These software packages relieve the user from the error prone and painstaking process of deriving and programming the equations of motion. However, they lack the specialized components specific to physiological systems such as muscles and do not support real-time interactive virtual reality simulations with the patient in the loop. More specialized software such as SIMM[3], OpenSim[2], AnyBody (AnyBody Technology, Denmark) provide tools for building accurate musculoskeletal models of the human limb but they have largely focused on biomechanical analyses such as those in gait labs and do not support development and real-time simulation of sophisticated feedback control systems and practical tasks and work environments. MSMS on the other hand, is designed to model prosthetic and human limbs and the task environment, and to simulate the limb’s behavior under different control inputs and external forces. Further, MSMS simulations can be performed in interactive virtual reality environments where the patients can test drive their neural prostheses and learn to operate them before they are built and delivered to them.

1. Development Methods and Software Architecture We have used an iterative software development process and applied professional software engineering tools and practices to gradually add, integrate, and test new features in MSMS. Each iteration period adds a small number of features and produces fully functioning and tested software that can and have been used in a number of applications over the course of its development. MSMS’s tools for construction, visualization, and rendering of models are programmed in Java and Java3D. The models are stored in XML files where a standard format is defined for storing the parameters of each model component such as a segment or a muscle. The standardized

158

R. Davoodi and G.E. Loeb / MSMS Software for VR Simulations of Neural Prostheses

XML format facilitates parsing, validation, and manipulation of model data using readily available tools. The computationally heavy simulations of MSMS models on the other hand, are performed in Simulink where a combination of Simulink library blocks and C programming language are used to speed up the execution of simulations. Simulink provides access to Matlab’s toolboxes that can be used to simulate advanced control algorithms. More importantly, the entire Simulink model can be automatically compiled into low-level machine language and executed in a real-time xPC Target PC (Fig. 1). These real-time simulations are essential for building virtual reality simulations with the patient in the loop[4].

Figure 1. Architecture of MSMS software. Models of prosthetic limbs and rehabilitation tasks are built using MSMS’s interactive modeling tools. The models are then automatically converted to a physics-based simulation model that can be run in Simulink or real-time xPC Targert simulation environments. The latter enables the creation of virtual simulation environments for patient training and rehabilitation.

2. MSMS Features and Applications MSMS is software that is still under development and through our iterative development process is gradually acquiring new capabilities and features. Development to date has already endowed MSMS with enough features and capabilities to support a variety of applications as summarized below. MSMS has a graphic user interface that allows the users to interactively create, navigate, and edit models of human and prosthetic limbs and the objects in the task environment (Fig. 2). MSMS Models can be edited graphically or directly at the XML files. The latter may be used by more advanced users. To facilitate fast assembly of virtual environments, MSMS allows the users to quickly combine existing models of the limbs and the task environments. Using this feature, a patient may be paired with different prosthetic limbs to find the right match and then allowed to practice operating it in different task environments. Although all model components could be built from scratch within MSMS, it has a utility that allows the users to import existing models

R. Davoodi and G.E. Loeb / MSMS Software for VR Simulations of Neural Prostheses

159

built in other popular software tools. These tools allow importing musculoskeletal limb models from SIMM and prosthetic limb models from SolidWorks.

Figure 2. MSMS’s Graphic User Interface. MSMS models can be built from scratch, imported from other software environments, and/or assembled from existing models of patients, prosthetic limbs, and rehabilitation tasks.

The physics-based movement of the MSMS models can be simulated in the popular Simulink simulation environment. These simulations can be used to study the movement of the neural prostheses in response to control inputs and external influences thus enabling the user to test the performance of the prosthetic control systems and optimize and fit them to the patients. Further, the Simulink simulations can be automatically compiled and downloaded to a real-time xPC Target PC where it can be executed in real-time (Fig. 1). Predictability and responsiveness of the real-time simulations are essential for construction of rehabilitation and training VR applications that must interact with the patients. These features have been used in Darpa’s Revolutionizing Prosthetics Program to model and simulate the myoelectric and cortical control of multi-DOF prosthetic limbs. In our laboratory, we have used these features to simulate normal control of human limbs by the central nervous system and the control of paralyzed human limbs by functional electrical stimulation[5, 6]. Creation of VR applications is facilitated by an extensive set of tools in MSMS such as custom cameras and lights, sound playback, objects of arbitrary shapes and texture, and support for 3D stereoscopic displays that can render and display the VR scene from user's perspective. These tools have been used to build realistic models of human and prosthetic limbs and the virtual task environments simulating realistic rehabilitation tasks (Fig. 3).

160

R. Davoodi and G.E. Loeb / MSMS Software for VR Simulations of Neural Prostheses

Figure 3. MSMS models of multi-DOF prostheses and the task environments simulating rehabilitation tasks and games.

In MSMS, we have built a set of unique animation tools that are specifically designed to facilitate neural prostheses development and patient training. The animation data can be loaded from a motion file and used to animate a MSMS model. Alternatively, MSMS can receive the motion data from live sources such as a physicsbased simulation of the model running in the same PC or a networked PC, or a realtime motion capture system attached to a patient. This allows the creation of interactive VR applications where the motion of the MSMS model is animated in real-time in response to the control actions and patient inputs. These features have been used to develop virtual cortical control experiments where a non-human primate subject produces cortical control signals to control the movement of objects in virtual environment (Fig. 4). Another animation feature allows non-expert users to develop animations of daily life activities using intuitive PowerPoint interface. Using this feature, primitive movements such as elbow flexion, hand opening, hand closing, etc., can be assembled into a more complex sequence of motions. The order of movement, the precise timing between the primitive motions, and the speed of animation for each motion could be easily modified by simply arranging the order of the slides and editing their properties in PowerPoint (Fig. 5). The resulting motion sequence can be played back in an openloop manner for training and demonstrations or in a closed-loop interactive environment where the timing and speed of animation can be modified in response to user’s actions. This feature is currently used in Walter Reeds Army Medical Center (WRAMC) to study the VR treatment of phantom limb pain in amputee patients.

R. Davoodi and G.E. Loeb / MSMS Software for VR Simulations of Neural Prostheses

161

Figure 4. MSMS model of a virtual environment used to study the cortical control of movement by nonhuman primates.

Figure 5. Expert and non-expert users alike can assemble primitive movement animations to build animation sequences with precise timing in Microsoft PowerPoint.

162

R. Davoodi and G.E. Loeb / MSMS Software for VR Simulations of Neural Prostheses

3. Discussion MSMS provides a comprehensive framework for modeling and simulation of neural prostheses and development of VR applications to rehabilitate and train the patients. To date, the emphasis has been on development of tools for modeling and simulation of prosthetic limbs that has allowed the users to prototype virtual neural prostheses for amputees and develop VR applications to train patients to operate them. MSMS is currently been expanded by adding the tools required for modeling more complex anatomical structures such as the hand and fingers. The inclusion of Virtual Muscle[1] , the most accurate muscle force prediction software, and validated models of proprioceptors[7-9] as an integral part of MSMS will enable us to build accurate simulations of the complete physiological systems. MSMS is available for download at http://mddf.usc.edu.

4. Acknowledgement The development of MSMS is currently funded by DARPA REPAIR program. MSMS development has also been funded in the past by DARPA Revolutionizing Prosthetics Program, NSF Engineering Research Center for Biomimetic MicroElectronic Systems, and Alfred Mann Institute for Biomedical Engineering.

References [1] [2]

[3] [4] [5] [6] [7] [8] [9]

E.J.Cheng, I.E.Brown, G.E.Loeb, Virtual muscle: a computational approach to understanding the effects of muscle properties on motor control. J.Neurosci.Methods 101 (2000) 117-130. S.L.Delp, F.C.Anderson, A.S.Arnold, P.Loan, A.Habib, C.T.John, E.Guendelman, D.G.Thelen, OpenSim: open-source software to create and analyze dynamic Simulations of movement. IEEE Transactions on Biomedical Engineering 54 (2007) 1940-1950. S.L.Delp, J.P.Loan, A computational framework for simulating and analyzing human and animal movement. Computing in Science & Engineering 2 (2000) 46-55. M.Hauschild, R.Davoodi, G.E.Loeb, A virtual reality environment for designing and fitting neural prosthetic limbs. IEEE Transactions on Neural Systems and Rehabilitation Engineering 15 (2007) 9-15. R.R.Kaliki, R.Davoodi, G.E.Loeb, Prediction of distal arm posture in 3-D space from shoulder movements for control of upper limb prostheses. Proceedings of the IEEE 96 (2008) 1217-1225. R.R.Kaliki, R.Davoodi, G.E.Loeb, Prediction of elbow trajectory from shoulder angles using neural networks. International Journal of Computational Intelligence and Applications 7 (2008) 333-349. M.P.Mileusnic, I.E.Brown, N.Lan, G.E.Loeb, Mathematical models of proprioceptors. I. Control and transduction in the muscle spindle. Journal of Neurophysiology 96 (2006) 1772-1788. M.P.Mileusnic, G.E.Loeb, Mathematical models of proprioceptors. II. Structure and function of the Golgi tendon organ. Journal of Neurophysiology 96 (2006) 1789-1802. M.P.Mileusnic, G.E.Loeb, Force estimation from ensembles of Golgi tendon organs. Journal of Neural Engineering 6 (2009) 036001.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-163

163

Virtual Reality System in Conjunction with Neurorobotics and Neuroprosthetics for Rehabilitation of Motor Disorders Alessandro DE MAUROa,1, Eduardo CARRASCOa, David OYARZUNa, Aitor ARDANZAa, Anselmo FRIZERA NETOb, Diego TORRICELLIb, José Luis PONSb, Angel GILc and Julian FLOREZa a eHealth and Biomedical Department, VICOMTech, San Sebastian, Spain b Bioengineering Group, CSIC, Madrid, Spain c Spinal Cord Injury Hospital of Toledo, Spain

Abstract. Cerebrovascular accidents (CVA) and spinal cord injuries (SCI) are the most common causes of paralysis and paresis with reported prevalence of 12,000 cases per million and 800 cases per million, respectively. Disabilities that follow CVA (hemiplegia) or SCI (paraplegia, tetraplegia) severely impair motor functions (e.g., standing, walking, reaching and grasping) and prevent the affected individuals from healthy-like, full and autonomous participation in daily activities. Our research2 focuses on the development of a new virtual reality (VR) system combined with wearable neurorobotics (NR), motor-neuroprosthetics (MNP) and brain neuro-machine interface (BNMI) to overcome the major limitations of current rehabilitation solutions. Keywords. Rehabilitation, virtual reality, neurorobotics, motor-neuroprosthetics, brain neuro-machine interfaces.

Introduction The improvement of physical rehabilitation therapies depends on achieving a more interrelated and transparent communication between the human system and the machines. Studies have shown that VR is a technology suitable for rehabilitation therapy due to its inherent ability of simulating real–life tasks while minimizing the hazardous aspects of such activities. Mirelman et al. have shown in [1] that lower extremities training of individuals with chronic hemiparesis using a robotic device coupled with VR improves walking ability in the laboratory and the community better than robot training alone. Most of the gait rehabilitation systems currently used for therapy are based both on treadmills and body weight support. Such systems are the Biodex Gait Trainer, the Robomedica system, InMotion [2] and the Lokomat (Hocoma). MOTEK Medical’s V-Gait [3] combines a treadmill capable of comprehensive ground reaction force measurements with a real-time motion capture system and a 3D virtual 1

Corresponding author: Alessandro De Mauro, VICOMTech, San Sebastian, Spain; E-mail: [email protected]. 2 This research is part of the HYPER project funded by CONSOLIDER-INGENIO 2010, Spanish Ministry for Science and Innovation.

164

A. De Mauro et al. / VR System in Conjunction with Neurorobotics and Neuroprosthetics

environment. We present a research part of the project HYPER (Hybrid MNP and NR devices for Functional Compensation and Rehabilitation of Motor Disorders) which aims at both the development and clinical validation of the first hybrid technology system for rehabilitation and functional compensation of motor disorders. The main challenges addressed are a significant improvement in results and faster rehabilitation of the functions using the assist-as-needed paradigm and new technologies.

1. Methods Both upper and lower halves of the patients are treated and the main emphasis is on allowing them to perform daily life activities. Users groups have been identified in order to adjust therapy and system components to different needs. Additionally, several scenarios have been detailed. Each of them includes a specific components configuration (MNP, NR, BNMI and VR). The therapy is subdivided in different states from the state 0 in which the injury happens until the state in which the patient is fully rehabilitated (Figure 1).

Figure 1. On the left: rehabilitation using the HYPER system: the therapy is specific for different states. On the right: scheme of the HYPER concept: NRs use volitional commands to drive paralyzed or paretic part of the body. MNPs bypass the damaged sensory-motor systems using FES.

Figure 1 shows also a conceptual scheme of the HYPER system. NRs use volitional commands from the BNMI devices for controlling a (mechatronic) wearable exoskeleton which applies controlled forces to drive paralyzed or paretic part of the body. MNPs bypass the damaged sensory-motor systems using Functional Electrical Stimulation (FES). This stimulation of motor and/or sensory nerves generates movements by activating paralyzed muscles. VR provides realistic, safe and patient specific environments together with repeatable exercises at different levels of difficulty. Ranges of movements important in the rehabilitation for CVA or SCI patients have been identified by medical doctors. The daily life functions can be always considered as combinations of them. In detail, upper body joints (and related movements) are: shoulder (flexion, extension, abduction, adduction, outward medial rotation, inward medial rotation); elbow (flexion, extension, pronation, supination); wrist (flexion, extension, abduction, adduction). Similarly lower body joints are: hip (flexion, extension, abduction, adduction, medial and lateral rotation); knee (flexion, extension); ankle (plantar flexion, dorsal flexion, inversion and eversion). For each of them, ranges of movements have been specified in order to assess patient skills. Those data will be

A. De Mauro et al. / VR System in Conjunction with Neurorobotics and Neuroprosthetics

165

used to define the rehabilitation exercises for each specific phase. A conceptual scheme of the system and an example of VR training session are shown in Figure 2.

Figure 2. Components of the HYPER system. On the left: MNP and NR systems. On the right: VR system configuration and snapshots of simple VR scenes: reaching, moving and grasping a virtual object.

Currently, patient movements are tracked by a motion tracking system based on radio frequency. Transmitters are positioned on each joint. This solution offers good tracking performances, but it suffers from the use of cables, which is not an optimal solution when considering the patients' needs. Hence, we are currently evaluating the use of one or several cameras for a marker-less tracking based on coloured gloves (similar to [4]). The matrices received from the tracker are used for a real-time representation on the screen. The trajectory data is stored in a database for elaboration and therapy assessments. OpenSceneGraph is used for the 3D rendering.

2. Conclusion We present an overall architecture and the development status of the first system that combines NR, MNP and VR for rehabilitation and functional compensation. This system will pave the way to enhanced rehabilitation therapies and to a new generation of wearable compensational devices for people suffering from motor disorders.

References [1] A. Mirelman and P. Bonato, Effects of Training With a Robot-Virtual Reality System Compared With a Robot Alone on the Gait of Individuals After Stroke, Stroke 40 (2009), 169-174. [2] B.T. Volpe, H.I. Krebs, N. Hogan, L. Edelstein, C. Diels and M. Aisen, A novel approach to stroke rehabilitation: Robot-aided sensorimotor stimulation, Neurology 54 (2000), 1938-1944. [3] S. Subramanian, L.A. Knaut, C. Beaudoin, B.J. McFadyen, A.G. Feldman, and M.F. Levin, Virtual reality environments for post-stroke arm rehabilitation, Journal of Neuroengineering and Rehabilitation 4 (2007), 20. [4] R.Y. Wang et al. Real-time hand-tracking with a color glove, ACM Trans. on Graphics (2009), 1–8.

166

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-166

Modeling the Thermal Effect of the Bipolar Electrocautery for Neurosurgery Simulation Sébastien DELORME 1, Anne CABRAL, Fábio AYRES, and Di JIANG Industrial Materials Institute, National Research Council Canada

Abstract. Real-time surgical simulation requires computationally-fast models describing the interaction between surgical instrument and tissues. In this study, a model for predicting the temperature distribution in brain tissue when using a bipolar electrocautery is proposed and validated against experimental in vitro animal data. Joule heat generation and heat conduction in the tissue are considered. The agreement between simulated temperature distributions and experimental data could be improved by modeling the output power as a function of electrical resistance between the electrodes, and by considering the heat exchange with surrounding air and bipolar tips. Keywords. Neurosurgery, simulation, electrocautery.

Introduction Surgical simulation requires modeling the interaction between surgical tools and biological tissues. In neurosurgery, the bipolar electrocautery (Figure 1) has become an essential tool for control of bleeding by coagulating tissues and blood vessels [1]. Its two metallic tips are electrodes connected to a high-frequency sine wave generator. When alternating current is applied and both tips are in contact with tissue, the area around the tips heats up. Temperatures up to 80C have been measured on liver tissue after 10 seconds of electric current application [2]. Coagulation is achieved in areas that have exceeded a critical temperature. After cauterization, small but visible blood vessels disappear, while vascularised tissues become less pink. Unintended heat transfer into the surrounding tissues can result in unwanted thermal damage to blood vessels or brain tissue. Simulators can help learning how to safely use the bipolar cautery near critical areas, a common scenario in neurosurgery.

Figure 1. Bipolar electrocautery handle. 1 Corresponding Author: Industrial Materials Institute, 75 de Mortagne Blvd, Boucherville, QC, Canada J4B 6Y4; E-mail: [email protected]

S. Delorme et al. / Modeling the Thermal Effect of the Bipolar Electrocautery

167

The objective of this study is to model the shape and extent of the cauterized area, as a function of controllable factors such as applied electrical power, duration of electrical power application, and distance between the electrode tips. A validation of the simulation results against in vitro experimental data is presented. Implementations of the model for real-time computation in a surgical simulator are discussed.

1. Methodology 1.1. Numerical Model Two phenomena involved in the cauterization with a bipolar were considered in the numerical simulations: Joule heating and heat conduction in the tissue. When an electrical field is created inside brain tissue, heat is generated according to ܳሶ ൌ

ฮ‫ܬ‬Ԧฮ ߪ

ଶ

(1)

where ܳሶ is the time rate of heat generated per unit of volume [Τ͵ ], ߪ is electrical conductivity, and ‫ܬ‬Ԧ is electrical current density [ Τʹ ]. Assuming perfect contact between the electrodes and the tissue, the current density at any point ܲ can be obtained by superposition of current density for each electrode ‫ܬ‬Ԧ ൌ

ሬሬሬԦ ሬሬሬԦ ‫ܫ‬ ‫ݎ‬ା ‫ܫ‬ ‫ିݎ‬ ൈ െ ൈ ଶ ଶ ԡ‫ݎ‬ሬሬሬԦԡ ԡ‫ݎ‬ሬሬሬԦԡ ʹߨԡ‫ݎ‬ሬሬሬԦԡ ʹߨԡ‫ݎ‬ሬሬሬԦԡ ା ା ି ି

(2)

where ‫ ܫ‬is the electrical current intensity, ሬሬሬԦ ‫ݎ‬ା is the vector from the positive electrode to point ܲ, and ሬሬሬԦ ‫ ିݎ‬is the vector from the negative electrode to point ܲ (Figure 2). The temperature distribution in space and time ܶሺ‫ݔ‬ǡ ‫ݕ‬ǡ ‫ݖ‬ǡ ‫ݐ‬ሻ can be computed by solving the heat conduction equation ߩܿ

߲ܶ ൌ ‫ ׏‬ȉ ሺ݇‫ܶ׏‬ሻ ൅ ܳሶ ߲‫ݐ‬

where ߩ is mass density, ܿ is specific heat capacity, and ݇ is thermal conductivity.

Figure 2. Computation domain dimensions, electrodes position, and vectors definition for Eq. (2).

(3)

168

S. Delorme et al. / Modeling the Thermal Effect of the Bipolar Electrocautery

Temperature was integrated over time using a finite difference explicit scheme, over a 3D domain of 5 by 10 by ሺͳͲ ൅ ‫ܮ‬ሻ mm, where ‫ ܮ‬is the distance between the electrodes centers (Figure 2), divided in a regular grid with 0.2 mm steps between nodes (approximately 100,000 nodes depending on ‫)ܮ‬. The second derivative was approximated by a second order central difference (MatLab del2 function), which uses cubic extrapolation of the temperature for the boundary nodes of the domain. Electrocautery generators allow setting a nominal power and will control the duration of alternate current bursts at constant voltage to obtain the desired average output power at a nominal resistance value. If the resistance between the electrodes is different than the nominal resistance, the output power will be less than the nominal power [4]. Such behavior is expected: one could model a real voltage source as an ideal constant voltage source with an internal resistance to account for power losses in the voltage generating equipment. In that case, the maximum output power is given when the load resistance matches the internal resistance. The electrical current intensity was calculated from the nominal power ܲ using

‫ܫ‬ൌඨ

ܲ ܴ

(4)

where ܴ is an estimate of the equivalent electrical resistance of the tissue obtained by ܴൌ

ܸ ͳ ௅ ฮ‫ܬ‬Ԧฮ ͳ ͳ ͳ ൌ න ݀‫ ݔ‬ൌ ൬ െ ൰ ‫ܫ‬ ‫ ܫ‬௥೐ ߪ ʹߨߪ ‫ݎ‬௘ ‫ ܮ‬െ ‫ݎ‬௘

(5)

where ‫ݎ‬௘ is the radius of the electrode tips. The electrodes tips were centered on the tissue-air boundary of the computation domain. The electrode tips were assumed to be hemispherical and penetrating the tissue by a distance ‫ݎ‬௘ . Therefore a zero electrical current density was imposed at all nodes within a distance ‫ݎ‬௘ from the electrode center. Heat transfer outside of the tissues (e.g. electrodes, air) was not considered. To ensure convergence of the finite difference time-domain integration, time steps were defined using the following Courant-Friedrichs-Lewy condition in 3D [3]: ο‫ ݐ‬൑

ߩܿ ଶ ο‫ݔ‬ ͸݇

(6)

The following physical constants were used: ߪ ൌ ͲǤͳͻπǦͳ ȉǦͳ in vitro at ܶൌ͵͹Ԩ, with 1.75% increase per Ԩ ; ߩ ൌ ͳǤͲͶൈͳͲ͵ Τ͵ ; ܿ ൌ ͵Ǥ͸ൈͳͲ͵ Τȉ ; and ݇ ൌ ͲǤͷ Τȉ. In a simulation of an in vivo situation, the physiological tissue temperature (ܶ଴ ൌ ͵͹Ԩ) would be applied as the initial condition over the entire domain and as the boundary condition on all boundaries of the domain except the brain-air interface. For in vitro, a different temperature could be used to match the experimental conditions. To simulate vaporization of the water content of tissues, temperatures were limited to 100C, and at nodes that have reached such temperature Joule heating was fully cancelled by latent heat absorption.

S. Delorme et al. / Modeling the Thermal Effect of the Bipolar Electrocautery

169

1.2. In Vitro Experiments In vitro experiments were done to measure the thermal effect of the bipolar electrocautery on animal brain. An infrared camera (SC620, FLIR Systems Inc), a digital video camera (HDR-XR200 AVCHD Handycam, Sony) and a bipolar electrocautery handpiece (22.5 cm Stainless Steel Bipolar Bayonet Forceps, Medicon), connected to an electrosurgical generator (GN060 Bipolar Coagulator, Aesculap), were mounted over a specimen platform (Figure 3). The specimen platform had macro and micro height adjustment. The bipolar handpiece was tilted 30 degrees from vertical to allow thermal imaging vertically. Three fresh brains from 3-month old calves obtained from a slaughterhouse were tested within 4 hours post-mortem. Upon arrival at the laboratory, the brains were maintained at body temperature in a heated bath of phosphate buffered saline solution. A brain was then placed in a plastic dish on the specimen platform. The platform was raised until the two tips of the bipolar were in contact with the brain surface. The distance between the bipolar tips was set to a desired value using an adjustable spacer. The power on the generator was set to a desired nominal value. While recording with the two cameras, the power was activated using a foot pedal for 10 seconds. After the recording was stopped, the specimen platform was lowered, and the dish was moved in order to perform another test at a different location at least 20 mm away from any other testing area. When no more areas were available for testing, another brain was taken out of the heated bath for testing. Throughout the experiment, the brain was kept wet and the bipolar was kept clean of adhered tissue. Testing conditions selected to highlight the effect of nominal power, distance between bipolar tips, and duration, are described in Table 1.

infrared camera video camera

power generator

brain sample

specimen platform Figure 3. Experimental setup

bipolar handpiece

170

S. Delorme et al. / Modeling the Thermal Effect of the Bipolar Electrocautery

Table 1. Testing conditions Parameter

Symbol

Test 1

Test 2

Test 3

Test 4

Electrode tip radius

re

0.5 mm

0.5 mm

0.5 mm

0.5 mm

Duration of power application

T

3 sec

3 sec

10 sec

3 sec

Nominal power on generator

P

13 W

5W

5W

13 W

Distance between electrode tips

L

3 mm

3 mm

3 mm

6 mm

2. Results Simulations were run under the same conditions as the selected in vitro experiments. The initial tissue temperature was set in the simulations to the average measured tissue temperature before applying electrical power. Simulated and experimental temperature distributions on the tissue-air boundary plane are showed in Figure 4.

3. Discussion 3.1. Experimental Validation of the Simulation Simulation with nominal power resulted in hotter temperature profiles than experimentally measured. Bipolar generators are designed to output variable power depending on electrical resistance between the electrodes. The output power matches the set power when resistance is approximately between 50 and 100 π , for the generator used in this experiment [4]. In our experiment, the electrical resistance was not measured. In the simulations we used brain resistivity data from the literature, which varies between sources as well as between in vivo and in vitro conditions [5][6]. Resistance is also affected by the contact area between the electrode and the tissue. In our simulations, we assumed perfect contact over the surface of the electrode tip, whereas in the experiments the electrodes were at an angle and contact force was difficult to control due to viscoelasticity of brain tissue. Although tissues were kept moist by periodical irrigation with PBS, tissue moisture, which also has an impact on electrical resistance, might have varied during and between experiments. Furthermore, the presence of pia mater over the grey matter might also have altered the effective electrical resistance between electrodes. Therefore, the power was adjusted until the simulated temperature at the midpoint between the electrodes matched that of the experiment, in order to simulate a situation where the electrical resistance between the electrodes would have been measured. The simulated temperature distribution profile with the adjusted power better matched the experimental results. However, the tissue was too hot near the electrodes and fell too fast away from the probe.

S. Delorme et al. / Modeling the Thermal Effect of the Bipolar Electrocautery

171

Figure 4. Simulated temperature distribution on the tissue-air plane at nominal power (column 1), at adjusted power (column 2), and experimental results (column 3), for the following testing conditions: Pnominal = 13 W, L = 3 mm, and t = 3 sec (line 1); Pnominal = 5 W, L = 3 mm, and t = 3 sec (line 2); Pnominal = 5 W, L = 3 mm, and t = 10 sec (line 3); Pnominal = 13 W, L = 6 mm, and t = 3 sec (line 4). All results are displayed with the same color map (top right).

172

S. Delorme et al. / Modeling the Thermal Effect of the Bipolar Electrocautery

Experiments with measurement of output power or of electrical resistance between the electrodes must be done to further validate these results and identify potential model improvements. In the simulations, the limit temperature of 100Ԩ was reached in nodes close to the electrodes. However, the experimental results rarely showed maximum temperatures of 100Ԩ. Apart from measurement errors, this could be due to other potential thermal phenomena that were not considered in the simulation, such as heat conduction from the tissue to the electrodes, as well as heat conduction and convection from the tissue to the air. Simulation of some of these phenomenon increases considerably the computational burden because it requires drastically smaller time steps to satisfy the CFL condition of Eq. (6). Nevertheless, each of these phenomena is expected to generate tissue temperatures cooler than in our simulations. Although the brain specimen were not perfused during the in vitro experiments, heat dissipation due to blood perfusion is expected to further cool tissues in vivo, and might have an important enough effect to require modeling for simulating in vivo interaction of the bipolar electrocautery with brain tissue. The effective electrode contact radius also has a significant impact on the current density which decreases approximately with the square of the distance from the electrode center. A locally denser mesh around the electrodes might have helped provide more accurate temperature distribution in this area, but local refinement is not possible with regular grid methods such as the finite difference method. 3.2. Implementation in Surgical Simulator In a surgical simulator, the bipolar electrocautery can be moved around freely over the tissue surface during power application. The Joule heating phenomenon can be implemented for real time computation using a 3D core of a linear or non-linear heat injection profile over a voxelised domain, and superposition over time to account for the accumulation of thermal energy in the brain tissue. However, heat conduction continues after the bipolar has moved away from the heated area, which requires computing heat conduction over the whole simulated domain. Using the explicit scheme resolution of the heat conduction problem, the computational time was 15 times greater than the simulation time, over a domain limited to 5 mm distance around the bipolar tips. Future work will focus on proposing solutions for implementing real time computation of heat conduction.

References [1] [2] [3] [4] [5] [6]

L.I. Malis, Electrosurgery, J Neurosurg 85 (1996), 970-975. E.W. Elliott-Lewis, A.M. Mason, D.L. Barrow, Evaluation of a new bipolar coagulation forceps in a thermal damage assessment, Neurosurg 65 (2009), 1182-1187. W.H. Press, S.A. Teukolsky, W.T. Vetterling, B.P. Flannery, Numerical Recipes: The Art of Scientific Computing, Cambridge University Press, New York, 2007. High-frequency surgery unit GN 060, Instructions for use/Technical description. http://www.aesculapusa.com/default.aspx?pageid=498, retrieved Nov 1, 2010. K.R. Foster, H.P. Schwan, Dielectric properties of tissues and biological materials: a critical review, Crit Rev Biomed Eng 17 (1989), 25–103. J. Latikka, T. Kuurne, H. Eskola, Conductivity of living intracranial tissues, Phys Med Biol 46 (2001), 1611–1616.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-173

173

CliniSpace™: A Multiperson 3D Online Immersive Training Environment Accessible through a Browser Parvati DEV, PhD, W. LeRoy HEINRICHS, MD PhD, Patricia YOUNGBLOOD, PhD Innovation in Learning Inc, Los Altos Hills, CA

Abstract. Immersive online medical environments, with dynamic virtual patients, have been shown to be effective for scenario-based learning (1). However, ease of use and ease of access have been barriers to their use. We used feedback from prior evaluation of these projects to design and develop CliniSpace. To improve usability, we retained the richness of prior virtual environments but modified the user interface. To improve access, we used a Software-as-a-Service (SaaS) approach to present a richly immersive 3D environment within a web browser. Keywords. Medical education, simulation, virtual patient, virtual world, roleplaying.

1. Introduction Immersive online medical environments, with dynamic virtual patients, have been shown to be effective for scenario-based learning [1]. However their use has not yet become routine. Using the metaphor of “crossing the chasm” [2], most of these applications are used by a few innovators and have not yet crossed the chasm into mainstream use. Studies that have evaluated these systems may provide insight into the opportunities and barriers for virtual environments to achieve mainstream use.

2. Method User feedback from prior studies was reviewed to pinpoint issues that might prevent uptake of virtual medical environments for training. These included responses to questionnaires, answers in focus groups, and informal comments on prior systems. The feedback was grouped into categories based on our existing 7-point framework (described below) for analysis of new technologies for medical learning [3]. 1. Robustness of System 2. Review by Content Experts 3. Usability Testing 4. Validity Testing 5. Assessing Learning Outcomes 6. Integration into the Curriculum 7. Transfer of Learning to Clinical Practice

174

P. Dev et al. / A Multiperson 3D Online Immersive Training Environment

Item 1 addresses the reliability of the software and hardware system. Item 2 uses review by subject matter experts to ensure the accuracy and appropriateness of the material for the target audience. Item 3 examines the ease of the user interface, including navigation and learning to use the system. Item 4 compares the performance of experts with that of less experienced professionals and novices to establish construct validity. Item 5 focuses on ‘what’ and ‘how much’ individuals have learned from using the educational application. Item 6 assesses the degree to which the new technology will be easily integrated into the existing curriculum or training program. Item 7 addresses the issue of transfer of skills and knowledge from the training environment to the work environment.. The summarized feedback was used to guide the design of the next system, within the constraints of the available technology (and the available budget). Because the available technology had improved over the time course of these studies, it was possible to implement features that could not have been implemented earlier. Further evaluation studies are planned to determine changes in user feedback in response to changes in the system.

3. Results A summary of the feedback is presented in Table 1. Most of the feedback was in one of the first four categories of our framework. This Feedback guided our design of CliniSpace Table 1: Summary of feedback from prior studies.

Pro 1.

Robustness of System Client installation is easy Once installed, crashes and failures are rare. Operating the patient avatar needs some learning.

2.

Con

Server installation and setup is a technical task. Server is not installed at the learning institution. Institutional firewalls are a major barrier. Institutions lack computers with adequate graphics capabilities.

Review by Content Experts Environment was believable

Not enough medically important objects available

Patient pathophysiology dynamics was realistic

Pathophysiology models have not been validated rigorously

Vast teaching potential

Patient avatar is not responsive. Missing typical medical clues.

Safe environment to learn through mistakes

No curriculum has been developed.

Scenarios were believable. They correspond to situations that may really happen.

Scenario development should be improved. Roles and responsibilities need to be clear. If not, then there is confusion over logistics, such as who and when to transport patients.

P. Dev et al. / A Multiperson 3D Online Immersive Training Environment

175

Consider collecting scenarios and cataloging / sharing. Voice capability makes this excellent for teaching communication 3.

Usability Testing Felt immersed

Difficult to navigate. Turning the avatar is difficult. Sometimes I get stuck inside a wall. Curtain gets in the way. Mouse is not precise.

Watching the scenario from different angles, or a zoom-out view, gives a bigger picture than just one camera view.

Too many viewing (camera) options

Easier to play a role in VME than in face to face

Long learning curve means this cannot be used for infrequent sessions. Difficult to know who is who without names on the avatars. The avatars need to look different from each other. Difficult to tell who is talking when there are multiple teams

4.

Learning Outcome Performance improvement is comparable when using the physical manikin and the virtual environment for ATLS scenarios (objective measurement, not verbal feedback) Gives a better sense of the need for management in crisis scenarios Increases confidence in managing mass casualty scenarios

3.1. Robustness of System. While existing systems were robust, there were substantial difficulties in deployment. The problems were two-fold: a) the available computers had inadequate graphics capabilities to display the 3D virtual environments at the necessary quality and interactivity, and b) software download and installation was not feasible for computers that were behind a firewall. To solve the first problem, we examined a new generation of virtual world platform software driven by the need for mass access to videogames. Computers for hard core videogames require a separate high performance graphics card. Most laptops, on the other hand, have only a graphics chip integrated into the motherboard. Virtual world platform software, such as that from Unity Technologies (San Francisco) is optimized for high quality 3D on commodity computers and even mobile phones. The second problem does not yet have a solution for 3D. An application that runs in a browser, using the http protocol, is able to reach through the firewall. However, any application that renders a 3D environment requires installation of a plug-in. This

176

P. Dev et al. / A Multiperson 3D Online Immersive Training Environment

installation may be easy and automatic, as it is for Unity3D, but the download may be blocked by the firewall. With the advent of HTML5, the need to download a 3D plugin may disappear. We encountered a third problem which had not surfaced in our studies. The ability to support in-world voice interaction between the learners requires the opening of additional ports through the firewall for the voice signal. 3.2. Content Review by Experts. Content experts liked the teaching potential of virtual environments but felt that the environment lacked medical richness, and that the curricula had not been developed. They also worried that the pathophysiology models were not rigorously validated. We received permission and conducted a photographic survey of key spaces in a local hospital. Based on this survey, we modeled a number of hospital spaces (emergency department, intensive care unit, urgent care unit, a medical ward, and an examination room, along with a reception and waiting area, a conference and briefing room as well as hallways with signs linking spaces) with the equipment likely to be used in these spaces (figure 1). Project limitations prevented making all of the objects interactive but, potentially, this is possible.

a)

b) Figure 1. a) The reception area, b) the emergency room in CliniSpace.

Similarly we analyzed the expected behaviors of the virtual patient and selected those that we felt would give sufficient realism to the patient. These behaviors are implemented as animations triggered by learner actions or by changes in patient pathophysiology (figure 2). The pathophysiology model for trauma is based on extensive literature search. It is a phenomenological model, linking vital signs to key variables such as the rate of bleeding. Over much of its range, the model uses quantitative data that is validated in the literature. At its extreme points, such as near death, valid data is not available and must be provided from personal experience of physicians. Lack of a well-defined curriculum remains a significant gap. We are collaborating with professional societies and non-profit organizations to embed the use of CliniSpace-based role-playing scenarios in their curricula.

P. Dev et al. / A Multiperson 3D Online Immersive Training Environment

177

Figure 2. Representation of virtual patient behavior through animations. Left, vomiting. Right, scratching a rash on the neck.

3.3. Usability Testing. This was where we had the most detailed feedback. Problems included navigation, camera control, too many options, and a difficulty in recognizing individuals. Navigation is a property of the underlying platform, as well as of the implementation of the environment. Small changes implemented in CliniSpace navigation include being obstructed by solid objects such as walls, but being allowed to slide along them till a door is reached. This prevents getting stuck against a wall near a door. Duplication of function, such as that of the mouse and the arrow keys, is replaced by allowing the arrow keys to control translation (move forward, backward and turn) while the mouse controls selection of functions. This shortens the learning curve for navigation while retaining the user’s prior framework of using the mouse for selections. (For advanced users, gamer-style navigation controls are available, such as the use of the W, A, S, D keys.) Two camera views are provided, the view that a learner or role-player has of the patient and of all the other people in the room, and a view from above the patient which allows a better view of the full patient (figure 3).

Figure 3. The two camera views of the patient: from eye level, and from the ceiling.

Avatars have a morphology representing real people rather than gamer characters. Also variation of age, gender and ethnicity, as well as allowing each learner to select a unique name, allows easy recognition of individuals.

178

P. Dev et al. / A Multiperson 3D Online Immersive Training Environment

The simplified navigation and controls are countered by an increase in the number of interactive objects. It is yet to be determined if learners are able to navigate and use the large range of interactive objects. To simplify the learning curve about the range of available interactive objects, a menu button allows turning on or off the names of all the interactive objects (figure 4).

Figure 4. A view of the emergency room with all the interactive objects identified.

3.4. Learning Outcome. Feedback about virtual environments as a learning tool was almost uniformly positive. Suggestions that there may be difficulties were found to be issues of usability. The CliniSpace environment will be deployed in selected simulation centers. At the time of the conference, we expect to present early results of its usage and usability.

4. Discussion A new virtual medical environment, CliniSpace, was designed in response to extensive feedback from studies of earlier systems. It represents improvements in usability as well as in detail of representation of a medical environment. It also presents interactive virtual patients with dynamic pathophysiology that respond to a typical range of medications and procedures that would typically be performed in a medical environment. Possible uses of such virtual environments for learning include: • All critical events and codes

P. Dev et al. / A Multiperson 3D Online Immersive Training Environment

• • • • • • •

179

Hand-offs during shift change within a department and hand-offs between departments (ED ICU Surgery Ward Family) Interdisciplinary communication (with pharmacy, therapy, other). Multidisciplinary communication as in rounds. Communication within a hierarchy (nurse-physician communication). Bringing a difficult situation to notice. Communication with family Safety training Visualization and familiarization (staff reviewing a new unit or a new hospital; patients preparing for their next visit; vendors studying space specifications) On-boarding of new staff and trainees

Early evaluation of CliniSpace shows that most of the design goals were achieved – robustness, ease of installation, short learning curve, realism leading to suspension of disbelief. The system has proven attractive enough that numerous groups have requested, and contracted for, customization for their unique educational purposes. However, every hill climbed (every success) brings the next higher mountain into view! It is clear that our major desired application, out-of-the-box use of CliniSpace, will require the implementation of curricula that embed the use of this simulation capability.

References 1.

2. 3.

P. Youngblood, P.M. Harter, S. Srivastava, S. Moffett, W.L. Heinrichs, P. Dev, Design, development, and evaluation of an online virtual emergency department for training trauma teams, Simul Healthcare (2008), 3. G.A. Moore, Crossing the Chasm, Harper Paperbacks, 2nd ed, 2002. P. Youngblood, P. Dev, A framework for evaluating new learning technologies in medicine, AMIA Annu Symp Proc. (2005), 1163.

180

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-180

Medical Education through Virtual Worlds: The HLTHSIM Project Roy EAGLESON, Sandrine de RIBAUPIERRE ECE and CNS Depts. University of Western Ontario Sharla KING and Eleni STROULIA Fac. Education and CS Dept., University of Alberta

Abstract. Training tools using virtual reality (VR) are becoming more popular and cost-effective to develop and are increasingly adopted; yet there is no systematic means for evaluating their usability and pedagogical effectiveness. There are a wide range of training scenarios that can be scripted, from high level simulations of emergency response systems where participants using their avatars have to make complex decisions and communicate with each other, to low-level sensormotor skills-based trainers where surgeons can practice suturing and cutting. We propose a classification framework for simulator-based training, associating each type of simulation with a specification of the types of skills it is designed to exercise and a corresponding evaluation plan. In this framework, objective measures involving task time and error rates can be formalized at the lower levels, and related subjective and objective measures can be identified at the top. Our framework is being implemented under the auspices of a recently funded New Media project in Canada (GRAND NCE) that spans two health training and simulation facilities (CSTAR and HSERC).

Introduction The proliferation of Virtual-Reality (VR) platforms, ranging from high-fidelity, requiring specialized hardware and software to commodity virtual worlds, present a great opportunity for simulation-based education and training. We are developing a conceptual framework for using VR for general healthcare training and evaluation, with specific emphasis on emergency response and surgical scenarios. This paper reflects the perspective of a Healthcare Simulation project (HLTHSIM) within Canada’s National Centre for Excellence on Graphics and New Media (NCEGRAND). In this project, we will develop methodologies and tools for the design, development, and evaluation of augmented reality (AR) based simulations. To that end, we will develop platforms integrating sensing devices to perceive the real world, VR tools to simulate complex environments, processes and phenomena based on data collected through these sensing devices, and actuators through which to change the state of the real and virtual worlds for training future and practicing health professionals. From a methodological perspective, we are interested in exploring the pedagogical value of “serious games.” We have chosen to address two kinds of training scenarios with these platforms: surgical training and inter-professional health team problem solving within an emergency response context. Surgeons tend to perform better after playing video games, according to reports in the media. Far from being merely anecdotal reports,

R. Eagleson et al. / Medical Education Through Virtual Worlds: The HLTHSIM Project

181

recent studies have continued to add substance to these reports. Indeed, the interesting question is not 'whether' interactive multimedia exercises will improve surgical performance, but how can multi-modal training exercises best be designed for a range of surgical procedures? In addition, health care delivery is becoming increasingly teambased. Services require collaboration between front-line health teams and researchers in both face-to-face and online environments.

1. Framework: A Hierarchy of Training Scenarios In descriptive terms, when considering the timeline of any medical situation, one can recognize the following three phases: situational assessment, decision making and activity (composed of different tasks). For example, in an emergency medical response situation, first responders arriving on a scene, begin with a situational assessment. This is followed by some cognitive, problem-solving processes, towards a decision making to stabilize the situation. Finally, based on their decisions, the responders will take action performing a variety of tasks or direct actions. The characteristic of all tasks is that they are comprised by low-level visual-motor actions. Examining the surgical task in detail, we recognize a Kinematic level (kinematics of reaching, grasping, cutting, piercing) and a Dynamic level (control dynamics of pulling, cutting, piercing, probing). This descriptive account can be collected into a Surgical Skills abstraction hierarchy: 1. Cognitive or 'Knowledge Level' (assessment) 2. Decision-theoretic (reasoning, planning) 3. 'Action Scripts' or tasks (rehearsed responses to typical scenarios) 4. Kinematic Skill (positioning, grasping, moving) 5. Dynamical Skills (forces, balance, pressure) Using these levels, we are proposing to break down the concept of a 'training scenario' into layers of abstraction. This hierarchy is similar to those posed by Albus (1996) to describe the abstract levels of control that need to be considered when controlling a complex mission using telerobotic control of a remote robot. At the lowest levels, the training aspect is at the skill level, where movements and actions involving hand-eye coordination are converted to visual-motor programs. The skills are evaluated by measuring the speed/accuracy of the movements (Fitts' paradigm). As we move upwards through the hierarchy of levels, training involves skills that are composites of skills at the lower level. So that tasks, like suturing, are skills that are comprised by low-level movements: grasping and pulling, and are repeated and controlled with a set of rules which initialize, iterate, and complete the movement. In composite actions, the notion of 'task time' and 'accuracy' can be specified in terms of the individual task times of the composite movements, or as the overall sum. Similarly, for accuracy, the objective score for, say, a row of sutures, can be based on an assessment of the individual positions of the run of stitches, or by an overall measure of the uniformity of the pattern. Pursuing this argument further, as we move upwards through the hierarchy, the skills being trained are posed in a vocabulary that may include tasks like opening or closing (which in themselves comprise sub-tasks). In opening, the surgeon will cut the skin, achieve hemostasis, and reach his target; while for closing: the surgeon will finalize the hemostasis, close the incisions with sutures and apply a dressing. At this level, our perspective here is that objective measures of the skill level of the trainee will be posed in terms of some speed and accuracy measures -- yet these become

182

R. Eagleson et al. / Medical Education Through Virtual Worlds: The HLTHSIM Project

overall scores typically. In surgical assessment, these are often reported as 'outcome measures', and can be a mix of subjective and objective scores. However; if training is performed using VR, these measures can be determined if the training software is designed to measure the movements and response times of the trainee. If the evaluation can only be posed in terms of 'speed and accuracy' at any one of the levels of abstraction, the scores for evaluating the effectiveness of training at any particular level can be comprised by more objective measures which are formalized at the level that is immediately lower.

2. Layers of Knowledge and Skill in Medical Education Consider the distinct levels of knowledge and skills spanned by VR simulations. Domain Knowledge: At this level, we include the knowledge and skills relevant for diagnosis, namely the logical reasoning necessary for evaluation of the situation and the patient. One has assess the patient, bringing to bear all their relevant medical knowledge, applying it specifically to that patient, while integrating the patient’s history and the various labs and imaging results. Planning and Decision Making: Here, we include the processes that take place at the scene of an accident where emergency medicine technicians have to plan the on-site care, the transfer to a hospital and hand-off to trauma specialists. The decision making involves grouping, coordinating and communicating a number of tasks among several individuals and diverse disciplines to achieve a result. Evaluation at this level may include considerations of time and agreement of the enacted plan with those of experts. Task-level activities occur within a small time involving a motor action and may also require a basic decision. At this level, we include activities such as dressing a wound at the scene of an accident for example. Tasks may be simple with no decisions involved (such as tying a knot) or composed of sub-tasks (for closure, the surgeon or trainee needs to decide how far apart he wants the stitches, what type of suture he wants (single, continuous running etc). Task can be difficult to assess, since there may be no explicitly “true or wrong” task executions, so timing can still be looked at (especially in simple tasks), but accuracy is more important than error rate (not binary but scaled). At the movement level (kinematics), we examine aspects such as the displacement in space with surgical instruments to reach a target. For example in endoscopic surgery, the surgeon has to reach the target with the forceps while looking at the screen. At this level, assessment involves considering both speed and accuracy. Finally, at the dynamics level (forces, etc) we examine physiological control, such as the force with which a surgical instrument is grasped; the amount of pressure exerted on scissors/needle driver/forceps in order to use them. Training may be facilitated with repetition, and experience as well as decreased stress level during a procedure might also affect it, and explain the differences between novice and expert. 2.1. The “Surgical Training” Task: In order to improve surgical training, a variety of simulators are available. These include ‘box trainers’ and VR or AR simulators. Box trainers provide realistic ‘haptic feedback’ (the contact sensation of actual tissue or phantom materials) but do not have instrumentation for recording of trajectories or precise timing that would allow for an objective assessment of skill acquisition. Shanu (2002) and Stefanidis (2010) showed

R. Eagleson et al. / Medical Education Through Virtual Worlds: The HLTHSIM Project

183

surgical residents trained to perform laparoscopic surgery using these trainers were more proficient and made fewer errors in the operating room than those who received no virtual reality simulated education. We plan to develop a collection of training case studies using AR and multi-modal visual displays of 3D Biomedical imagery. The training cases will range from generic visual-spatial skills development, to the rehearsal of very specific surgical procedures and scenarios. We are exploring specific cases: • Laparoscopic Cholecystectomy (gall bladder removal) • Ultrasound-Guided Prostate Biopsy • Robotically-Assisted Mitral Valve Repair • 3D Neuroendoscopy The evaluation of these immersive multimodal displays requires methodologies from Experimental Psychology and Cognitive Science. • 2D Pattern Recognition (Pattern Detection Task) • 3D Shape Matching (Shepard and Metzler task) • Targeting time and accuracy (Fitts’ Methodology) • Trajectory Analysis (Accot and Zhai) Virtual reality is also used in one of our project to evaluate the ability of the trainee to see in stereo (simulation of the floor of the third ventricle as seen in an endoscopic third ventriculostomy procedure), and correlate it to his ability to see with a 3Dstereoendoscope, and localize a target to evaluate the benefit of the 3D vs. 2D. 2.2. “Collaborative Problem Solving” Task types: In this task, we will develop AR serious games across the real and virtual worlds to support three key mechanisms for learning as represented by the principles of Collin’s Cognitive Apprenticeship Model (Collins, 1991, Järvelä, 1995) – modeling, scaffolding and reflection. Modeling occurs when the student observes the expert or practitioner demonstrating a specific skill or task. Scaffolding describes the specific supports introduced into the learning environment to assist the learner with developing the skill. Finally, reflecting upon the experience allows for students to transfer the learning into new contexts. This model is characterized by: a) identification of the processes of the task and making it visible to the learners; b) situating abstract tasks in authentic contexts so relevancy to workplace is evident and c) varying the diversity of situations so learners can transfer their learning into new contexts. The objective is to support the development (on their own and in combination) of three types of skills: • collaborative skills related to interprofessional communication and collaboration, • clinical-diagnostic skills, with visualization of physiological processes, thus enhancing people’s understanding of them, and • low-level skills in the actual execution of tasks and procedures

3. Framework for Evaluation of Educational Outcomes The challenge with VR and AR is not the development of the environment, but rather how to evaluate the effectiveness of achieving specific learning objectives within this environment (de Freitas et al, 2010). The four dimensional framework of de Freitas and Oliver (2006) was used as a conceptual framework for development and exploration of an immersive virtual world. Dimensions of this framework, are outlined below:

184

1.

2.

3. 4.

R. Eagleson et al. / Medical Education Through Virtual Worlds: The HLTHSIM Project

Learner specifics – includes elements such as profile, role and competencies. The learner-centered focus highlights the importance of the learner-environment interaction and the need to ensure learning activities match required outcomes. Feedback to the learner and time for reflection are provided. Pedagogy – includes consideration of models of learning, such as associative, cognitive or social/situative. The learning theory selected will influence the type of learning outcomes. The selected models may focus on task-centred approaches or socially constructed approaches. Representation – refers to the level of immersive fidelity required for experience to achieve the learning outcomes and the interactivity of the learning experience. Context – may refer to where learning occurs, whether it’s a formal or informal context, a disciplinary context or whether the learning is conceptual or applied.

The use of VR, AR and online immersive worlds for skills training, and development of collaborative skills across health professionals are emerging as important tools. The systematic design of these educational tools is key for curricula and for their relevance to re-certification policies and processes. This project addresses these areas related to a broad set of applications within the contexts of surgery and emergency medicine.

Acknowledgements Supported by NSERC, the Informatics Circle of Research Excellence (iCORE) and Advanced Education and Technology, Gov’t Alberta, and the Graphics and New Media Centres of Excellence (NCE GRAND).

References: [1] Accot,J., and Zhai,S. (1997) Beyond Fitts’ Law: Models for Trajectory-Based HCI Tasks. Proc. SIGCHI’97 Conference on Human Factors in Computing Systems. pp.295-302. [2] Albus, J.S. (1996). "The Engineering of Mind". From Animals to Animats 4: Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior. MIT Press. [3] Collins, A. (1991). Cognitive apprenticeship and instructional technology. In L. Idol, & B. F. Jones (Eds.), Educational values and cognitive instruction: implications for reform (pp. 121-138). Erlbaum [4] de Freitas, S., Rebolledo-Mendez, G., Liarokapis, F., Magoulas, G., Poulovassilis, A. (2010) Learning as immersive experiences: Using the four-dimensional framework for designing and evaluating immersive learning experiences in a virtual world. Br J Educ Tech 41:1, pp. 69-85. [5] de Freitas, S., and Oliver, M. (2006) How can exploratory learning with games and simulations within the curriculum be most effectively evaluated? Comp Educ 46, pp. 249-264. [6] Duffy, F., Gordon, G., Whelan, G., Cole-Kelly, K. and Frankel, R. (2004) Assessing competence in communication and interpersonal skills: The Kalamazoo II report. Acad Med 79:6 , pp. 495-507. [7] Fitts,P (1954) The Information Capacity of the Human Motor System Controlling the Amplitude of Movement. Journal of Experimental Psychology, 47:381-391. [8] Joint Commission on Accreditation of Healthcare Organizations. Handoff communication project. http://www.centerfortransforminghealthcare.org/projects/about_handoff_communication.aspx [9] Järvelä, S. (1995). The cognitive apprenticeship model in a technologically rich learning environment:: Interpreting learning interaction. Learning and Instruction, 5, 231-259. [10] Shanu N. Kothari, Brian J. Kaplan, Eric J. DeMaria, Timothy J. Broderick, Ronald C. Merrell. Journal of Laparoendoscopic & Advanced Surgical Techniques. June 2002, 12(3): 167-173. [11] Shepard, R and Metzler. J. (1971) "Mental rotation of 3D objects." Science. 171(972):701-3. [12] Stefanidis, D., Hope, W.W., Korndorffer Jr., J.R., Markley, S., Scott, D.J. (2010).Initial Laparoscopic Basic Skills Training Shortens the Learning Curve of Laparoscopic Suturing and Is Cost-Effective. Journal of the American College of Surgeons 210 (4), pp. 436-440.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-185

185

Ubiquitous Health in Practice: The Interreality Paradigm Andrea GAGGIOLI a,b, Simona RASPELLI a,c Alessandra GRASSI a,b, Federica PALLAVICINI a,c, Pietro CIPRESSO a,b Brenda K. WIEDERHOLD d, Giuseppe RIVA a,b a Applied Technology for Neuro-Psychology Lab., Istituto Auxologico Italiano, Italy. b ICE-NET Lab., Università Cattolica del Sacro Cuore, Italy c Centre for Studies in Communication Sciences, University of Milan-Bicocca, Italy d Virtual Reality Medical Institute, Bruxelles, Belgium

Abstract. In this paper we introduce a new ubiquitous computing paradigm for behavioral health care: “Interreality”. Interreality integrates assessment and treatment within a hybrid environment, that creates a bridge between the physical and virtual worlds. Our claim is that bridging virtual experiences (fully controlled by the therapist, used to learn coping skills and emotional regulation) with real experiences (allowing both the identification of any critical stressors and the assessment of what has been learned) using advanced technologies (virtual worlds, advanced sensors and PDA/mobile phones) may improve existing psychological treatment. To illustrate the proposed concept, a clinical scenario is also presented and discussed: Daniela, a 40 years old teacher, with a mother affected by Alzheimer’s disease. Keywords: Interreality, Virtual Reality, Biosensors, Stress, Stress Management

1. Introduction Traditionally, clinical psychologists and therapists do not use technological tools in their clinical treatment: therapy is based on face-to face interactions or other settings that involve verbal and not-verbal communication without any technological mediation [1]. However, the development of new communication technologies is influencing the therapists’ world, too. Although the key role of traditional and functional face-to-face communication is not put in discussion, new communication tools can be very useful to enhance and integrate different steps of the clinical treatment (e.g. follow up) [2]. An important role is played by telehealth, defined by Nickelson [3] as the use of telecommunications and information technology “to provide access to health assessment, diagnosis, intervention, consultation, supervision, education and information, across distance” (p. 527). The key concept behind the word “telehealth” is not the focus upon the technology but upon the process of the psychotherapy, diagnosis or of other psychological activities that con be enhanced with the use of technological media and tools. Another emerging tool is virtual reality (VR) [4; 5]. Clinicians are using VR within a new human-computer interaction paradigm where users are not passive external observers of images on a computer screen but active participants within a computer-

186

A. Gaggioli et al. / Ubiquitous Health in Practice: The Interreality Paradigm

generated three-dimensional virtual world. In VR the patient learns to manipulate problematic situations related to his/her problem. The key characteristics of virtual environments for most clinical applications are the high level of control of the interaction with the tool, and the enriched experience provided to the patient [6]. A significant challenge is to use the ubiquitous computing paradigm to integrate telehealth and virtual reality in a seamless clinical experience. To reach this goal, in the paper we introduce a new ubiquitous computing paradigm for behavioral health care: “Interreality”.

2. The Interreality Approach In this paper we suggest a new paradigm for e-health – “Interreality” (see Figure 1) that integrates assessment and treatment within a hybrid environment, bridging physical and virtual world [7-9].

Figure 1. The link between virtual and real world in Interreality

By creating a bridge between virtual and real worlds, Interreality allows a full-time closed-loop approach actually missing in current approaches to the assessment and treatment of psychological disorders:

A. Gaggioli et al. / Ubiquitous Health in Practice: The Interreality Paradigm

• •

187

the assessment is conducted continuously throughout the virtual and real experiences: it enables tracking of the individual’s psychophysiological status over time in the context of a realistic task challenge. the information is constantly used to improve both the appraisal and the coping skills of the patient: it creates a conditioned association between effective performance state and task execution behaviors.

The potential advantages offered to stress treatments by this approach are: (a) an extended sense of presence: Interreality uses advanced simulations (virtual experiences) to transform health guidelines and provisions in experience; (b) an extended sense of community: Interreality provides social support in both real and virtual worlds; (c) a real-time feedback between physical and virtual worlds: Interreality uses bio and activity sensors and devices (PDAs, mobile phones, etc) both to track in real time the behavior and the health status of the user and to provide suggestions and guidelines. To illustrate the proposed concept, a clinical scenario is also presented and discussed: Daniela, a 40 years old teacher, with a mother affected by Alzheimer’s disease.

3. Interreality: The Technology From the technological viewpoint Interreality is based on the devices/platform described below (see Figure 2): -

3D Individual and/or shared virtual worlds: They allow controlled exposure, objective assessment, provision of motivating feedbacks.

-

Personal digital assistants and/or mobile phones (from the virtual world to the real one). It allows: objective assessment, provision of warnings and motivating feedbacks).

-

Personal biomonitoring system (from the real world to the virtual one). It allows: objective and quantitative assessment, decision support for treatment.

The clinical use of these technologies in the Interreality paradigm is based on a closed-loop concept that involves the use of technology for assessing, adjusting and/or modulating the emotional regulation of the patient, his/her coping skills and appraisal of the environment (both virtual, under the control of a clinician, and real, facing actual stimuli) based upon a comparison of that patient’s behavioural and physiological responses with a baseline or performance criterion. These devices are integrated around two subsystems: the Clinical Platform (inpatient treatment, fully controlled by the therapist) and the Personal Mobile Platform (real world support, available to the patient and connected to the therapist) that allow the: a. monitoring of the patient behaviour and of his general and psychological status, early detection of symptoms of critical evolutions and timely activation of feedbacks in a closed loop approach; b.

monitoring of the response of the patient to the treatment, management of the treatment and support to the therapists in their therapeutic decisions.

188

A. Gaggioli et al. / Ubiquitous Health in Practice: The Interreality Paradigm

Figure 2. The clinical advantages of the Interreality paradigm

4. Interreality in Practice: DANIELA - A Clinical Scenario Daniela is 40 years old, and she works as a teacher in a school of her town. She has a son, Stefano, who moved three years ago to America where he works and lives with his wife and newborn baby. She doesn’t see them very often, generally only during Christmas holidays and summer time. Her husband died last year of a heart attack, and she has been living alone since then. She believes her main resource in coping with her husband’s loss has been her mother’s support, and she never imagined that her mother would also have gotten sick so soon. Indeed Daniela is actually the primary home caregiver of her mother, who is affected by Alzheimer’s disease. Since the moment her mother received the diagnosis, providing her mother with home care has become Daniela’s main activity after work. Specifically, she spends an average of five hours per day in caregiving-related activities. Since Daniela thought she had effectively coped with both her son’s departure and her husband’s death, she imagined she could also successfully cope with this new negative event; and now she is unable to accept its totally destabilizing effects on her. What makes the situation even more difficult is the fact that Daniela believes her coping efforts are ineffective: she believes she has no control over the situation,

A. Gaggioli et al. / Ubiquitous Health in Practice: The Interreality Paradigm

189

insufficient resources to cope with such a long-lasting event, and an inability to deal with the difficulties in changing identity and acquiring the social role of caregiver the current situation requires. Daniela is exposed to chronic stress and is manifesting many of the difficulties associated with psychological stress: indeed she appears to have effectively dealt with previous stressors but not the current one. Indeed, the duration of a chronic stressor, the fact that it tends to be constantly rather than intermittently present and the changes in identity or social roles frequently associated with it may contribute to the severity of the stressor in terms of both its psychological and physiological impact.

4.1. The INTERSTRESS Solution (10 Biweekly Sessions + 2/4 Boosting Sessions) Daniela will first need to accept what she is going through. This will require a cognitive restructuring activity to allow for re-appraisal of the event. This should be followed by education and training regarding useful coping responses to the type of stressors she is dealing with. In general, she has the perception that her living conditions have become more and more stressful and she doesn’t know how to deal with this increasing pressure: for this reason she has decided to go to a therapist. When she arrives, the therapist welcomes her, and this gives Daniela an immediate sense of being less alone and makes her begin to feel better. After a short interview and some paper-and-pencil self-report questionnaires, the therapist decides to use the INTERSTRESS system. She asks Daniela to wear biosensors to monitor her physiological parameters. The therapist places the non-invasive sensors on Daniela and explains their value to her, beginning the education process. Then the therapist introduces Daniela to one of the virtual worlds – the Experience Island - where she is exposed to a virtual situation similar to the real life one. Within this virtual environment, Daniela has to help her mother with daily activities. The data fusion system allows the therapist to directly index how the various stressors are impacting Daniela’s psychophysiology, thus providing an objective understanding of the different stressors and their importance and impact on Daniela’s well-being. At the end of the clinical session, the therapist “prescribes” homework for Daniela. This, she explains, will allow Daniela to be an active participant in her own well-being. This will also allow Daniela to begin to practice the skills she has started to learn, thus making them become more readily available to her during stressful situations. The homework: First Daniela needs to expose herself to the recorded critical situation in the virtual world displayed on her PDA. Then she must expose herself to the real world situation. In real world situations, the biosensors will track her response and the Decisions Support System, according to the difference from her baseline profile, will provide positive feedback and /or warnings. Finally the therapist tells Daniela that she can press a “stress” button in the PDA if she feels more stressed: this will record her experiences and they can then speak about them in the next session, allowing the two as a team to problem solve between session difficulties and how to more effectively handle future situational stressors. At the start of any new session, the therapist uses the compliance data and warning log to define the structure of the clinical work. Also, the Decision Support System will analyze the stressful situations indicated by Daniela to understand more what happened and the context in which they occurred. In the new sessions, the virtual world is not only used for assessment but also for training and education. Within the environment, Daniela has the opportunity to

190

A. Gaggioli et al. / Ubiquitous Health in Practice: The Interreality Paradigm

practice different coping mechanisms: relaxation techniques, emotional/relational management and general decision-making and problem- solving skills. For example, if Daniela’s real world outcome is poor (e.g., she can’t do a task without feeling irritable and impatient when with her mother) she will experience again a similar experience in the virtual environment and will be helped in developing specific strategies for coping with it. Later, in the relaxation areas she will enjoy a relaxing environment and learn some relaxation procedures. As with any new skill, as Daniela has the opportunity to practice the coping skills, they become second nature, and these new behaviours replace the older, outdated behavior patterns which caused the initial overwhelming stress. The therapist now prompts Daniela to also visit another virtual world – the Learning Island. Within this world, Daniela learns how to improve her stress management skills and in particular she learns about the main causes of stress and how to recognize its symptoms, learn stress-management skills such as better planning, learn stress relieving exercises such as relaxation training and get the information needed to succeed. After some sessions, the therapist invites Daniela to participate in a virtual community (under therapist supervision initially) where she will meet other individuals who are stressed like her. Within this virtual world - Community Island - Daniela has the opportunity to discuss and share her experience with other users. However, in some cases Daniela experience new critical situations that may raise her level of stress. For example, she had to discuss with her boss in the morning and this left her feeling very upset during the rest of the day. At the end of the work day, when she returned home to care for her mother, she felt very excited/stressed and nervous and the Decision Support System alerted Daniela twice about this. Both the signals were sent also to the therapist who appeared on her PDA display as an avatar suggesting to Daniela some relaxation techniques. In the following sessions, Daniela tells the therapist that she feels better thanks to being able to frequently experience stressful situations within safe virtual environments. She also says that meeting other people in the community has helped her to find much-needed support and to discover new strategies to manage her emotions. With regard to this, she says also the community experience has helped her with seeing the stressor in a new perspective. Moreover, by listening to other’s experiences, she was facilitated in adopting new coping skills. Indeed, Daniela has developed the ability to help her mother more effectively and to find time to do other things. The therapist helps Daniela to cognitively restructure the critical situation, which now she is more able to deal with through the strategies she has learned. The last session ends with advice on the prevention of relapse.

5. Conclusions The clinical use of Interreality is based on a closed-loop concept that involves the use of technology for assessing, adjusting and/or modulating the emotional regulation of the patient, his/her coping skills and appraisal of the environment (both virtual, under the control of a clinicians, and real, facing actual stimuli) based upon a comparison of that patient’s behavioural and physiological responses with a training or performance criterion. Specifically, Interreality focuses on modifying an individual’s relationship with his or her thinking through more contextualized experiential processes. To discuss

A. Gaggioli et al. / Ubiquitous Health in Practice: The Interreality Paradigm

191

and evaluate the clinical use of the proposed approach we presented and detailed both the technical characteristics of the proposed approach and a clinical scenario. Obviously, any new paradigm requires a lot of effort and time to be assessed and properly used. Without a real clinical trial, the Interreality paradigm will remain an interesting, but untested concept. However, a recently funded European project, “INTERSTRESS – Interreality in the management and treatment of stress-related disorders (FP7-247685 – http://www.intertstress.eu) - will offer the right context to test and tune these ideas.

6. Referencess [1]

[2]

[3] [4]

[5] [6] [7] [8]

[9]

G. Castelnuovo, A. Gaggioli, F. Mantovani, and G. Riva, New and old tools in psychotherapy: The use of technology for the integration of traditional clinical treatments, Psychotherapy: Theory, Research, Practice and Training 40 (2003), 33-44. G. Castelnuovo, A. Gaggioli, F. Mantovani, and G. Riva, From psychotherapy to e-therapy: the integration of traditional techniques and new communication tools in clinical settings, CyberPsychology and Behavior 6 (2003), 375-382. D. Nickelson, Telehealth and the evolving health care system: strategic opportunities for professional psychology, Professional Psychology: Research and Practice 29 (1998), 527-535. G. Riva, M. Alcañiz, L. Anolli, M. Bacchetta, R.M. Baños, F. Beltrame, C. Botella, C. Galimberti, L. Gamberini, A. Gaggioli, E. Molinari, G. Mantovani, P. Nugues, G. Optale, G. Orsi, C. Perpiña, and R. Troiani, The VEPSY Updated project: Virtual reality in clinical psychology, CyberPsychology and Behavior 4 (2001), 449-455. G. Riva, Virtual reality: an experiential tool for clinical psychology, British Journal of Guidance & Counselling 37 (2009), 337-345. A. Gorini, A. Gaggioli, C. Vigna, and G. Riva, A second life for eHealth: prospects for the use of 3-D virtual worlds in clinical psychology, J Med Internet Res 10 (2008), e21. G. Riva, Interreality: A New Paradigm for E-health, Stud Health Technol Inform 144 (2009), 3-7. G. Riva, S. Raspelli, D. Algeri, F. Pallavicini, A. Gorini, B.K. Wiederhold, and A. Gaggioli, Interreality in Practice: Bridging Virtual and Real Worlds in the Treatment of Posttraumatic Stress Disorders, Cyberpsychology, Behavior and Social Networks 13 (2010), 55-65. J. van Kokswijk, Hum@n, Telecoms & Internet as Interface to Interreality, Bergboek, Hoogwoud, The Netherlands, 2003.

192

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-192

Bench Model Surgical Skill Training Improves Novice Ability to Multitask: A Randomized Controlled Study Lawrence GRIERSONa1, Megan MELNYKb, Nathan JOWLETTc, David BACKSTEINd, and Adam DUBROWSKIa, e a

The Learning Institute, The Hospital for Sick Children, Toronto, ON, Canada b University of Ottawa, Faculty of Medicine, Ottawa, Ontario, Canada c Department of Otolaryngology, McGill University, Montreal, Quebec, Canada d Deparment of Surgery, University of Toronto, Toronto, Ontario, Canada e Wilson Centre, Faculty of Medicine, University of Toronto, Ontario, Canada

Abstract. Skills training in simulation laboratories is becoming increasingly common. However, the educational benefit of these laboratories remains unclear. This study examined whether such training enables better performance on the simultaneous execution of technical skill and knowledge retention. Twenty-four novice trainees completed the elliptical excision on baseline testing. Following baseline testing twelve of the novices completed a technical practice (simulation training group) session, while the other twelve did not (control group). One week later, all participants returned for dual-task follow up testing in which they performed the excision while listening to a didactic lesson on the staging and treatment of cutaneous melanoma. The dual-tasking during the post test was standardized, whereby excision sutures 3 and 5 were performed alone (single), and sutures 4 and 6 were performed concurrently with the didactic lecture (dual). Seven additional trainees also participated as controls that were randomized to listen to the didactic lesson alone (knowledge retention alone group). Knowledge retention was assessed by a multiple choice questionnaire (MCQ). Technical performance was evaluated with computer and expert-based measures. Time to complete the performance improved among both groups completing the elliptical excision on follow-up testing (p < 0.01). The simulation training group demonstrated superior hand motion performance on simultaneous didactic lesson testing (p < 0.01). Novices from the no-training group performed statistically worse while suturing concurrently with the didactic lesson (p < 0.01). The pretraining of novices in surgical skills laboratories leads to improved technical performance during periods of increased attention demands. Keywords. Motor skills, attention, multi-tasking

1

Corresponding Author

L. Grierson et al. / Bench Model Surgical Skill Training Improves Novice Ability to Multitask

193

Introduction It has been said that surgery is comprised twenty-five percent of technical skills and seventy-five percent of decision making skills [1]. Intra-operative decision making requires the ability to concurrently devote attention away from the technical task in order to process important information; such as, the patient’s physiologic parameters and possible complications. However, it is well known that humans have a limited attention capacity [2] that can be exceeded when two attention demanding tasks are executed simultaneously. Fitts and Posner [3] presented a model for skill acquisition in which learning develops through three distinct phases; cognitive, associative and autonomous. The cognitive phase consists of receptive learning, which involves the identification of component parts of the skill. The associative phase involves linking the component movements into a smooth action through practice and feedback. And in the autonomous phase, one becomes master of the skill and requires little or no conscious thought or attention to perform. Upon reaching the autonomous phase, learners can engage in simultaneous tasks with no detriment in technical performance. In fact, in a recent theoretical paper, Gallagher and colleagues [4] describe how master surgeons, having automated their technical skills, are able to process significantly more information than novices while operating [5]. Furthermore, it has been postulated that simulation training allows for the development of ‘pre-trained’ novices, whose technical performances have become automated to the point where they too can devote more attention away from their hands. Currently, a large proportion of medical and surgical education is based on training videos, presentations and intra-operative education by experienced surgeons. However, alternatives to surgical education are being investigated in order to address the medico-ethical implications associated with the traditional apprenticeship model [613]. The use of simulators for surgical training allows for repeated practice and improvement of a surgical skill and has the potential to reduce the attention resources required during performance. Furthermore, pre-training to the point of partial skill automation may free up novices’ attention resources and allow for a richer learning experience [4]. The current study explores how pre-training by means of bench models impacts attention resource sharing, and aims to identify means of rapid operative judgment and decision-making capacity development. Specifically, the purpose of this study is to assess if the pre-training of novices results in freeing up attention resources that can be devoted to performance on simultaneous knowledge retention tasks. The knowledge retention task used in this study is a didactic lecture and multiple-choice questionnaire on the staging and management of cutaneous melanoma. The concurrent technical task was the elliptical excision. We hypothesized that, as a result of simulation based practice and consequent partial automation of their technical performance, a group of pre-trained novices will demonstrate better performance on one or both of the simultaneous tasks when compared to a group that is not pre-trained. The idea is that any improved technique when the knowledge task is performed concurrently can be attributed to a mechanism that involves the reallocating of attention resources. In demonstrating performance improvements among pre-trained novices on one or both of the combined technical and memory tasks, we hope to support the theoretical underpinnings of the effectiveness of early simulation training in surgical education [4].

194

L. Grierson et al. / Bench Model Surgical Skill Training Improves Novice Ability to Multitask

1. Methods 1.1. Participants Thirty-one (31) third-year clerkship medical trainees enrolled at the University of Toronto volunteered for the study. Volunteers with previous experience at elliptical excision were excluded from participating. Informed consent was obtained and the study was conducted in accordance with the guidelines set out by the University of Toronto Research Ethics Board and the 1964 Declaration of Helsinki. 1.2. Procedures Participants were randomly assigned to one of three groups; knowledge retention alone group (n = 7), simulation training group (n = 12), and control group (n = 12). Participants randomized to the simulation training and control groups viewed an instructional video in its entirety before performing a baseline test of the elliptical excision. This video was made using commercial software (Adobe Premiere Pro, Adobe Systems Inc., San Jose, CA) and depicted a faculty plastic surgeon demonstrating the elliptical excision in four specific stages; sizing and outlining, incising, undermining and excising, and closing the ellipse with seven sutures. Immediately following this baseline test, the simulation training group completed a one-hour practice session with unlimited access to the instructional video and materials [8,14]. One week later, the participants from these two groups returned to complete another test of their elliptical excision ability. However, during this session, the participants performed while simultaneously listening to a scripted didactic lesson on the staging and management of cutaneous melanoma as outlined by the American Joint Committee on Cancer. The didactic lesson was read aloud by a trained health care professional and was broken into segments timed to specific stages of the task; sizing and outlining of the ellipse, undermining and excising, and during the 1st, 3rd and 5th sutures during the closing stage (dual task). Therefore, the 2nd, 4th, and 6th sutures were performed without the secondary task (single task). The timing of the lesson was done in order to accommodate individual differences in the time to complete the technical task and to allow for comparison of individual technical performance with and without the secondary task. Participants were instructed not to practice the elliptical excision during the oneweek retention period nor were they privy to the topic of the didactic lesson. Following the retention testing, all participants proceeded immediately to completing the MCQ. The MCQ was designed by a faculty surgical oncologist and was used to assess knowledge retention of the scripted didactic lesson. Participants were informed at the start of the didactic lesson that there would be a graded MCQ on the material to follow. Participants randomized to the knowledge retention alone group heard the scripted didactic lesson then immediately completed the MCQ. In order to capture the participants’ technical performance, electro-magnetic position sensors were mounted securely on the back of their hands. The sensor positions are expressed in relationship to a rigid reference body that is associated with the Imperial College Surgical Assessment Device (ICSAD) (Patriot, Polhemus, VT). This device has been previously validated as an appropriate tool for assessing hand motion economy during surgical procedures [15-17]. A porcine leg with a circular ink

L. Grierson et al. / Bench Model Surgical Skill Training Improves Novice Ability to Multitask

195

lesion one centimeter in diameter was used as a bench training model in the video and throughout all stages of the study [18].

1.3. Outcome Measures and Analyses The MCQ was graded for all three groups using a standardized marking scheme. For the two groups randomized to complete the simultaneous tasks, hand motion variables were used to assess technical performance. The specific hand motion data captured by ICSAD were the total time (s), the total number of hand movements, and the path length (m) travelled for each hand during each stage of the task (sizing and outlining, incising, undermining and excising, and each of the 7 sutures). For the purposes of this study only the dependent measures associated with the third, fourth, fifth and sixth sutures were subjected to comparative analysis. Focusing on this four suture portion of the excision was particularly interesting because it afforded us the opportunity to identify any differences that emerge between procedures performed with an additional (Sutures 3 and 5) demand on attention and those performed without any secondary demand on attention’s resources (Sutures 4 and 6). We omitted sutures 1 and 2 from analysis to avoid comparing outcomes that could be related to the well-documented warm-up decrement [19]. Suture 7 was similarly omitted from analysis because it, as the finishing knot, was shown to be categorically different than the previous four sutures. In order to ensure the sutures were equally difficult and that the simulation training and control groups were equivalent in their initial ability to suture closed the elliptical excision, their baseline data for each dependent measure was subjected to a 2 Group (simulation training, control) X 4 Suture (s3, s4, s5, s6) analysis of variance (ANOVA). To assess the impact of training on the participants’ ability to perform the sutures in the presence of secondary demands on attention, their each measure of retention data was subjected to a 2 Group (simulation training, control) X 2 Attention Condition (Dual Task, Single Task) X 2 Suture (1st (i.e., suture 3 and suture 4), 2nd (i.e., suture 5 and suture 6) ANOVA. A one-way ANOVA was employed to uncover any differences in MCQ performance between the three groups. Effects significant at p < 0.05 were decomposed using Tukey’s HSD method of post hoc analysis.

2. Results 2.1. Baseline The baseline analyses of the mean number of movements inherent to, and the mean path length of, the sutures performed by the participants revealed no significant differences between any of the sutures or the groups. However, the analysis of the mean time to complete each of the sutures revealed significant main effects for group, F (1, 22) = 11.93, p = .0022, and suture, F (3, 66) = 6.70, p < .0005. Post hoc analysis of the group effect indicated that the control group (62.0 ± 15.9 s) took significantly longer than the simulation training group (46.3 ± 3.3 s) to complete each of the sutures. The decomposition of the suture main effect indicated that the 3rd (60.7 ± 3.7 s) and the 4th (54.0 ± 2.9 s) sutures did not differ from each other but were performed over a significantly longer period than the 5th (52.2 ± 3.5 s) and 6th (49.7 ± 2.7)

196

L. Grierson et al. / Bench Model Surgical Skill Training Improves Novice Ability to Multitask

sutures, which did not differ from each other. Importantly, this indicates that the performance comparisons carried out in our retention analyses of sutures performed with and with-out a challenge to attention are homogeneously appropriate. 2.2. Retention The analysis of the mean number of movements made when performing the retention sutures revealed a significant main effect for suture, F (1, 22) = 7.33, p < .013. Post hoc analysis of this effect revealed that the first suture (50.0 ± 2.9) in either attention condition (i.e., 3rd and 4th sutures) were performed with significantly more movements than the conditions’ second sutures (36.9 ± 3.19) (i.e., 5th and 6th sutures). The analysis of the mean path length participants performed when making the sutures revealed main effects for group, F (1, 22) = 5.15, p < .034, and suture, F (1, 22) = 6.35, p < .02. Post hoc analysis of the group main effect revealed that the path lengths of the simulation training group (16.2 ± 1.50 cm) were significantly shorter than those of the control group (21.5 ± 2.2 cm). In a fashion similar to the number of movements analysis, the follow-up analysis of the suture main effect showed that the first sutures (19.8 ± 1.4 cm) had significantly longer path lengths than the second sutures (17.8 ± 1.5 cm); regardless of attention condition. The retention analysis of the mean time to complete the sutures revealed a significant suture main effect, F (1, 22) = 5.27, p < .046, in which the first sutures (51.1 ± 3.2 s) took significantly more time than the second sutures (46.8 ± 3.6 s). Interestingly, the analysis of mean time to complete the sutures also yielded a significant group by attention condition interaction, F (1, 22) = 5.27, p < .032. The post hoc decomposition of this finding showed that while the control group performed equally as quickly as the simulation training group in the single-task attention condition, they were significantly slower when they had to deal with the dual-task challenge (Fig. 1). The simulation training group did not differ in mean suturing time in either attention condition. Importantly, the lack of a group main effect for this finding (p = .063), suggests that the interaction is not a direct function of the time differences yielded from the baseline analysis.

Figure 1. Mean time (s) to complete one suture during one-week follow-up simultaneous task testing.

L. Grierson et al. / Bench Model Surgical Skill Training Improves Novice Ability to Multitask

197

2.3. MCQ Analysis Surprisingly, there were no statistical differences in MCQ scores among the three groups in our study. Considering the fact that all three groups (simulation group = 9.25 ± 2.34/25; control group = 8.25 ± 2.37/25; knowledge retention alone group = 8.62 ± 2.01/25) performed the MCQ at a level below a passing grade (<60%), this lack of differences may be best explained by the high difficultly of the lecture material and questions.

Conclusions This study sought to determine whether the technical improvement associated with training on a surgical skill would offer trainees any advantage in performing under conditions of higher cognitive demand; specifically, when asked to perform the skill while attempting to extract simultaneously-presented information extraneous to the task at hand. To do this we compared individuals who participated in a training session with those who did not, in their abilities to perform suturing skill during an elliptical excision while concurrently listening to an unrelated didactic lecture with the purposes of answering questions about the information at a later time. Our primary hypothesis was based on the learning model of Fitts and Posner [3], which explained that through practice we move from a highly cognitive and attention demanding state of performing to one that requires less of the available attention resource. With this in mind, we expected that through the training session, performers would cognitively compartmentalize the stages of the excision in an associative manner that would, in turn, increase their capacity to devote attention to the lecture, allowing them to recall the lecture information more effectively while not impacting their ability to perform the skill. To some degree, the study results support this hypothesis; however, not as plainly as demonstrating that the trained participants exhibit more efficient excision movements and lecture test scores than the untrained group. Rather, the present results reveal that while sutures were performed equally as well by both the trained and untrained groups when done so in the absence of attention distraction, the trained individuals in the dual-task situations completed the sutures the same as in the single task conditions. A surprising lack of statistical differences in test scores among the three study groups and the overall low scores indicated that the MCQ was difficult. However, considering that distractions within the operating room are common, it is our contention that the didactic lesson continued to serve as an appropriate challenge to attention resources for interpreting how training impacts the way technical performance is mitigated by secondary task engagement. Thus, although the training did not seem to provide our participants with any apparent additional attention capacity, it did prevent them from causing detriment to their technical performance when bifurcating their focus of attention. In this way we must consider the possibility that attempting to pay attention to information secondary to the completion of the excision, that is the mere refocusing of attention, requires a small measure of cognitive resource; over which the trained group had gained access. If this is indeed the case then extending training and/or reducing the difficulty of the lecture material should yield clearer results.

198

L. Grierson et al. / Bench Model Surgical Skill Training Improves Novice Ability to Multitask

Acknowledgements This research was supported by the Natural Sciences and Engineering Research Council of Canada.

References [1] F.C. Spencer, Teaching and measuring surgical techniques - the technical evaluation of competence, Bulletin of the American College of Surgeons 63 (1978), 9-12. [2] D.E. Broadbent, Selective and control processes, Cognition 10(1981), 53-58. [3] P.M. Fitts, M.I. Posner, Learning and Skilled Performance in Human Performance, Brock-Cole, Belmont, CA, 1967. [4] A. G. Gallagher, E. M. Ritter, H. Champion, G. Higgins, M. P. Fried, G. Moses, D. Smith, and R. M. Satava, Virtual reality simulation for the operating room: Proficiency-based training as a paradigm shift in surgical skills training, Annals of Surgery 241(2005), 364-372. [5] K.E. Hsu, F.Y. Man, R.A. Gizicki, L.S. Feldman, G.M. Fried, Experienced surgeons can do more than one thing at a time: Effect of distraction on performance of a simple laparoscopic and cognitive task by experienced and novice surgeons, Surgical Endoscopy 22(2008), 196-201. [6] R.W. Barnes, Surgical handicraft: Teaching and learning surgical skills, American Journal of Surgery 153(1987), 422-427. [7] J.M. Hamdorf, J.C. Hall, Acquiring surgical skills. [review], British Journal of Surgery 87(2000), 28-37. [8] N. Jowett, V. LeBlanc, G. Xeroulis, H. MacRae, A. Dubrowski, Surgical skill acquisition with selfdirected practice using computer-based video training, American Journal of Surgery 193(2007), 237242. [9] R. Kneebone, D. ApSimon, Surgical skills training: Simulation and multimedia combined, Medical Education 35(2001), 909-915. [10] R. Kneebone, Simulation in surgical training: Educational issues and practical implications, Medical Education 37(2003), 267-277. [11] R. K. Reznick, Teaching and testing technical skills, American Journal of Surgery 165(1993), 358-361. [12] R. K. Reznick, G. Regehr, H. MacRae, J. Martin, W. McCulloch, Testing technical skill via an innovative "bench station" examination, American Journal of Surgery 173(1997), 226-230. [13] M. Bridges, D. L. Diamond, The financial impact of teaching surgical residents in the operating room, American Journal of Surgery 177(1999), 28-32. [14] A.N. Summers, G.C. Rinehart, D. Simpson, P.N. Redlich, Acquisition of surgical skills: A randomized trial of didactic, videotape, and computer-based training, Surgery 126(1999), 330-336. [15] S.D. Bann, M.S. Khan, A.W. Darzi, Measurement of surgical dexterity using motion analysis of simple bench tasks, World Journal of Surgery 27(2003), 390-394. [16] V. Datta, S. Mackay, M. Mandalia, A. Darzi, The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model, Journal of the American College of Surgeons 193(2001), 479-485. [17] S. Mackay, V. Datta, M. Mandalia, P. Bassett, A. Darzi, Electromagnetic motion analysis in the assessment of surgical skill: Relationship between time and movement, ANZ Journal of Surgery 72(2002), 632-634. [18] D.J. Anastakis, G. Regehr, R.K. Reznick, M. Cusimano, J. Murnaghan, M. Brown, C. Hutchinson, Assessment of technical skills transfer from the bench training model to the human model, American Journal of Surgery 177(1999), 167-170. [19] J. A. Adams, The second facet of forgetting: A review of warm-up decrement, Psychological Bulletin 58(1961), 257-273.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-199

199

A Design of Hardware Haptic Interface for Gastrointestinal Endoscopy Simulation a

Yunjin GUa and Doo Yong LEE a, 1 Department of Mechanical Engineering, KAIST, Daejeon, Korea

Abstract. Gastrointestinal endoscopy simulations have been developed to train endoscopic procedures which require hundreds of practices to be competent in the skills. Even though realistic haptic feedback is important to provide realistic sensation to the user, most of previous simulations including commercialized simulation have mainly focused on providing realistic visual feedback. In this paper, we propose a novel design of portable haptic interface, which provides 2DOF force feedback, for the gastrointestinal endoscopy simulation. The haptic interface consists of translational and rotational force feedback mechanism which are completely decoupled, and gripping mechanism for controlling connection between the endoscope and the force feedback mechanism. Keywords. Haptic interface, medical simulation, gastrointestinal endoscopy

Introduction Virtual reality based medical simulation has been introduced to train novices in gastrointestinal endoscopic procedures which require at least 100 to 180 procedures to be competent in the skills. Previous simulations including commercialized simulation [2, 4, 7-9] have mainly focused on providing realistic visual feedback, and consideration on the haptic feedback has been relatively ignored even though it is also important to provide realistic sensation to the user. In this paper, we propose a novel design of a portable haptic interface, which provides 2DOF force feedback, for gastrointestinal endoscopy simulation.

1. Design Objective Previous haptic interfaces for endoscopy simulation [1-6] can be classified into two types. The first type provides force feedback through the force feedback mechanism which is rigidly connected with the endoscope [3, 4, 6]. The force feedback mechanism moves along with the endoscope, and usually there is no slip between them. Hence, it measures position accurately and provides precise and high bandwidth force feedback. One drawback of this type of device is that it should cover whole workspace of the simulation, so the size of the device is relatively large compared to the other type of devices. Large inertia is another drawback of this type of haptic interface. The second 1

Corresponding Author: Professor, Department of Mechanical Engineering, KAIST, Daejeon, Korea; E-mail: [email protected]

200

Y. Gu and D.Y. Lee / A Design of Hardware Haptic Interface

type of haptic interface provides force feedback through a friction mechanism which is in contact with endoscope. Those devices usually have relatively small inertia and compact size. But slip between the endoscope and the device can degrades accuracy in the position measurement and performance of the force feedback. In this paper, we propose a new type of haptic interface which can overcome the problems of the previous haptic interfaces. The haptic interface selectively connects the endoscope with the force feedback mechanism only when it provides feedback force. The endoscope can move freely when there is no feedback force. Then, the haptic interface does not have to contain the whole workspace of the simulation, and also no moment of inertia exists during free motion of the endoscope.

2. Design The haptic device consists of force feedback mechanism for providing translational and rotational force, and gripping mechanism for selectively connecting the endoscope with the force feedback mechanism. The gripping mechanism is shown in Figure 1. The structure of the gripping mechanism is constructed in the outer ring. The main role of the gripping mechanism is to grip and tightly connect the endoscope with the force feedback mechanism. It is the most important mechanism in the design because tight connection between the endoscope and the device is essential for providing precise and transparent force feedback. An electric motor acts as a switch for gripping or releasing the endoscope. The motor does not have to keep the torque during gripping. In the rotational force feedback mechanism, the outer ring is connected with an electric motor which produces the rotational feedback force. The rotational force produced by the motor is transferred to the endoscope through the gripping mechanism which is rigidly attached to the outer ring. The translational force feedback mechanism is shown in Figure 2. Two plates are attached on the both side of the outer ring, and they move along the rails in the base cylinder as shown in Figure 2. The two plates are attached to the outer ring through bearings which allows relative motion in roll direction, so the rotational and the translational motion can be decoupled.

Figure 1. Gripping and rotational force feedback mechanism

Y. Gu and D.Y. Lee / A Design of Hardware Haptic Interface

201

Figure 2. Translational force feedback mechanism

3. Conclusion A novel design of haptic interface for gastrointestinal endoscopy simulation is proposed. When the feedback is not provided, the haptic device releases the endoscope and the endoscope can move freely. When the feedback is provided, the gripping mechanism grips the endoscope. Translational and rotational movements of the endoscope are decoupled. Since fixing the endoscope is critical for sensible feedback, the gripping mechanism is the most important mechanism. When the gripping mechanism works, the moment of inertia of the part connected with the endoscope is changed. As the future work, haptic control methods and haptic renderings, considering the change of the moment of inertia, have to be developed.

Acknowledgement This work was supported by Mid-career Researcher Program through NRF grant funded by the MEST(No.2010-0000093).

References [1] K. Ikuta, K. Iritani, and J. Fukuyama, Mobile Virtual Endoscope System with Haptic and Visual Information for Non-invasive Inspection Training, Proceedings of the 2001 IEEE International Conference on Robotics & Automation (2001), 2037-2044. [2] KEYMED LTD. International Publication Number WO 03/050783. [3] O. Körner, R. Männer, Implementation of a Haptic Interface for a Virtual Reality Simulator for Flexible Endoscopy, Proceedings of the IEEE 11th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (2003), 22-23. [4] HT Medical Systems Inc. U.S Patent 2004/0048230 A1 [5] D. Ilic, T. Moix, N. Mc Cullough, L. Duratti, I. Vecerina, and H. Bleuler, Real-Time Haptic Interface for VR Colonoscopy Simulation, Proceedings of the 13th Annual Conference on Medicine Meets Virtual Reality (2005), 208-212. [6] J.Y. Kwon, H.S. Woo, D.Y. Lee, Haptic Device for Colonoscopy Training Simulator, Proceedings of the 13th Annual Conference on Medicine Meets Virtual Reality (2005) 277-279. [7] Simbionix Ltd.[Online]. Available: http://www.simbionix.com [8] Immersion Co.[Online]. Available: http://www.immersion.com [9] Simbionix, U.S. Patent 6,857,878 B1

202

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-202

Open Surgery Simulation of Inguinal Hernia Repair Niels HALD a, Sudip K SARKER b, Paul ZIPRIN a, Pierre-Frederic VILLARD c and Fernando BELLO a,1 a Department of Surgery and Cancer, Imperial College London, UK b Whittington Hospital, London, UK c LORIA, Nancy University, Nancy, France

Abstract. Inguinal hernia repair procedures are often one of the first surgical procedures faced by junior surgeons. The biggest challenge in this procedure for novice trainees is understanding the 3D spatial relations of the complex anatomy of the inguinal region, which is crucial for the effective and careful handling of the present anatomical structures in order to perform a successful and lasting repair. Such relationships are difficult to illustrate and comprehend through standard learning material. This paper presents our work in progress to develop a simulation-based teaching tool allowing junior surgeons to train the Lichtenstein tension-free open inguinal hernia repair technique for direct and indirect hernias, as well as to enforce their understanding of the spatial relations of the involved anatomy. Keywords. Virtual reality, open surgery, hernioplasty, simulation, e-learning

Introduction An inguinal hernia is an unnatural protrusion of abdominal contents through a weakness in the abdominal wall of the inguinal area. The treatment of inguinal hernias is one of the most common surgical procedures in the US and Europe [1], with open mesh repair being the preferred method of treatment in many European countries [2]. Improvements in patient safety in recent years has lead to increased supervision of training surgeons, which in turn prevents the junior surgeons from performing surgical tasks independently [3]. While these improvements are a clear benefit for the safety of the patients, it increases the burden on senior surgeons and obstructs junior surgeons from gaining the confidence from performing procedures independently [3]. From our literature review and discussions with surgeons, it has become apparent that one of the hardest and most crucial challenges in repairing inguinal hernias is understanding the three dimensional spatial relations of the complex anatomy in the inguinal region [4]. It is our belief that these spatial relations are best taught using three-dimensional models in order to minimize the level of abstraction required by the trainee. A wide range of teaching tools utilizing 3D or pseudo 3D are available for teaching the anatomy of the inguinal region and hernias, ranging from paper cut [5] and plastic models, to simple animation based simulators such as the one provided by Simbionix 1

Corresponding Author: Fernando Bello, [email protected]

N. Hald et al. / Open Surgery Simulation of Inguinal Hernia Repair

203

(http://www.etrinsic.com/). However, the level of interaction and user involvement in the currently available tools is restricted. Our goal is to develop a system that can contribute to the confidence, skills and knowledge of the training surgeons, with a special focus on teaching the anatomy and its spatial relations, as well as the tasks involved in the repair procedure. By doing so, we aim to optimise training by senior surgeons and further improve patient safety by helping trainee surgeons progress through the learning curve faster. We are building our system around the Lichtenstein repair technique. This technique has been chosen over other open repair techniques due to the wide acknowledgement and scientific evidence supporting it [1]. The ultimate goal is to deploy our system as an e-learning application, allowing surgeons easy access to the application, without the dependence of expensive specialized hardware. Based on our initial analysis and goal, we have arrived at the following overall specifications for our system: • • • • •

The system should be web-based and ensure easy and independent accessibility The system should require a minimum of hardware - this means no requirements of specialized equipment, such as haptic devices and the possibility to run the application on a standard laptop The system should not require supervision to use and only require basic knowledge of the anatomy and the procedure The system should be intuitive, allowing trainees to use it independently There is a high emphasis on realistic anatomical models and the ability to explore these

The remainder of the paper details the analysis and design of the system, current implementation and results, as well as future planned work.

1. Methods Our system has been designed around a hierarchical task analysis (HTA) [3], as well as in continuous correspondence with expert surgeons and observations made in the operating theatre. Using a HTA of the Lichtenstein procedure as a foundation, we have compiled an implementation plan detailing A) the virtual environment, B) the user interactions and C) the requirements of deformable models. The implementation plan has then been used to guide the system development according to the overall specifications outlined above. The next sections will cover the use of the HTA, the virtual environment, the user interactions and the deformation models. 1.1. HTA in System Design The HTA is a systematic chronological breakdown of a procedure, detailing tasks, subtasks, roles and branching of tasks. We decided to use the HTA as a foundation for an implementation plan in order to have a solid framework for our discussions with surgeons during the design phase of our project. The HTA for the Lichtenstein procedure used here is presented in [3]; it contains 16 tasks and 46 sub-tasks. Since the individual subtasks require various skills, it has been our goal to determine the most suitable implementation for each individual task that would allow our system to convey

204

N. Hald et al. / Open Surgery Simulation of Inguinal Hernia Repair

the crucial details of the surgical procedure. In collaboration with surgeons resident at St Mary’s and Whittington Hospitals (London, UK), the HTA was used to iteratively develop and expand the implementation plan on which the simulation system was based. The final version of the implementation plan encompasses the HTA itself, input from the literature review, observations from video recordings of inguinal hernia repair surgery (eleven open and one laparoscopic), observations from hernia repair procedures (three laparoscopic, two open), and continuous feedback, discussions and input from expert surgeons. The plan contains details on how we intend to implement the 46 subtasks of the procedure. 1.2. Environment Our environment is implemented in Java, using Java3D (https://java3d.dev.java.net/) for 3D rendering. The 3DScience (http://www.3dscience.com/) models by the Zygote Media Group are used as a basis for the anatomical models of skin, muscles, intestine, ligament, bone, blood vessels and nerves. Whilst these models are of high quality and suitable for general visualization and rendering purposes, the level of anatomical detail present does not accurately reflect the anatomy of the inguinal region: the inguinal canal is not present, while the spermatic cord is in a position that bears little resemblance to where it is expected through normal growth and development. We have undertaken the careful modification and extension of the 3DScience models to correctly reflect this anatomy in close collaboration with surgeons. Modification of the 3D models and generation of additional models has been carried out using the Blender 2.5 software package (http://www.blender.org/). 1.3. User Interactions Our implementation plan contains a wide range of different user interactions, such as selection and application of the correct tool, selection from multiple interaction choices (e.g. location of incision), 3D navigation and manipulation and exploring the anatomy. We have carefully designed the individual interactions to focus on the key elements to be taught to the trainees. For instance, when incising the external oblique muscle, we focus on teaching the trainee where to cut, not how to cut. As a result, rather than cutting freely, we present the user with a series of incision sites, where the user will have to choose the correct location for the incision, thus allowing the system to easily detect and feedback to the user when he/she tries to perform an incision in an erroneous location. Other tasks, such as the mobilization of the spermatic cord, give the user freedom to drag, twist and explore. The chronological flow of the procedure is controlled using state machines. The purpose of the state machines is both to ensure that the user does not divert from the correct order of execution of the tasks in the procedure, as well as to determine when the different deformation models should be active or paused to minimize computational load. One general state machine governs the overall flow of the tasks in the procedure, while a number of sub-state machines govern the progression between the different interactions in the procedure. When the execution of all the state machines comes to an end, the procedure has been completed and the user is presented with feedback on her/his performance. A very important aspect of feedback is to highlight any erroneous actions, together with

N. Hald et al. / Open Surgery Simulation of Inguinal Hernia Repair

205

recommendations of how to execute the same action correctly. Examples of wrong actions could be: placing the polypropylene mesh at a wrong angle with respect to the spermatic cord, or attempting to perform the incision in the skin with disregard to anatomical landmarks. 1.4. Deformation Modelling The anatomy of the inguinal region differs greatly in its geometric and physical properties. We have put much consideration into choosing deformation models that are well suited to handle the properties, geometry and interaction that are required in our simulation system. The anatomy to be modelled has very different geometries and properties. As a consequence, various models are required to simulate them. They can be categorized in terms of their geometry as follows: • • • •

Rods – The spermatic cord and, to some extend the hernia sack, are both cylindrical structures Surfaces – The external oblique is a very thin surface Volumes – The layer of skin and fat is a thick layered volume Bundles – The fibrous net in the inguinal canal consists of a chaotic network of fibres attached to the spermatic cord

The CoRdE or Cosserat rod model [6, 7] was chosen for the spermatic cord as this model encompasses the geometry of the cord, while supporting the necessary pulling and twisting that the cord is subject to while the surgeon mobilises it and explores its contents. Direct and indirect hernias are also modelled using the CoRdE model. We have explored methods for linking the indirect hernia sack to the spermatic cord. One method investigated is the use of simple springs, but Cosserat elements turned out to be a more feasible method. The Cosserat elements allow the models not only move together, but also bend and twist together. We extended the CoRdE model to a surface in a similar way to the net described in [7] in order to yield the surface models required for the external oblique muscle, and the polypropylene repair mesh. To ensure coherence with the interactions required for the external oblique in our system, we developed a cutting scheme for this Cosserat surface model, which is performed by splitting a control point into two control points and reassigning local elements to ensure no exchange of bending or translational forces between the two points, resulting in the control points coming apart. Simple individual randomly generated springs were used to simulate the chaotic network of fibres holding the anatomy of the inguinal region in place. Each fibre-spring connects one of the deformation models to the walls of the inguinal canal and holds them in place until the user dissects the fibres. We chose a spring-based Free-Form Deformation (FFD) model, similar to the dynamic FFD model of SOFA (http://www.sofa-framework.org/), to simulate the deformation and cutting of the skin and the underlying adipose tissue. Cutting is enabled by aligning two spring-based blocks over the incision site, locking the mass points together and letting the user gradually dissolve the locks using a cutting tool.

206

N. Hald et al. / Open Surgery Simulation of Inguinal Hernia Repair

2. Results One of our key goals was to make the virtual environment simple and recognizable to the trainees to ensure that they will be able to intuitively understand and interact with it. To achieve this, we have made our virtual operating theatre a close, but simpler, recreation of a real operating theatre. This involved building 3D models of the necessary tools and equipment in the operating theatre. Our virtual operating theatre and adapted 3D models can be seen in Figure 1.

Figure 1. To the left, the initial view of the virtual operating theatre. To the right, the 3D anatomical models, including muscle, vessels, ligaments, bone and facia.

A selection of the various interactions possible is shown in Figure 2.

Figure 2. Clockwise from top left: The untreated patient, betadine applied, drape applied, incision in the fat, retraction of the fat, incision in the external oblique, lifting the external oblique with clips, grabbing the spermatic cord.

Based on our experiments, a Cosserat rod model of only twenty elements suffices to define the centreline of the spermatic cord and provide a fast and realistic biomechanical deformation. Our initial validation of this model by surgeons has

N. Hald et al. / Open Surgery Simulation of Inguinal Hernia Repair

207

indicated that it performs in a convincing manner. We have mapped a textured cylindrical mesh and animations onto the model, as shown in Figure 3.

Figure 3. The Cosserat rod based spermatic cord and attached hernia. From the top left: the cord, the cremasteric muscle opened, the hernia partly separated, the hernia separated, the hernia sack opened, the contents of the hernia sack reduced and the hernia sack incised.

The Cosserat surface resulting from our extension of the CoRdE model is highly stable and an abundance of physical parameters allow us to modify the surface to yield the desirable behaviour and configuration. Figure 4 compares the polypropylene mesh and the Cosserat surface subject to gravity.

Figure 4. To the left the polypropylene mesh, to the right the un-textured Cosserat surface under gravitational pull. The spheres in the Cosserat surface represents control points, while the grid and lines represents bending elements.

Whilst our method for cutting the Cosserat surfaces described above is limited to splitting control points, it satisfies our needs well, as the correct incision will be predefined, and the emphasis is on choosing the correct incision site from a set of multiple possible locations, rather than performing the actual cut manually.

208

N. Hald et al. / Open Surgery Simulation of Inguinal Hernia Repair

3. Discussion and Conclusion This paper presents the design and initial implementation of a virtual reality system for open inguinal hernia repair. Our use of a HTA in the design and implementation phase of the project assures its clinical relevance and authenticity. The subdivision of the surgical tasks in the HTA proved useful in the design phase of our system - not only did the HTA break down a complex operation to simpler tasks and sub-tasks, it also provided the foundation for useful and productive discussions between clinical staff and developers. We have found the CoRdE model to be a flexible and powerful model. While the per-element computations are relatively heavy, the model requires only very few elements to handle complex and large deformations. At the same time, the quantity of adjustable physical parameters in the model allows us to simulate a wide range of materials with the same model. Our extension of the rod model to a surface model has resulted in a very stable surface. While the cutting scheme in our Cosserat surface can only be performed at control points, it is suitable for our purposes. Our future work on this system will encompass further validation and the final stages of development, including summative and formative feedback. Validation of the modifications made to the anatomical 3D models is required to ensure that the models correctly reflect the anatomy of the inguinal region. Validation of the interactions, deformation models, and integrated system to evaluate its value as a teaching and learning tool are all planned. Further development of the tasks involved in the procedure is required to ensure the natural and correct flow of the procedure. The ultimate aim of this work is to assist trainees progression through the learning curve and, as a result, improve patient safety, the quality of future surgeons, while relieving some of the pressure on senior consultants.

Acknowledgement This research is funded by the London Deanery (http://www.londondeanery.ac.uk/).

References [1] [2]

[3] [4] [5]

[6] [7]

P. Primatesta & M. J. Goldacre, Inguinal Hernia Repair: Incidence of Elective and Emergency Surgery, Readmission and Mortality, International Epidemiological Association 25 (1996), 835-839. M. P. Simons, T. Aufenacker, M. Bay-Nielsen, J. L. Bouillot, G. Campanelli, J. Conze, V. Schumpelick, D. de Lange, R. Fortelny, T. Heikkinen, A. Kingsnorth, J. Kukleta, S. Morales-Conde, P. Nordin, S. Smedberg, M. Smietanski, G. Weber & M. Miserez: European Hernia Society guidelines on the treatment of inguinal hernia in adult patients, Hernia 13 (2009), 343-403 S. K. Sarker, A. Chang, T. Albrani, C. Vincent, Constructing hierarchial task analysis in surgery, Surgical Endoscopy 22 (2008), 107-111 C. M. Townsend, R. D. Beauchamp, B. M. Evers, K. L. Mattox, Sabiston Textbook of Surgery: The Biological Basis of Modern Surgical Practice 16th Ed., W.B. Saunders Company, Philadelphia, 2001 B. D. Mann, A. Seldman, T. Haley & A. K. Sachdeva, Teaching Three-dimensional Surgical Concepts of Inguinal Hernia in a Time-effective Manner Using a Two-dimensional Paper-cut, The American Journal of Surgery 173 (1997), 542-545 J. Spillmann & M. Teschner: CoRdE: Cosserat Rod Elements for the Dynamic Simulation of OneDimensional Elastic Objects, ACM SIGGRAPH Symposium on Computer Animation (2007), 1-10 J. Spillmann & M. Teschner: Cosserat Nets, IEEE Transactions on Visualization and Computer Graphics 15 (2009), 325-338

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-209

209

SML: SoFMIS Meta Language for Surgical Simulation a

Tansel HALICa and Suvranu DEa,1 Department of Mechanical, Aerospace and Nuclear Engineering, Rensselaer Polytechnic Institute

Abstract. Developing a surgical simulation is a very complex task, requiring various components and usually the involvement of multiple people. As the complexity of the simulator increases, the functionality and the functional semantics of the components become very challenging to manage. In order to mitigate these issues, in this work we introduce a novel language that facilitates the development of surgical simulations and also supports semantic embedding in the surgical simulation development without extra programming effort. Keywords. Surgery simulation development, Language design, Semantics derivation.

Introduction In order to increase the effectiveness of the simulation design and development, we devised a new language called SoFMIS Meta Language (SML) as part of our inhouse developmental software framework, SoFMIS (Software Framework for Multimodal Interactive Simulations). In a typical simulation, the majority of the functionalities must occur immediately after a simulation action is triggered. For instance, in a typical surgical simulation when the cauterization starts, buzzing sound starts. This replicates the actual sound of an electrocautery device. Therefore, the sound action is triggered if and only if the cauterization starts. As another example, in a vehicle simulation the switching on of the ignition starts the engine. During the simulation development, these actions are taken care of internally by the developers. In complex simulation environments, due to increased code complexity and multiple developers’ input, intricacies arise, rendering difficult both the management of the code and the tracking of the flow of the overall simulation logic. To solve these issues, SML seeks to easily create these logical actions and make the flow of the simulation more manageable. To this end, it provides syntax for the developers to define their simulation actions within the framework. Functions of modules or newly implemented components could be easily connected to each other to carry out the simulation logic. This entire process simplifies the development phase and also promotes collaboration between simulation developers and designers. In addition to functionality, language also allows the derivation of the simulation flow from the input syntax. This language, unlike other similar approaches, can express functionality and 1

Corresponding Author: Dr. Suvranu De, Department of Mechanical, Aerospace and Nuclear Engineering, Rensselaer Polytechnic Institute, Email: [email protected]

210

T. Halic and S. De / SML: SoFMIS Meta Language for Surgical Simulation

semantics because it does not require extra effort to define semantics bindings and actions [1-3].

1. Methods and Materials This language is an enriched add-on for C++, specifically designed for real-time simulations with SoFMIS. The SML uses the underlying messaging system of the framework, which simplifies message sending and receiving during the development phase. Therefore, in the design phase, the message flow between the components can be easily generated. Since the SML functionality is primarily built on the top of the message system, it is trivial to visualize the semantics flow. The messaging system acts as a separate component in the SoFMIS, whose main responsibility is to deliver messages to the registered event listeners. Therefore, in each simulation frame the messaging component periodically checks the messages in its buffers and delivers based on the priority. The development with SML is performed in the C++ code. The SML syntax is directly embedded in the C++ files. Our own SML compiler performs the compilation and generates the meta header and implementation files before the actual compilation is done (see Figure 1). To this end, the meta-compiler reads the syntax declared in the classes and based on the language definition, it generates the implementation code which performs the necessary links to the SoFMIS messaging system. The SML defines the set of actions that will take place in case the rules are valid. Any incorrect language rules are directly reported as error to the developers.

Figure 1. Compilation phase of SML embedded code.

In the current language definition, there are two major actions: emit and receive. These actions respectively send and obtain messages when the associated rule occurs. At present, there are four rules: always (emit or receive messages all the time), change in variable (emit or receive when there is a change in the corresponding variable), conditional statement (emit or receive if a conditional is true) and incase (emit or receive if another message is received);

T. Halic and S. De / SML: SoFMIS Meta Language for Surgical Simulation

211

Figure 2. Current SML BNF Definition

The language structure is defined as context free grammar and in Backus-Naur Form [4]. Current preliminary SML syntax is listed in Figure 2. Based on the syntax, each statement has a beginning and end. Within the statement, the message body has the action and its corresponding rule statement. The rule statement can also include more than one message through the incase statement. This incase statement serves as a condition during the message reception, but it also serves as a way of emitting multiple messages. There are other actions and constructs defined for visualization, debugging, flow visualization, data structure support and other purposes that increase the functionality of the language. Although the language has compilation-time binding, we also introduce dynamic binding for many-to-many relationships which could only be resolved during runtime. As an example, in a surgery simulation, special actions should be taken when particular blood vessels are perforated accidentally, such as triggering massive blow flow. Therefore, the virtual object that represents the vessel should be dynamically bound to the proper blood flow module. The corresponding vessel object and the blood flow object are registered to the dynamic binding module of SML that creates the one-to-one association.

2. Results We tested the language in our existing laparoscopic adjustable gastric band (LAGB) simulator [5],[6]. We wrote the data record component responsible for keeping track of the events during the simulation for post-assessment of the surgeon’s skills. During the surgery, the activities such as bleeding, cutting or accidental perforations were sent to the data record with SML bindings embedded in the implementation. The automatically derived semantics with SML is illustrated in Figure 3.

212

T. Halic and S. De / SML: SoFMIS Meta Language for Surgical Simulation

Figure 3. Case Study: An Application of the SML for LAGB simulator

3. Conclusions Our SML successfully derives the semantics and simulation flow. Moreover, it simplifies the development and design phase since most of the code is automatically generated with our SML compiler on behalf of the simulator developers. In addition, the semantics of the language allowed us to obtain useful context information about the simulation logic flow. Further language functionality, such as support of more complex data types and classes are part of our ongoing work. Acknowledgements The authors gratefully acknowledge the support of the NIH/NIBIB through grant #R01EB005807. We also thank Dr. Ganesh Sankaranarayanan on his thoughtful suggestions and proof reading this paper.

References [1]

[2]

[3] [4] [5]

[6]

S. Irawati, D. Calderón, and H. Ko, “Spatial ontology for semantic integration in 3D multimodal interaction framework,” ACM international conference on Virtual reality continuum and its applications, 2006, p. 135. J. De Boeck, C. Raymaekers, and K. Coninx, “Comparing NiMMiT and data-driven notations for describing multimodal interaction,” Task Models and Diagrams for Users Interface Design, 2007, pp. 217–229. M. Gutiérrez, D. Thalmann, and F. Vexo, “Semantic Virtual Environments with Adaptive Multimodal Interfaces” Multimedia Modelling Conference, 2005, p. 277-283. D.E. Knuth, “Backus normal form vs. backus naur form,” Communications of the ACM, vol. 7, 1964, pp. 735–736. A. Maciel, T. Halic, Z. Lu, L.P. Nedel, and S. De, “Using the PhysX engine for physics-based virtual surgery with force feedback,” The International Journal of Medical Robotics and Computer Assisted Surgery, vol. 5, 2009. G. Sankaranarayanan, J. D. Adair, T. Halic, M. A. Gromski, Z. Lu, W. Ahn, D. B. Jones, and S. De, “Validation of a Novel Laparoscopic Adjustable Gastric Band Simulator,” Surgical Endoscopy, 2010.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-213

213

A Software Framework for Multimodal Interactive Simulations (SoFMIS) Tansel HALICa, Sreekanth A. VENKATAa, Ganesh SANKARANARAYANANa, Zhonghua LUb, Woojin AHNa, Suvranu DEa,1 a Department of Mechanical, Aerospace and Nuclear Engineering, Rensselaer Polytechnic Institute b Intelligent Manufacturing and Control Institution, Wuhan University of Technology

Abstract. The development of a multimodal interactive simulation is a very elaborate task due to the various complex software components involved, which run simultaneously at very high rates with maximum CPU load. In this work, we propose a multimodal parallel simulation framework called SoFMIS to create rapid interactive simulations such as surgical simulations. Our framework offers great flexibility and customization allowing simulation developers and researchers to concentrate on the simulation logic rather than component development. Keywords. Real-time Simulation, Interactive Simulation Framework, Haptics, Physics Simulation.

Introduction Highly demanding features of physics-based interactive simulations, such as plausible and robust physics simulations, high quality visual rendering, collision detection and multimodal sensation as haptic feedback, constitute very challenging tasks in software development. Furthermore, the interactivity and the support of multimodality impose performance constraints on each individual component as well as the overall system. In addition to performance constraints, the components need to communicate (e,g, one component output is required for other component input for execution) in order to achieve multimodality and seamless interaction. All these difficulties require particular foci from different disciplines. In surgical simulation development, crucial components such as real-time tissue simulation, tool-tissue interaction, and realistic rendering could be similar in two different surgical simulations, given usually developers and researcher groups write their own software from scratch and then customize it. Their initial efforts normally become redundant due to wasting time solving software problems or implementing and designing a physics simulation that is already addressed by some other research group. However, a framework with the support of multimodalities and built-in components can drastically minimize the development time of a surgery. This allows more time to be invested on the actual challenges of the simulation scenario rather than in the development of the necessary components. 1

Corresponding Author: Dr. Suvranu De, Department of Mechanical, Aerospace and Nuclear Engineering, Rensselaer Polytechnic Institute, Email: [email protected]

214

T. Halic et al. / A Software Framework for Multimodal Interactive Simulations (SoFMIS)

Several research efforts have been carried out to build a software framework for rapid development of surgical simulation. SOFA [1] is a highly active group which provides numerous components such as deformable bodies, collision detection, and fluid dynamics. Their architecture uses separate descriptions of objects, e.g. as physics models, rendering models, collision models, etc. These objects are stored in the scene graph for both simulation and rendering. Another framework, GiPSi [2], includes mass spring models, Finite Element Methods (FEM) along with a collection of time integrators and explicit solvers that use BLAS and LAPACK functions. Similarly, SPRING [2] is a framework for creating collaborative environments for real time surgical simulation. Its library supports multiple input devices through developed sensor abstraction which is intended to read the data from underlying hardware. Open tissue [3] is an open source toolkit that is composed of a collection of libraries varying from collision detection, matrix libraries, medical imaging and scientific visualization to physics simulation, while Tuchsmid et al.[4] support an object oriented modular simulator framework consisting of soft tissue deformation library, graphics engine, audio for effects, etc. Likewise, Chai3D [5] is a framework that integrates both haptic and visual rendering at the same time and supports various haptic devices with a unified API. There are also other frameworks that target game physics such as ODE [6], PhysX [7], Bullet [8], and Havok [9] that can be adopted for surgical simulations [10],[11].

1. Methods and Materials One of the essential motivations of the SoFMIS is to allow users to utilize only the components that they need through a modular and extendible framework. The SoFMIS design allows plugging or removing functional modules by its modular structure. Moreover, the components can be easily extended by custom implementation. Therefore, SoFMIS can be described as a module oriented framework. The entities in our framework could be separated into three major groups: objects, modules and interfaces. In general, the objects in the framework only store the information about each object’s current state. For instance, the rigid body or FEM (Finite Element Method) is an object that indicates its classification, encapsulates every detail that FEM requires such as material property and geometry boundary information, but not its functionality. A scene is an abstraction which includes the objects and attached objects iterators in the environment. The modules represent main functionalities of the framework. For instance, the simulation engine is a module that coordinates the physics processes in the scene. Similarly, the viewer is dedicated to all the rendering tasks of the scenes. Both modules and objects can be also regarded as conceptual definitions, even though they have implementation correspondence. The general outline of the framework and modules can be seen in the Figure 1. The modularity of the SoFMIS is also achieved through the loose connection between the modules. In order to assure component independency, an event mechanism is adopted into our design. It simply takes the responsibility for communication between modules. The events and event manager handles both content and context events. Moreover, it allows custom events to be generated by the developers. The communication backbone is entirely managed by the event manager module.

T. Halic et al. / A Software Framework for Multimodal Interactive Simulations (SoFMIS)

215

Figure 1. SoFMIS Overall Architecture.

The events are categorized into two groups: context and content based. For example, the events triggered by the haptic device for updated positions are considered to be content-based whereas the bleeding of a tissue in the surgical simulation is regarded as a context-based event. Context-based events are dealt with easily through our simulation language called SoFMIS Meta Language (SML). This language simplifies the development of simulations and also allows semantic embedding during the simulation development. The SoFMIS architecture also allows the use of other physics engines such as PhysX and Bullet in addition to built-in physics modules. The physics of the external library is scheduled by our simulation module. Thus, the threading and synchronization are handled by the SoFMIS framework without any additional programming effort by the developer.

2. SoFMIS Modules SoFMIS has numerous components to create interactive simulations, such as rendering, simulation components, and profiler components etc. The viewer module is responsible of rendering the objects in the scenes. In each rendering frame, the viewer module traverses the scenes and, based on the type; it calls the appropriate rendering function for its corresponding objects. The viewer module handles a full range of functions, from low-level to higher level such as basic windowing functions, stereo viewing and shadow mapping. Shader support is provided in the framework for flexible and highquality realistic rendering. Similarly, a texture manager is available for texture loading, storing or other operations over images like several filters. The simulator module is the master process of the particular simulators [12]. We support multiple simulators such as rigid body dynamics, finite element, and position based dynamics [13]. The simulator module parallelizes and controls the simulation through its thread pool. The simulator scheduler is controlled by our built-in profiler module that tracks the simulation performance and tunes the thread priorities and the size of the thread pool. There are also other components in the SoFMIS, such as a collision detection and contact response module which computes and resolves the collision of scene objects. There is a device interface manipulation module as well, which supports various

216

T. Halic et al. / A Software Framework for Multimodal Interactive Simulations (SoFMIS)

hardware, and a visual effects module that allows low-cost non-physics based simulation. In addition, framework provides a scripting engine for real-time configuration of the scene and an authoring module that designs the simulation environment and manipulates scene files, importing and exporting.

(a)

(b)

Figure 2. SoFMIS Module for (a) Smoke and (b) Bleeding employed in LAGB simulation.

3. Results We successfully employed our SoFMIS effects module in a Laparoscopic Adjustable Gastric Band (LAGB) simulator [14] . This module is responsible for realistic effects with low computation cost. In our simulator, it was used for realistic smoke generation and bleeding during an eloctrocautery procedure. This module is a GPU based nonphysical approach that aims to remove the computational burden from CPU through SoFMIS shaders. The snapshots of the effects can be seen in Figure 2.

4. Conclusions We presented the overall design of our SoFMIS framework. Our framework provides a parallel architecture with various components for interactive simulations, particularly surgical ones. Its flexibility and easy-to-use design makes it an effective software platform for surgical simulation development. Furthermore, its modular structure allows adopting the framework components in existing simulators. Our LAGB simulator validation studies demonstrate the competency of this framework.

Acknowledgements The authors gratefully acknowledge the support of this work by the NIH/NIBIB through grant # R01EB005807.

T. Halic et al. / A Software Framework for Multimodal Interactive Simulations (SoFMIS)

217

References [1] J. Allard, S. Cotin, F. Faure, P.J. Bensoussan, F. Poyer, C. Duriez, H. Delingette, and L. Grisoni, “Sofaan open source framework for medical simulation,” Studies in health technology and informatics, vol. 125, 2006, p. 13. [2] M.C. Cavusoglu, T.G. G\öktekin, F. Tendick, and S. Sastry, “GiPSi: An open source/open architecture software development framework for surgical simulation,” Medicine meets virtual reality 12: building a better you: the next tools for medical education, diagnosis, and care, 2004, p. 46. [3] K. Erleben, J. Sporring, and H. Dohlmann, “OpenTissue-An Open Source Toolkit for Physics-Based Animation,” 2005. [4] S. Tuchschmid, M. Grassi, D. Bachofen, P. Fr\üh, M. Thaler, G. Szekely, and M. Harders, “A flexible framework for highly-modular surgical simulation systems,” Biomedical Simulation, 2006, pp. 84–92. [5] F. Conti, F. Barbagli, D. Morris, and C. Sewell, “CHAI: An open-source library for the rapid development of haptic scenes,” IEEE World Haptics, 2005. [6] “Open Dynamics Engine - home.”, http://www.ode.org/ [7] Physx, “NVIDIA PhysX Physics Simulation for Developers.”, http://developer.nvidia.com/object/physx.html [8] Bullet, “Game Physics Simulation.”, http://bulletphysics.org/wordpress/ [9] “Havok - Home.”, www.Havok.com [10] A. Maciel, T. Halic, Z. Lu, L.P. Nedel, and S. De, “Using the PhysX engine for physics-based virtual surgery with force feedback,” The International Journal of Medical Robotics and Computer Assisted Surgery, vol. 5, 2009. [11] Z. Lu, G. Sankaranarayanan, D. Deo,, D. Chen,, and S. De, “Towards Physics-based Interactive Simulation of Electrocautery Procedures using PhysX,” Boston: IEEE Haptics symposium, 2010. [12] J. Andrews, Designing the Framework of a Parallel Game Engine, February, 2009. [13] M. M\üller, B. Heidelberger, M. Hennix, and J. Ratcliff, “Position based dynamics,” Journal of Visual Communication and Image Representation, vol. 18, 2007, pp. 109–118. [14] G. Sankaranarayanan, J. D. Adair, T. Halic, M. A. Gromski, Z. Lu, W. Ahn, D. B. Jones, and S. De, “Validation of a Novel Laparoscopic Adjustable Gastric Band Simulator,” will appear in Surgical Endoscopy, 2010.

218

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-218

Simulation of Vaginal Wall Biomechanical Properties from Pelvic Floor Closure Forces Map Shin HASEGAWA a,1, Yuki YOSHIDA a, Daming WEI a, Sadao OMATA b and Christos E. CONSTANTINOU c a Biomedical Information Technology Lab., University of Aizu, Japan b Nihon University College of Engineering World Research Center for Advanced Engineering & Technology, Japan c VA Medical Center (SCI) & Urology, School of Medicine, Stanford University, CA

Abstract. We simulated the way that pelvic floor muscles (PFM) generate zonal compression on the vagina and urethra in order to maintain urinary continence. Raw data were obtained using a probe to map the distribution of vaginal closure forces. Simulation model was made using ordinary Spring-mass model. The biomechanical properties are applied to the spring of the model. We simulated four models that are applied to asymptomatic subjects as controls and patients based on information obtained from the measured force maps using a vaginal probe. PFM values are measured when subjects are relaxed and during voluntary PFM contraction. Results show that simulation clearly distinguished between controls and patents and demonstrates that in the controls, after a period of 0.075 sec from the time when the rest force was added, the model was deformed to a neutral shape, and after another period of 0.075 sec from the time when the contract force was added at intervals of 0.001 sec, the closure force reaches maximum. The results render the simulation of the vaginal wall deformations that was obtained directly by the force maps. It shows that in controls the wall model is significantly deformed compared to that from the patient's model. In this research we simulated the response of the vaginal walls using spring mass model and the force maps of vaginal closure forces applied to control subjects and patients. The process of deformation of the vaginal wall is thus visualized demonstrating the relative pathologic differences between the two groups. Keywords. Vaginal Wall Simulation, Pelvic Floor Muscle, SUI

Introduction An important component of the mechanism of continence is attributed to the activation of the PFM, generating zonal compression of closure pressures on the urethra [1] and vagina [2]. Simulation of the distribution of the forces involved can lead to a better clinical understanding of the relative contribution generated by each individual muscle component along the length of the urethra. To develop a realistic simulation model it is essential to include, in addition to the anatomical configuration, the influence of the 1

Corresponding Author: Biomedical Information Technology Lab., University of Aizu, AizuWakamatsu, Fukushima-ken, 965-8580, Japan; E-mail:[email protected]

S. Hasegawa et al. / Simulation of Vaginal Wall Biomechanical Properties

219

elastic constituents [3] of the relevant tissue characteristics. In this research, we simulate the activity generated along the length of the vagina using a new measuring system [4]. Data were obtained from continent and incontinent subjects [5]. Visualizations of the comparative restitution of the shape of the vaginal wall from voluntary PFM contractions were constructed to illustrate the closure mechanism.

1. Tools and Methods Data from a total of 22 asymptomatic controls and 9 patients were obtained using the methods described in [4]. We developed simulation models using control data from asymptomatic normal subjects and stress urinary incontinent (SUI) patients. These simulation models were constructed assuming vaginal dimensions of 6x3.5x8 cm, Figure 1a, as defined by individual measurements of the force maps given by Figure 2. Figure 2 (a) and (b) shows the force map of the rest and construction of normal subjects. Figure 2 (c) and (d) shows the force map of the rest and construction of SUI patients. Each element is considered to be a structural spring and shear spring, assumed to be a non linear spring which has suitable biomechanical properties. The biomechanical properties are applied with reference to the spring constant [6]. To generate stress values, the parameters of the biomechanical properties are approximated using the least square method (Figure 1b). The points in figure 1 b shows the biomechanical properties of each stress which reported by Rubod et al [6]. The line shows the approximated value using the least square method when applied to the spring of our simulation model. The force maps of vaginal closure and depth are set to be the same value as those the measured force map pressures applied to the model. Calculations were made at the resting level force which was added to the first 0.075 sec, and the contraction force was added in another period of 0.075 sec at intervals of 0.001 sec. The bar shows the displacement length from the shape of the model; added rest force 0.075 sec. Solution of the forces applied to the each element of the model is done using the Euler methods and updated for each element position.

Figure 1. a) The vaginal wall simulation model. b) Biomechanical properties approximated by least-square.

220

S. Hasegawa et al. / Simulation of Vaginal Wall Biomechanical Properties

2. Results We simulated two models that are applied to controls, based on the measured force maps. The PFM values in both of cases are given when contracted and during rest are illustrated by the force maps given by Figure 2. Figure 3 shows the stop motion of the controls simulation starting after 0.075 sec applied the force map figure 2 (a) to another period of 0.75 sec applying the force map figure 2 (b). Figure 4 shows patients simulation applying Figure 2 (c) and (d). Figure 5 shows the results of the simulations. First row shows controls and second shows SUI patients. First column shows the difference map derived from the shape after 0.075sec added rest force. Second and third column show the anterior and posterior view of the simulated model. Fourth column shows the opening vagina view. The simulation given by figure 3 and 4, demonstrate the period of 0.075 sec at intervals of 0.01 sec that is added to the contraction force after a period of 0.075 sec from the time when the rest force added. As shown by comparing the simulations given by figures 3 and 4, the wall surface of the controls subjects demonstrate a more undulating shape than in SUI patients. Results shows the vaginal wall of asymptomatic controls is more readily subjected to deformation and also restitute back to original shape more quickly compared to patients. Taken together the results show that in SUI the vaginal wall demonstrates distinct biomechanical properties.

Figure 2. PFM force maps: a) Rest. Normal subjects. b) Contraction. Normal subjects. c) Rest. SUI patients. d) Contraction. SUI patients.

Figure 3. Control subjects: Stop motions of the simulation adding contraction force. a) Anterior view. b) Posterior view. c) Vaginal opening view.

S. Hasegawa et al. / Simulation of Vaginal Wall Biomechanical Properties

221

Figure 4. SUI patients stop motions of the simulation adding contraction force. a) Anterior view. b) Posterior view. c) View of vaginal opening.

Figure 5. The results of the simulation. First row shows controls and second shows patients. First column shows the difference map from the shape that after 0.075sec added rest force. Second and third column show the anterior and posterior view of the simulated model. Fourth column shows a view of the opening of the vagina.

3. Discussion and Conclusion In this research we simulated the response of the vaginal walls to compression by activating the PFM using the spring mass model and force maps of asymptomatic control subjects and SUI patients. The process of deformation of the vaginal wall is thus visualized demonstrating the relative pathologic differences between the controls and SUI groups. These simulations indicate the influence of the elastic constituents of these tissues. Clearly information regarding the constitutive properties of these tissues would render

222

S. Hasegawa et al. / Simulation of Vaginal Wall Biomechanical Properties

the simulation model accurate and may further explain the extent to which the variations observed in these simulations represent the physical character of the anatomy evaluated. Furthermore, the extend to which the vagina fits around the probe can be evaluated by implementing the collision detection model. Clearly computational simulations of the pelvic floor are needed to better understand the cause of many of the clinical disorders encountered and it is important to combine the anatomical information from image analyses [5] with unique in vivo force profiles for different relevant loading scenarios [2] to the in vitro acquired anisotropic material property values [3,7]. Although we expect the pelvic muscle to have different material property values compared to other muscles, the characteristic features of the underlying material model, having large deformations, incompressibility, non-linear anisotropic passive elasticity, permanent inelastic deformation upon nonphysiological overstretch, and superposed anisotropic active muscle contraction, need to be considered from a biomechanical point of view. Future analysis using specifically written finite element approach will require the complex interplay between active and passive muscle forces and the long-term adaptation in response changes, for example caused by surgical removal of tissue or implantation of tissue graft. Furthermore the incorporation of the urethra and sphincter should be included since the urethra is relevant when leakage takes place in subjects with SUI. What is ultimately needed is to integrate geometry, forces, and material property values, to provide a unique virtual test-bed for probing different treatment strategies for pelvic floor disorders and for optimizing surgical process parameters before testing them in humans. In such a model, simulations can be carried out incorporating the influence of posture as well as active forces produced reflexly and voluntarily. The transmission characteristics of closure forces to the bladder and urethra can be a significant contributory factor in the prevention of SUI [8]. Finally the outcome of virtual operating procedures using different materials can be simulated and their suitability considered within an objective framework [9]. To achieve this goal it is essential to also incorporate the remainder of the anatomical structures within the pelvic cavity within the context of finite element analysis protocols [10]. While the simulations presented were based on measurements done using data obtained considered within the focus of SUI it is appropriate however to indicate that the scope of these simulations can also be extended to clinical conditions related to other functions of the pelvic floor such as vaginal delivery[10]. A simulation integrating the mechanical properties of the tissues involved as evaluated using a probe, with the dynamic aspects relating the reflex activity generated using image analysis would be the ultimate goal [12]. In this way parameters such as velocity, acceleration and trajectory of displacement of the PFM can me modeled and functionally characterized and incorporated into the model. Such approach can aid experimentation and validation, of novel methodologies to be tested in detail and analyzed, both in terms of their inherent physiological aspects and also in terms of the resulting computational simulations of the organs contained in the human pelvic cavity, of the female pelvic floor.

Acknowledgments: Study supported by NIH grant R01 EB006170 to CEC.

S. Hasegawa et al. / Simulation of Vaginal Wall Biomechanical Properties

223

References [1] C. E. Constantinou, Principles and methods of clinical urodynamic investigations, Critical Reviews In Biomedical Engineering 7(1982), 229-264 [2] K. Shishido, Q. Peng, R. Jones, S. Omata, C. E. Constantinou, Influence of Pelvic Floor Muscle Contraction on the Profile of Vaginal Closure Pressures of Continent and Stress Urinary Incontinent Women, The Journal of Urology 179(2008), 1917—1922. [3] B. Chen, Y. Wen, X. Yu, and M. L. Polan, The Role of Neutrophil Elastase in Elastin Metabolism of Pelvic Tissues from Women with Stress Urinary Incontinence, Neurourology and Urodynamics 26(2007), 274—279. [4] C. E. Constantinou, and S. Omata, Direction sensitive sensor probe for the evaluation of voluntary and reflex pelvic floor contractions. Neurourology and Urodynamics 26(2007), 386—91. [5] C. E. Constantinou, Dynamics of the Female Pelvic Floor, International Journal Computational Vision and Biomechanics 1(2007):69—81. [6] C. Rubod, M. Boukerrou, M. Brieu, P. Dubois, and M. Cosson, Biomechanical Properties of Vaginal Tissue. Part1: New Experimental Protocol, The Journal of Urology 178(2007), 320—325. [7] Y. Murayama, S. Omata, T. Yajima, Q. Peng, K. Shishido, D. M. Peehl, and C. E. Constantinou, High Resolution Regional Elasticity Mapping of the Human Prostate, IEEE Engineering Medical Biology (2007), 5802-5805. [8] C. E. Constantinou, D. E. Govan, Spatial distribution and timing of transmitted and reflexly generated urethral pressures in the healthy female, The Journal of Urology 127(1982):964—969 [9] J.S. Afonso, P. A. L. S. Martins, M. J. B. C. Girao, R M. Natal Jorge, A. J. M. Ferreira, T. Mascarenhas, A. A. Fernandes, J. Bernardes, E. C. Baracat, G. Rodrigues de Lima, B. Patricio, Mechanical properties of polypropylene mesh used in pelvic floor repair, International Urogynecology Journal 19(2008), 375—380. [10] D. D’Aulignac, J. A. C. Martins, E. B. Pires, T. Mascarenhas, R. M. Natal Jorge, A shell finite element model of the pelvic floor muscles, Computer Methods in Biomechanics and Biomedical Engineering 8(2005), 339—347. [11] M. P. L. Parente, R. M. Natal Jorge, T. Mascarenhas, A. A. Fernandes, J. A. C. Martins, Deformation of the pelvic floor muscles during a vaginal delivery, International Urogynecology Journal 19(2008), 65—71. [12] Q. Peng, R. Jones, K. Shishido, C. E. Constantinou, Ultrasound Evaluation of Dynamic Responses of Female Pelvic Floor Muscles. Ultrasound Medical Biology 33(2007), 342-352.

224

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-224

A Generalized Haptic Feedback Approach for Arbitrarily Shaped Objects Rui HU, Kenneth E. BARNER and Karl V. STEINER Department of Electrical and Computer Engineering, University of Delaware, USA

Abstract. In surgery procedures, haptic interaction provides surgeons with indispensable information to accurately locate the surgery target. This is especially critical when visual feedback cannot provide sufficient information and tactile interrogation, such as palpating some region of tissue, is required to locate a specific underlying tumor. However, in most current surgery simulators, the haptic interaction model is usually simplified into a contact sphere or rod model, leaving arbitrarily shaped intersection haptic feedback between target tissue and surgery instrument less unreliable. In this paper, a novel haptic feedback algorithm is introduced for generating the feedback forces in surgery simulations. The proposed algorithm initially employs three Layered Depth Images (LDI) to sample the 3D objects in X, Y and Z directions. A secondary analysis scans through two sampled meshes and detects their penetration volume. Based on the principle that interaction force should minimize the penetration volume, the haptic feedback force is derived directly. Additionally, a post-processing technique is developed to render distinct physical tissue properties across different interaction areas. The proposed approach does not require any pre-processing and is applicable for both rigid and deformable objects. Keywords. Haptic Feedback, Surgery Simulation, Penetration Volume, Layered Depth Images.

Introduction For many years, haptic rendering has garnered great attention in multiple fields, including surgery simulation, virtual reality and computer games. Haptic feedback in virtual reality is coupled with visual information to improve the simulation environment realism. In surgery simulation applications, haptic rendering can serve to provide surgeons with useful properties, such as tissue stiffness information, so that the user will better understand the surgery situation through touching and palpation. For example: recently in [1], Doyle et al. explored the possibility of adding an artificial haptic response to Phacoemulsification cataract surgery simulation to enhance training effectiveness. However, the complex geometry shape of 3D objects, especially deformable surgery objects, makes haptic rendering a computation-intensive task. In early research, to alleviate computational complexity, researchers simplified the surgery instrument into a point. Salisbury et al. [2] successfully achieved real-time point-based haptic feedback with polygon models. In their organ exclusion simulation,

R. Hu et al. / A Generalized Haptic Feedback Approach for Arbitrarily Shaped Objects

225

Kuroda et al. [5] track fingertips as points and generate the haptic force for each point using a pre-computed stiffness matrix based on the finite element method (FEM). As a comparison, the Rensselaer Polytechnic Institute group members, such as Suvranu De and Yi-Je Lim focus on using mesh-free methods in many applications [3]. Beyond the point-based haptic model, a sphere-based model is also employed in surgery simulation due to its simplicity. In [4], Trier et al. model the tip of a drilling instrument as a sphere. For the arbitrarily shaped haptic model interaction case, researchers typically employ methods based on the penetration depth between two triangle meshes [6]. In [7], Fisher et al. consider calculating the penetration volume as “The most complicated yet accurate method” to describe the extent of intersection. Following this idea, Weller et al. [8] proposes a new data structure, the “Inner Sphere Tree” to estimate the penetration volume. However, this approach is not applicable for deformable objects. In [9], Faure et al. apply Layered Depth Image (LDI) to calculate the penetration volume for the purpose of deformation simulation, but the LDI generation method employed was too slow for real-time haptic response. To achieve accurate haptic feedback for deformable objects, we accelerate the LDI generation method to efficiently compute the penetration volume. Three LDIs are generated for each new haptic frame. The proposed algorithm is well suited for soft, deformable objects. We resort to a post-processing technique to render tissue stiffness for organs with different materials at different locations, such as a relatively stiffer tumor tissue surrounded by healthy, soft body tissue.

1. Methods 1.1. Haptic Force Generation 1.1.1. Layered Depth Image and Penetration Volume Generation. Layered Depth Image (LDI) was first introduced as a rendering technique [11]. The LDI generation procedure is essentially a sampling process operating on the triangle meshes through a user-defined viewport and resolution. Suppose we shoot a ray at the center of each pixel in the viewport and the ray direction is orthogonal to the view plane. Each time the ray intersects with a triangle mesh, a sequence of intersected fragment information is stored in the LDI. A generated LDI contains three types of information at all sampled fragments: (1) the depth values through the viewing direction, (2) the object this fragment belongs to, and (3) the object normal at this fragment. LDI greatly facilitates determining the penetration volume because, by sequentially scanning the stored information at a pixel, one can easily find the fragment pair where two objects penetrate. This is based on the fact that if we follow a ray, every fragment the ray encounters with a normal opposing the ray direction means that the ray is entering a mesh, while a normal aligned with the ray direction indicates the ray is exiting a mesh. By multiplying a specific penetration length with a pixel area dependent on resolution, we approximate the penetration volume at this pixel. Figure 1 illustrates this process of penetration volume generation. In Figure 1-(a), the bounding volume intersections of two objects are detected. The left column of the green box

226

R. Hu et al. / A Generalized Haptic Feedback Approach for Arbitrarily Shaped Objects

represents the view plane. The volume is rasterized and sampled as shown in Figure 1(b) and 1-(c). The generated LDI is shown in Figure 1-(d), were the arrow direction indicates the normal direction; the color represents the object that is sampled. We store the fragments and then pick up collision pairs in each pixel, as shown in Figure 1-(e). Finally, in Figure 1-(f), the length of the penetration section is calculated at each pixel.

Figure 1. An illustration of calculating intersection volume from a generated Layered Depth Image (LDI)

1.1.2. Efficient LDI Calculation. Researchers have been investigating the LDI generation for years [12, 13]. Recently, Liu et al. [10] show that the Compute Unified Device Architecture (CUDA)-based LDI generation exhibits the best performance. In our implementation, we have adopted a similar algorithm for LDI generation. This is due to its excellent performance, and the fact that CUDA made parallel penetration length query and operations on penetration volume available on GPUs. 1.1.3. Volume Based Haptic Force ͳ

Similar to [9], we define the potential energy as: ൌ ʹ , where k is a user-defined ʹ parameter to scale this energy. V is the penetration volume. Suppose the calculated LDI is along the Z direction. Given the assumption that the haptic force serves to minimize the intersection volume of two objects, the interaction force in the viewing direction (Z direction) at a given coordinate ( ǡ ǡ ) is: ǡ ൌ െ

ቚሺ ǡ ሻ ൌ െ

ቚ ሺ ǡ ሻ ሺͳሻ

When taking force direction into consideration, i.e. if the haptic-related mesh is pushed to left, the haptic force should follow the negative X-axis direction. Also, if no haptic model is involved, no haptic force should be generated. The direction factor is ൌ െ , where ܱ‫ ܮ‬and ܱܴ are: Ȁ ൌ ቊ

ͳǡ ݈݂݁‫ݐ݄݃݅ݎݎ݋ݐ‬ ሺʹሻ Ͳǡ ݈݂݁‫ݐ݄݃݅ݎݎ݋ݐ‬Ǧ

The haptic force in Z direction is the integration of forces over penetration boundary:

R. Hu et al. / A Generalized Haptic Feedback Approach for Arbitrarily Shaped Objects

ൌ ‫ ׬ ׬‬ȁ

227

ȁ ሺ͵ሻ

If we write Equation (3) into a discrete form for calculation, with every unique X becomes 1. Therefore we have: and Y standing for a pixel on the viewport, the term

ൌ σ σ ൌ σ σ ሺͶሻ Equation (4) indicates that the final haptic feedback force in one direction is simply the sum of all penetration volume weighed by force direction. By repeating this two more times for the X and Y directions, we generate the haptic force: ‫ ܨ‬ൌ ሺ‫ ܺܨ‬ǡ ‫ ܻܨ‬ǡ ‫ ܼܨ‬ሻ, which can be sent to the haptic device directly or used for post-processing. 1.2. Haptic Force Post-Processing In order to render the distinct feedback forces at different locations, we apply following post-processing method called local stiffness vertex method to enhance the haptic force: 1. At simulation initiation, load a series of vertices into memory. Three pieces of information is stored for each vertex ݅: (1) The vertex number: ܰ݅ ; (2) The influence radius of this vertex: ܴ݅ ; and (3) The stiffness parameter of this vertex: ܵ݅ . 2. During simulation, keep monitoring the position of the instrument center ܲ‫ ܥ‬. 3. The distance between the instrument center and any stored vertex i, ‫ ݅ܿܦ‬is compared with the vertex radiusܴ݅ . 4. If any ‫ ݅ܿܦ‬is smaller than ܴ݅ , we use the following equation to scale the force:

ൌ σ ሺ

ʹ ሺ Ǥ ሻ

ሻሺͷሻ σ ሺ ሻ Ǥ

where ሺሻ is a monotonic continuous function, with ሺͲሻ ൌ ͳǡ ሺͳሻ ൌ ͳȀ . ሺሻ is an influence kernel describing how the stiffness parameter decreases from the center of the vertex to its surrounding area. We are finalizing improvements to the algorithm that dynamically add stiffness properties to penetration volumes. This algorithm will support large, arbitrary instruments.

2. Results The proposed algorithm has been implemented on a PC with an i7-930 CPU and a GeForce GTX 295 video card. Here we evaluate this algorithm under various scenarios. In [8], it is stated that the LDI-based volume calculation “is restricted to image space precision.” Consequently, it is important to evaluate how the algorithm is influenced by insufficient volume sampling, which is caused by low LDI resolution. The generated haptic feedback forces, based on different resolutions, are compared with a highresolution (512 ൈ 512) result. The normalized absolute error is calculated as:

ൌ ȁ െ ͷͳʹ ȁȀ ͷͳʹ ሺ͸ሻ

228

R. Hu et al. / A Generalized Haptic Feedback Approach for Arbitrarily Shaped Objects

where ‫ ݉ܨ‬is the m-resolution output force. ‫ܨ‬ͷͳʹ is the high-resolution force used as a benchmark. Three scenarios are adopted to test the algorithm. In each scenario, two objects are colliding. The objects used in each test environment are: (1) a tumor and a cylindrically-shaped object; (2) a tumor-shaped object and a duodenum-shaped object; and (3) a tumor-shaped object and a bunny-shaped object. The numbers of triangles of the aforementioned meshes are: tumor: 10,000; cylinder: 1,340; duodenum: 2,000; bunny: 70,000. The simulation scenes and the feedback force error are shown as below:

Figure 2. screen shots of three test scenarios

Figure 3. Tumor-cylinder scene error comparison

Figure 5. Tumor-bunny scene error comparison

Figure 4. Tumor-duodenum scene error comparison

Figure 6. Frame rate for five different image resolutions

From Figure 3-5, we can easily find that volume errors caused by low LDI resolution do have a negative impact on the haptic force. However, in most cases, the influence is less than 3%. Therefore, the lower resolution can also be used for real-time simulations. Another aspect critical in haptic simulation is haptic refreshment rate. To evaluate the computation time of proposed algorithm, we tested the haptic frame rate for the three scenarios. Figure 6 shows the frame rate comparison. Judging from the

R. Hu et al. / A Generalized Haptic Feedback Approach for Arbitrarily Shaped Objects

229

error and frame rate, we can conclude that, although adopting a resolution of 32 can yield a higher haptic response rate, the error may be relatively large. Using a resolution of 64 is more reliable and yields an acceptable frame rate of approximately 200 FPS. We use a scene describing a “cylinder sweeps across a cube” to demonstrate that by introducing local stiffness vertices, the post-processing method can lead to a varying feedback force effect. The scene is shown in Figure 7. In this scenario, when no local stiffness vertex is added, the haptic feedback force is constant, always along the positive Y axis (from “a” to the tip of the cylinder). To introduce different stiffness properties at individual regions, i.e. making the cube feel stiffer or softer at different regions, we consider two stiffness vertex placement scenarios: (1) One local stiffness vertex with a large influence range is placed at position “a” in Figure 7. Thus, the middle part of cube will generate additional force, making it feel stiffer. (2) Two local stiffness vertices with different influence ranges and stiffness values at locations “b” and “c” in Figure 7. In this scenario, the cube will feel stiffer at “b” and “c” while softer in between. In Figure 8, the feedback forces of both scenarios are compared. The kernel function adopted is a raised cosine function: ͳ

ሺሻ ൌ ሺͳ ൅ ሺሻሻ ሺ7ሻ ʹ

In Figure 8, the feedback force is recorded as the cylinder travels from point “b” to “c”. Monitored forces show the proposed algorithm successfully renders the desired stiffness properties. The output force is drastically converted from its original flat pattern into a curve. If one can carefully spread local stiffness vertices over the target tissue, realistic heterogeneous stiffness properties can be easily perceived.

Figure 7. Interaction scene with local stiffness vertices

Figure 8. Feedback force for different local stiffness vertices scenarios.

We have integrated the haptic feedback algorithm into our simulator with constantly deforming objects. Vertices with higher stiffness values are scattered around the deformable tissue representing the tumor area in an organ. Surgical residents can use multiple tools to touch and feel the location of these tumors for training exercises.

3. Conclusions The main contribution of this paper is an haptic feedback algorithm for deformable objects. Unlike past algorithms based on simplifying the haptic model, our method operates on the original volume so that the irregular objects interactions are accurately

230

R. Hu et al. / A Generalized Haptic Feedback Approach for Arbitrarily Shaped Objects

presented and rendered. Thus, the algorithm can be widely applied in open surgery simulations for different instruments and arbitrary anatomy. Future work in this algorithm will include reinforcing the post-processing method for rendering different physical properties. We will work closely with our surgical collaborators at Thomas Jefferson University to address these issues.

Acknowledgements The authors gratefully acknowledge the partial support by the U.S. Army Telemedicine & Advanced Technology Research Center (TATRC) under grant number W81XWH09-1-0577, Eric Wickstrom-Thomas Jefferson University, PI, and by the NIH-NCRRfunded Delaware INBRE program, grant number 2 P20 RR016472-09, David S. Weir University of Delaware, PI.

References [1] [2] [3]

[4] [5]

[6]

[7]

[8]

[9]

[10]

[11] [12] [13]

Doyle, L., Gauthier, N., Ramanathan, S., Okamura, A., A Simulator to Explore the Role of Haptic Feedback in Cataract Surgery Training, Medicine Meets Virtual Reality 16, 106-111, 2008. Salisbury, Z., Zilles, B., Salisbury, K., A constraint based god-object method for haptic display, Proceedings of the IEEE Conference on Intelligent Robots and Systems, Vol. 3, 146-153, 1995. Lim, Y-J., Wei, J., De, S., On some recent advances in multimodal surgery simulation: a hybrid approach to surgical cutting and the use of video images for enhanced realism, Presence, Vol. 16, No 6, 563-583, 2007. Trier, P., Noe, K., Sørensen, M., and Mosegaard, J., The Visible Ear Surgery Simulator, Medicine Meets Virtual Reality 16, 523-525, 2009. Kuroda, Y., Hirai, M., Nakao, M., Sato, T., Kuroda, T., Nagase, K., Yoshihara, H., Organ exclusion simulation with multi-finger haptic interaction for open surgery simulator, Medicine Meets Virtual Reality 15, 244-249, 2008. Kim, Y., Otaduy, M., Lin, M. and Manocha, D., Six Degree-of Freedom Haptic Display Using Localized Contact Computations, Symposium on Haptic Interfaces For Virtual Environment and Teleoperator Systems, 209-212, 2002. Fisher, S., and Lin, M., Fast penetration depth estimation for elastic bodies using deformed distance fields, Proceedings of International Conf. on Intelligent Robots and Systems, Vol. 1, 330336, 2001. Weller, R., Zachmann, G., A Unified Approach for Physically-Based Simulations and Haptic Rendering, ACM Siggraph Video Game Symposium, Proceedings of the 2009 ACM SIGGRAPH Symposium on Video Games, 151-159, 2009. Faure, F., Barbier, S., Allard, J., and Falipou, F., Image-based collision detection and response between arbitrary volumetric objects, Proceedings of the 2009 ACM SIGGRAPH Symposium on Video Games, 155-162, 2009. Liu, F., Huang, M., Liu, X., and Wu, E., Single pass depth peeling via CUDA rasterizer, International Conference on Computer Graphics and Interactive Techniques archive, SIGGRAPH 2009: talks, 2009. Shade, J., Gortler, S., He, L., and Szeliski, R., Layered depth images, Proceedings of the 25th annual conference on Computer graphics and interactive techniques, 231-242, 1998. Case, E., Interactive order-independent transparency. Tech. rep., NVIDIA Corporation, 2001. Bavoil, L., Myers, K., Order independent transparency with dual depth peeling. NVIDIA OpenGL SDK, 2008.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-231

231

Piezoelectric Driven Non-toxic Injector for Automated Cell Manipulation H. B. HUANG a,1 , Hao SU b , H. Y. CHEN c and J. K. MILLS a Dept. of Mechanical and Industrial Engineering, University of Toronto b Dept. of Mechanical Engineering, Worcester Polytechnic Institute c Dept. of Automation and Mechatronics, Harbin Institute of Technology Shenzhen Graduate School a

Abstract. Stimulated by state-of-the-art robotic and computer technology, Intra Cytoplasmic Sperm Injection (ICSI) automation aims to scale and seamlessly transfer the human hand movements into more precise and fast movements of the micro manipulator. Piezo-drill cell injection, a novel technique using piezo-driven pipettes with a very small mercury column, has significantly improves the survival rates of ICSI process. It is found that complications are due, in large part, to toxicity of mercury and the damage to the cell membrane because of the lateral tip oscillations of injector pipette. In this paper, a new design of piezo-driven cell injector is proposed for automated suspended cell injection. This new piezo-driven cell injector design centralizes the piezo oscillation power on the injector pipette which eliminates the vibration effect on other parts of the micromanipulator. Detrimental lateral tip oscillations of the injector pipette are attenuated to a desirable level even without the help of mercury column. This mercury-free injector can sublime the piezoelectric driven injection technique to completely non-toxic level with great research and commercial application in gene injection, in-vitro fertilization, ICSI and drug development. Keywords. Biomanipulation, piezoelectric driven injector, cell microinjection

Introduction Due to the importance of biological cell injection technology, significant research has been carried out to automate laborious cell injection tasks. Last decade has witnessed the dedicated research effort on cell injection automation from a diverse array of aspects: cell holding devices, cell injection force control, visual serving, cell injection process control, etc [1-2]. Recently, a promising cell piercing technology called piezodrill was proposed. Kimura [3] originally presented the piezo-driven ICSI procedure and a high survival rate of sperm-injected mouse oocytes that reaches 80% can be obtained. Researches in the field also discovered that a very small mercury column in the injector pipette can significantly improve the success rate of ICSI [4]. However, the use of mercury in laboratory has detrimentally potential toxicity effects. Researchers have been investigating the influence of mercury column and lateral pipette oscillation on the zona piercing mechanism in ICSI. Experiments conducted by Ediz [6] and the simulation by Gan [7] support that the presence of mercury generally 1

Corresponding Author: H. B. Huang, Dept. of Mechanical and Industrial Engineering, University of Toronto, Toronto, Ontario, Canada M5S 3G8; E-mail: [email protected], [email protected].

232

H.B. Huang et al. / Piezoelectric Driven Non-Toxic Injector for Automated Cell Manipulation

reduces the amplitude of the pipette tip oscillation. As for the zona piercing mechanism in piezo-ICSI, Kimura [3] and Fan [5] stated that the axial pipette motion facilitates the piercing of the mouse zona pellucida. However, Ediz [8] found that the pipette has a much larger lateral oscillation comparing with axial oscillation and argued that the lateral oscillation may collaterally damage the zona piercing process. While there are different opinions on these issues, it is a consensus that complications are due, in large part, to toxicity of mercury and the damage to the cell membrane because of the lateral tip oscillations of injector pipette. In this paper, a piezo-driven ultrasonic cell injection technique is introduced as a universal cell injection method. To address the aforementioned issues, the contributions of this paper are: (1) a novel design of piezo-driven cell injector which centralizes the piezo oscillation power on the injector pipette through a piezo stack located near the injector pipette. Significantly, this eliminates the vibration effect on other parts of the micromanipulator. Hence, only a small piezo stack is required to actuate the pipette tip and perform the piezo-drill cell injection process and (2) high amplitude oscillation of the injector pipette due to the system noise is substantially reduced without a mercury column. From cell injection experiments that were performed on zebrafish embryos, under reasonable driven frequency and amplitude, the injector pipette easily pierces through the cell membrane with much lower injection speed and with virtually no deformation of the cell membrane. This novel technology approach has demonstrated significant potential in high-precision cell injection with minimum damage to injected cells comparing with previous results.

Figure 1. Test-bed for the suspended cell injection system.

Figure 2. Conventional commercial piezo-drill system [9].

1. Methods & Materials 1.1. Cell Injection System Setup Figure 1 illustrates an automatic suspended cell injection system developed in our laboratory for this research. Based on the previous research [2], this newly-designed system not only considers the characteristic of the biological experiment undertaken, to make the operation more convenient, but also adds new functions such as microscope autofocus. This system is designed to simulate automatic cell injection of large batches of suspended cells (such as fish embryos) in biological engineering processes. To achieve this purpose, the system is designed for use with cells arrays instead of holding cells individually. A specially designed cell holding device was fixed on an actuated

H.B. Huang et al. / Piezoelectric Driven Non-Toxic Injector for Automated Cell Manipulation

233

rotary plate, permitting the cells to be held and transported one by one, into the field of view of the microscope for injection.

(c) Figure 3. A new design of piezo-driven cell injector. (a) Exploded drawing; (b) Assembly drawing; (c) The close sight of the piezo-driven cell injector.

1.2. Novel Piezo-driven Cell Injector Design The piezo-drill cell injector structure has remained the same since it was invented. A commercial piezo-driven cell injection system (PIEZODRILL, Burleigh) [9] is shown in Fig. 2(a). It consists of the injection pipette, the pipette holder, the holding clip, and the piezo actuator, as shown in Fig. 2(b). The ultrasonic mechanical pulses transfer longitudinally through pipette holder, the holding clip, the glass pipette and finally reach the pipette tip. The injection pipette, pipette holder, and the holding clip will vibrate during the cell injection process. The vibration of so many parts will unavoidably introduce unwanted mechanical vibration to the pipette tip. As discussed in the research of Ediz [8], when this piezo-drill pipette is operated in air, the largest . Even with the help of mercury lateral vibration of the pipette tip will reach 270 column and the pipette is immersed in mineral oil, the largest lateral vibration of the pipette tip still reaches 37 . And the use of mineral oil and mercury may lead to contamination of cells etc.. To eliminate unwanted lateral pipette vibration and centralize the piezo power on the injector pipette, a new design of this injector system was proposed, as shown in Fig. 3. Fig. 3(a) and (b) are respectively the exploded view and assembly view of machine parts. The piezo stack is assembled on the end of the piezo-driven cell injector and to minimize the load of piezo stack, the parts cannot be moved are made of low density material acrylic (to lower down the vibration mass). The injection pipette can be easily replaced and is fixed through the rubber ring. The soft plastic pressure tube provides for the injection of solutions into cells. Fig. 3(c) shows the close sight of the new piezodriven cell injector.

2. Results To verify the effectiveness of the proposed approach, experiments were performed using the cell injection system as shown in Fig. 1. The cells selected for our experiment target were Zebrafish embryos. The diameter of Zebrafish embryo is approximately

234

H.B. Huang et al. / Piezoelectric Driven Non-Toxic Injector for Automated Cell Manipulation

~1.2mm (including chorion) and ~600 (without chorion). The thickness of zebrafish embryo is 3 . The radius of the injector pipette is . The injected zebrafish embryos were all placed in our special designed cell holding device. It is known that too large lateral vibration will cause damage to cells in piezo-driven cell injection. Existing piezo-drill systems utilizes an adjustable given pulse voltage, to inspire the piezo actuator and drill the cells [5-8]. The three parameters: amplitude, frequency, and duration can be adjusted and trained by the user for particular exercise. But it is found in our research that the harmful lateral tip oscillation has a great relationship with the frequency of the inspiring signal. Then a single frequency sine wave voltage signal, which is always used in ultrasonic cutting, was used here for piezo-driven cell injection. A series of sine wave signals (frequency 0.5Hz~40KHz, amplitude 10V) were applied as the inspired signal of piezo actuator. The lateral oscillations of the pipette tip were observed and recorded by CCD camera. When the injector pipette is placed in exactly the same situation of cell injection (the pipette tip is immersed in the water), the frames in Fig.4 show clearly the lateral oscillations of the pipette under different frequency of inspired signal. When the input signal frequency is below 21 KHz, very slightly lateral vibration can be recognized. But between 22 KHz and 23 KHz, the lateral vibration amplitudes increase significantly and the amplitudes reach its maximum when the frequency of the input signal frequency is 22.5 KHz. At this time, the vibration frequency of the piezo actuator reaches the resonance frequency of the injector pipette and the ). When the amplitude of the lateral oscillation of pipette tip reaches its maximum (30 frequency improves higher than 23 KHz, the lateral vibration decreased back to slightly.

Figure 4. Lateral vibration of pipette under the environment of cell injection.

2.1. Cell Injection Experiment Experiments were performed to show the effectiveness of piezo-driven cell injection. In all injection experiments, although each test used a different embryo cell, the mechanical properties of all cell biomembranes are uniform. The trained parameter of piezo-driven cell injection of zebrafish embryo is 20.1 KHz and 10V. The injection . velocity is constant with 175 Two entire cell injection processes with piezo-driven cell injection and conventional cellular piercing technology were shown in Fig. 5 for comparison. It is clear to see that when we inject the zebrafish embryo with piezo-driven cell injector, the injector pipette contact the cell membrane and cutting through it with nearly no

H.B. Huang et al. / Piezoelectric Driven Non-Toxic Injector for Automated Cell Manipulation

235

deformation, as shown in Fig. 5(b). But when conventional cellular piercing technology is used, the cell membrane is deformed by the injection force. When injection force reaches the broken threshold, the injector pipette pierces through the cell membrane and into the cells. It is clear to see a large deformation on cell membrane in Fig. 5 (a).

Figure 5. (a) Cell injection process under the conventional cellular piercing technology; (b) Cell injection process with the piezo-driven cell injection technology.

3. Conclusions In this paper, a novel piezo-driven cell injector was developed and validated in cell injection of zebrafish embryos. According to experiments of zebrafish embryos cell injection, this piezo-driven cell injector exhibits three advantages over the conventional cellular piercing technology. 1) It utilizes the ultrasonic cutting force while not the piercing force to penetrate the cell membrane. The injecting speed requirement is less stringent than the conventional ones thus resulting in better controllability in cell injection motion control. 2) With nearly no deformation of the cells during the cell injection period, cell damage is dramatically less. This technique could result in comparably high survival rate and success rate. 3) With the new piezo-driven cell injector, a small piezo stack is sufficient to perform the cell injection process. And with appropriate frequency and amplitude, the detrimental lateral tip oscillations of injector pipette are reduced to a satisfying level even without mercury column.

References [1] W.H. Wang, et al., A fully automated robotic system for microinjection of zebrafish embryos, PLoS ONE, Vol. 2, No. 9, e862. doi:10.1371/journal.pone.0000862, 2007. [2] H. B. Huang, et al., Robotics cell injection system with vision and force control: Towards automatic batch biomanipulation, IEEE Trans. on Robotics, vol. 25, no. 3 (2009), 727-737. [3] Y. Kimura, et al., Intracytoplasmic sperm injection in the mouse, Biol. Reprod., vol. 52, no. 4 (1995), 709–720. [4] Y. Kimura, et al., Analysis of mouse oocyte activation suggests the involvement of sperm perinuclear material, Biol. Reprod., vol. 58, no. 6 (1998), 1407–1415. [5] M. Fan, et al., Vibration study of piezodriven pipettes immersed in viscous liquids,” 100(7), 2006. [6] K. Ediz, et al., Effect of Mercury Columan on the Microdynamics of the Piezo-Driven Pipettes, J. of Biomechanical Engineering, vol. 127 (2005), 531-535. [7] Y. Gan, et al., A Study of the zona piercing process in piezodriven intracytoplasmic sperm injection, 104(4), 2008. [8] K. Ediz, et al., Microdynamics of the Piezo-Driven Pipette in ICSI, IEEE Trans. on Biomedial Engineering, vol. 51, no. 7 (2004), 1262-1268. [9] http://www.cadence-tech.com.sg/Burleigh/Piezo%20Drill.htm

236

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-236

Virtual Arthroscopy Trainer for Minimally Invasive Surgery Vassilios HURMUSIADISa, 1, Kawal RHODEb, Tobias SCHAEFFTERb, Kevin SHERMANc a Primal Pictures Ltd, London, UK b Division of Imaging Sciences, King’s College London c The Royal College of Surgeons of England

Abstract. The presented work introduces an innovative technology solution to a major challenge in minimally invasive surgical training. It is focused on the development of a low-cost, real-time simulation of arthroscopy accessible online via the web. The aim is to enable users to develop their cognitive skills and to comprehend the disorientating images seen through arthroscopes. The simulation is incorporated into a software training tool that provides virtual arthroscopy and e-learning and assessment. Keywords. Arthroscopy, surgical simulation, real-time interaction.

Introduction Arthroscopy is a minimally invasive procedure for the diagnosis and treatment of joint pathologies. Due to advancements in medical technology and the development of new surgical techniques, arthroscopy has become the most commonly performed minimally-invasive procedure with the advantages of fast execution and short patient recovery times. Virtual reality and augmented reality systems have been developed with the aim to create an image-assisted approach in the operating environment [1][2]. Research in this area also led to the development of software environments devoted to the training aspects of arthroscopy procedures [3]. Such systems provide accurate 3D models of the joint and interaction based on haptic and force feedback interfaces. An inherent problem in arthroscopy is the small field of view and the limited working space inside the joint, which does not allow prompt understanding of the targeted anatomy and pathology. Correct positioning and orientation of the arthroscopic camera and surgical tools in relation to anatomy is one of the biggest obstacles a novice trainee has to overcome. This is also a major cause for delay and error during the procedure, increasing patient risk. Trainees also need to understand the techniques for visualizing areas of the joint obscured by the hard joint surfaces by utilizing the combination of the viewing angle of the arthroscope and independent rotation of the arthroscope and camera to maximize the effective field of view. Further problems encountered by trainees include disorientation when rotation of the limb is required to access certain parts of the joint (such as the lateral compartment) and difficulty in

1

Corresponding Author

V. Hurmusiadis et al. / Virtual Arthroscopy Trainer for Minimally Invasive Surgery

237

recognizing the size of lesions due to the wide angle lenses. The skills and knowledge required to overcome many of these problems can be learned without requiring a simulator that incorporates haptic feedback. The presented work is focused on the development of a low-cost, real-time simulation of arthroscopy. The aim is to enable users to comprehend the often disorientating images seen through arthroscopes and to learn the cognitive aspects of arthroscopy. The project outcome is the first purely software solution for arthroscopy training. The initial module focuses on knee arthroscopy. The plan is to follow on with other joints, such as the shoulder, elbow, wrist, hip, ankle and the spine. The goal is to enable more surgeons to be trained more effectively, and at significantly lower cost.

Figure 1. View through simulated arthroscopic camera.

1. Methods The core system comprises of 3D anatomy models of the knee, simulated arthroscopic camera and surgical tool models. The detailed generic knee anatomy models are based on 3D reconstructions from the Visible Human data-sets. The virtual arthroscopic camera is based on optical simulation of a standard arthroscope from KARL STORZ GmbH (figure 1).

Figure 2. Detailed tools and arthroscopic camera models.

Surgical tool models were also based on common probe and hook tools used in diagnostic procedures (figure 2). Real-time interactive control allows the user to manipulate the arthroscope and tools with a standard mouse or game-pad. A “bird’s eye” view shows the actual arthroscope and surgical tool position and orientation in relation to related anatomical structures. The system automatically detects collisions of the arthroscope and tools with anatomical structures and is capable of distinguishing

238

V. Hurmusiadis et al. / Virtual Arthroscopy Trainer for Minimally Invasive Surgery

between collisions with hard and soft tissue. Each collision generates audio and motion feedback in order to alert the user of incorrect instrument placement. The system allows the knee to be flexed and extended, and for small amounts of varus and valgus deformity to be applied to the joint, to open up the appropriate joint spaces. An integrated e-learning and assessment model is based on a series of surgical training tutorials grounded in the standard Procedure Based Assessment (PBA) used in Trauma & Orthopaedic training in the UK, as described in the Orthopaedic Curriculum and Assessment Project (OCAP), developed by the British Orthopaedic Association and Orthopaedic Specialty Advisory Committee. Assessment is based on measurable criteria, such as: time to achieve correct access to a specific structure, number of attempts to achieve correct access, number of collisions with soft/hard structures, velocity of instrument tip during collision. The system also contains a number of case studies of pathologies with labeled real arthroscopy videos. A key feature of the system is that it can be used anywhere where there is access to the Internet, with no special equipment being required. The tutorials are modular in nature and incorporate automated feedback on performance, allowing progress to be evaluated.

2. Results Expert evaluation was carried out by orthopaedic surgeons at the Royal College of Surgeons of England (RCSE) and King’s College London. Training evaluation also took place at RCSE using a group of trainees using the system and a group of trainees using the Hillway Knee model (phantom plastic foam model). Questionnaire and feedback forms were used to assess knowledge, performance and confidence based on the standard Procedure Based Assessment (PBA) for orthopaedics. Initial feedback on the system has indicated that the ability to learn the cognitive aspects of the surgical skill is appreciated. The feedback also reinforces the need for a detailed and structured tutorial approach to guide the trainee through the early stages of learning. It is accepted that the surgical trainee will still need to progress through a stage of learning with haptic feedback (either on more expensive “VE”simulations, physical models or on real patients) in order to master the skill of triangulation and other practical aspects of the procedure, but mastering the cognitive aspects first should maximize the efficiency and cost effectiveness with which the other learning modalities are used. There should be a corresponding decrease in the unnecessary exposure of patients to trainees who are still in the process of learning the cognitive aspects of the operation.

References [1] J.D. Mabrey, S.D. Gillogly, J.R. Kasser, H.J. Sweeney, B.Zarins, M. Howard, W.E. Garrett, R. Poss, and D.W. Cannon, “Virtual reality simulation of arthroscopy of the knee”, Arthroscopy: The Journal of Arthroscopic and Related Surgery, 18(6), 2002. [2] S.P. Oakley et al., “Arthroscopy – a potential “gold standard” for the diagnosis of the chondropathy of early osteoarthritis” in AosteoArthritis and Cartilage 2005, 13, pp. 368-378. Elsevier Ltd ed. 2005. [3] L. Moody, A. Waterworth, A.D. McCarthy, P.J. Harley and R. H. Smallwood, “The feasibility of a mixed reality surgical training environment” in Virtual Reality, 12, pp.77-86, 2008. [4] D. Pitts, D. Rowley, J. Sher, “Assessment of performance in Orthopaedic Surgery”, J Bone Joint Surg, 87 B, No 9, pp 1187 – 1191, 2005

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-239

239

Design for Functional Occlusal Surface of CAD/CAM Crown Using VR Articulator Tomoko IKAWAa, Takumi OGAWAa, Yuko SHIGETAa, Shintaro KASAMAa, Rio HIRABAYASHIa, Shunji FUKUSHIMAa, Asaki HATTORIb and Naoki SUZUKIb. a Department of Fixed Prosthodontics Tsurumi University School of Dental Medicine b Institute for High Dimensional Medical Imaging, Jikei University School of Medicine { ikawa-tomoko, ogawa-t, shigeta-y, kasama-shintaro, hirabayashi-rio, fukushimas}@tsurumi-u.ac.jp,{hat, nsuzuki}@jikei.ac.jp

Abstract. In this present study, we introduce an approach that utilizes the VR articulator to mimic lateral excursions and design a functional occlusal surface. We then take the resultant occlusal surface from this approach and compare it with a conventional method. As a result, we developed a novel CAD/CAM system which can render a functional occlusal surface, via a VR articulator. The marginal fit and occlusion in our CAD/CAM crown was sufficient to apply to the clinic.. Keywords. CAD/CAM, Virtual reality articulator

Introduction In the dental field, several CAD/CAM systems have been developed [1] , and numerous dental CAD systems are commercially available and applied in the clinic [2]. One of the major roles of dental prosthesis is to reconstruct the patient’s dentition and to improve the masticatory function. In this present study, we would like to introduce an approach that utilizes the VR articulator to reproduce lateral excursions and design a functional occlusal surface. Once the process is completed, we will take the resultant occlusal surface from this approach and compare it with a conventional method. 1. Methods & Materials 1.1. Scanning and Reconstruction of Dentition and Die Models A virtual dentition model (VDM) was reconstructed through laser surface scanning data of dental casts which had been scanned with a 3D scanner (Shofu Inc. Kyoto, Japan) via 3D mesh modeling software VRMesh 4.1 (Fig.1). The relative position between the upper and lower VDMs was decided via the labial-buccal surface data, which was taken from the upper and lower casts placed at the intercuspal position.

240

T. Ikawa et al. / Design for Functional Occlusal Surface of CAD/CAM Crown

Figure 1. Virtual dentition models (VDM). a. Upper and lower dentition models. b. Abutment tooth

1.2. Development of VR Articulators The VR articulator was designed to produce a prostheses with a functional occlusal surface via SolidWorks in VR space. The VR articulator design was based on the construction and size of an existent semi-adjustable articulator (Shofu. Inc., Kyoto) (Fig. 2). The restriction point and the guiding paths were set on the VR articulator to reproduce the lateral excursion. The restriction points were set on the center of the condylar ball on the working side. The guiding paths were set on the incisal guide table and condylar guidance. The incisal guide pin and condylar ball were moved along these paths (Fig. 3a, b). The VR articulator was driven to adjust the occlusal surface during the lateral excursions. The incisal guide pin was moved 5 mm to right and left side. The dental antagonist data in each phase was merged as a functional antagonist model.

Figure 2. Design of VR Articulator Figure 3. Movement of the Left lateral excursion a. bottom view b. Front view

1.3. Design of the Functional Occlusal Surface In the first step, the equivalent type of tooth on the opposite side was copied and mirroring was performed. Subsequently the abatement teeth data was removed from the crown data. Through these steps, the base of the crown was molded in VR space. In the next step, the occlusal surface was designed via a VR articulator. 1.4. Assessment of the Designed VR Crown To assess the designed VR crown, the CAD/CAM crowns were generated according to a traditional approach. Subsequently, the accuracy of fit and the replication of the occlusal relationship in ICP and occlusion were assessed. The vertical gap between the finish line on the abutment teeth and the margin on the dental crown was measured

T. Ikawa et al. / Design for Functional Occlusal Surface of CAD/CAM Crown

241

through a digital microscope. 3 measurement points were then set on each surface. The clearance gap was compared between the conventional crown and CAD/CAM crown. The occlusal contacts were assessed with/without CAD/CAM crown via an articulator, via Black silicone technique. 2. Results The vertical gap between the finish line on the abutment teeth and the margin on the dental crown was 28±5 micro-meters in the conventional crown, 35±13 micro-meters in the CAD/CAM crown (Fig.4). The differential of the accuracy of fit between VR and conventional crown was within 10 micrometers. It was possible to compare the occlusal surfaces and the occlusal relationship with 2 types of crown, as there was no difference of the marginal fit between either factures. Fig.5 shows the occlusal contacts of the CAD/CAM crown in VR, the digital photo image (a, b), and with both images superimposed (c).

Figure 4. The gap between the finish line on the abutment teeth and the margin on the dental crown.

Figure 5. Comparison of the occlusal contacts: a. Designed occlusal contacts on the CAD crown in VR space (VR crown), b. Occlusal contacts on the CAD/CAM crown in real space (real crown), c. The VR crown image overlaid on the real crown image.

3. Conclusions We developed a novel CAD/CAM system which can render a functional occlusal surface, via a VR articulator. The marginal fit in our CAD/CAM crown was sufficient to apply to the clinic. Moreover, through our CAD/CAM system, we could produce a crown with the appropriate occlusal contacts. References [1] Liu PR, Essig ME. Panorama of dental CAD/CAM restorative systems. Compend Contin Educ Dent 29(8):482, 484, 2008. [2] Tinschert J, Natt G, Hassenpflug S, Spiekermann H. Status of current CAD/CAM technology in dental medicine. Int J Comput Dentistry. 7(1): 25-45, 2004.

242

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-242

Biopsym: a Learning Environment for Trans-Rectal Ultrasound Guided Prostate Biopsies Thomas JANSSOONEa, Grégoire CHEVREAUa,b, Lucile VADCARDc, Pierre MOZERb and Jocelyne TROCCAZa, 1 a TIMC-IMAG Laboratory, UMR 5525 CNRS UJF, Grenoble France b La Pitié Salpétrière Hospital, Paris, France c LSE (Laboratory for Educational Science), UPMF, Grenoble, France

Abstract. This paper describes a learning environment for image-guided prostate biopsies for cancer diagnosis; it is based on an ultrasound probe simulator virtually exploring real datasets obtained from patients. The aim is to make the training of young physicians easier and faster with a tool that combines lectures, biopsy simulations and recommended exercises to master this medical gesture. It is designed particularly to help improve the acquisition of the three-dimensional representation of the prostate required for practicing biopsy sequences. The simulator uses haptic feedback to compute the position of the virtual probe from three-dimensional (3D) ultrasound recorded data. This paper presents the current version of this learning environment. Keywords. Simulator, ultrasound, biopsy, educational content.

Introduction Prostate cancer is the most widespread cancer in men and the second cause of death after lung cancer in many countries. A primary diagnosis can be suspected from a blood analysis with an abnormal level of Prostate Specific Antigen (PSA) or from a suspicious digital rectal examination of the gland. However, the only way to reliably confirm prostate cancer is to detect one or several positives samples from the anatomopathological analysis of prostate biopsies. The simplest way to access the prostate for biopsy is through the rectum (see Figure 1 – left and middle). In a typical transrectal prostate biopsy, the doctor introduces an ultrasound (US) probe into the rectum to visualize the prostate and thus directs the needle to the sites which must be sampled. A mechanical guide attached to the US probe directs the needle to the target. The procedure must follow a well defined protocol in order to take sufficient samples to confirm or negate the presence of cancer. One very standard protocol is the 12-core process (see figure 1 – right), which must be carried out in a precise manner to minimize the discomfort to the patient. A protocol is needed because cancer is generally not visible in the US images, so the samples must be taken as 1

Author for correspondance. [email protected] - TIMC Lab, IN3S – Domaine de la Merci – School of Medicine – 38706 La Tronche cedex - France

T. Janssoone et al. / A Learning Environment for Trans-Rectal Ultrasound Guided Prostate Biopsies 243

regularly as possible throughout the prostate. Additional sites corresponding, for example, to suspicious zones located during a MRI examination can be added to the standard protocol.

Figure 1. US-guided prostate biopsy. (Left) anatomy, (middle) typical ultrasound image, (right) 12-core protocol.

This procedure needs a good 3D representation of the prostate, however, standard US images are 2D. Building an accurate mental 3D representation, well coordinated with the 2D US images, requires very good hand-eye coordination which may be difficult to acquire; as a consequence biopsies can quickly be poorly distributed. The conventional teaching method for this practice is only based on companionship: the student learns by observing an expert doing the operation and then reproduces it. With this method, it may be very difficult to understand the intra-rectal gesture only by observation. The hand-eye coordination and the spatial localization may be hard to acquire and may need long training, which is difficult to realize in a clinical environment. Moreover the evaluation of the performance of the trainee is purely qualitative. The following sections present a new way to teach this gesture using an ultrasound-based prostate biopsy simulator integrated in a complete learning environment with exercises.

1. Material and Methods In order to set up a new way of teaching ultrasound guided prostate biopsies, a complete learning environment is being built. The user, most often an intern urologist, must be completely autonomous with the system and should be able to learn everything about the procedure, from theoretical aspects like the anatomy of the organ, to practice exercises with simulation of biopsy sequences on specific cases. This learning environment should also provide practical advice about what to do during the execution of this gesture. Such a system aims at proposing a new method to learn this gesture as realistically and completely as possible. 1.1. The Simulator BiopSym simulates ultrasound biopsies of the prostate. A first kernel of the system was developed in 2008 [1]. Its principle is to use a force feedback device, Sensable Omni Phantom (see Figure 2), to simulate the interaction of the probe with the anatomy of the

244 T. Janssoone et al. / A Learning Environment for Trans-Rectal Ultrasound Guided Prostate Biopsies

patients. The stylus of this device represents the ultrasound probe. 3D ultrasound images acquired on real patients are used in the visualization. In the new version, whosegraphical user interface has been redesigned, evaluation of the trainee has been included, and exercises and educational content have been added.

Figure 2. (Left) US probe, (middle) Phantom modeling and (right) clipping planes.

The evaluation of the virtual biopsies is made using prostate segmentation [2]: the issue is to quantify the way in which the 12 zones of the prostate have been reached and what amount of tissue has been sampled inside the prostate (it is not rare to get samples out of the gland). The segmentation produces a mesh representing the organ and its bounding box; its edges are known. To locate all the sample sites, a simple decomposition of the box into 12 zones is done and comparisons between them and the coordinates of the needles in the simulator validate or invalidate the sample. 1.2. Architecture 3D US volumes used in the application were acquired during real biopsy sessions at La Pitiè Salpètrière hospital in Paris, France. Patient data (age, prostate size, PSA level, etc.) were also recorded in a SQL database. BiopSym uses open source libraries to compute the simulation. Users of this application will be urologists, so significant care was taken in avoiding common errors of medical software user interfaces. Lack of consistency and non intuitive design [3],[4],[5] are the most frequent criticisms made by medical software users. These properties were considered in order to build a user-friendly software with a “look & feel” design (see Figure 3).

Figure 3. Screenshots of BiopSym.

1.3. Exercises The list of exercises was described in a didactic survey based on an analysis of the objectives and methods used by urologists during prostate biopsies; it was made by a student in educational science of UPMF University working on the project. Its aim is to

T. Janssoone et al. / A Learning Environment for Trans-Rectal Ultrasound Guided Prostate Biopsies 245

help the user acquier a good mental representation of the prostate during an echoguided intervention as well as to help improve the user’s theoretical knowledge. Exercises include: (1) a questionnaire to determine the probability of a cancer regarding patients’ data (PSA level, prostate volume, age, etc.), (2) image recognition tasks, such as volume computation on US images (selection of diameters and height) or area localizations (selection of the bladder on a US volume for instance) and (3) several types of simulations with different constraints (volume of the prostate, position of the patient, target, MRI/US registration, etc). During simulation, assistance can be provided to help the user, and can consist, for example, in adding additional views (coronal, 3D, etc) that the user would not have in a standard biopsy procedure (see Figure 4).

Figure 4. Virtual biopsy simulation with assistance.

Evaluations of these exercises allow the software to target the user’s weaknesses and to subsequently recommend specific correction exercises in order to improve their control of the gesture. Thus, the user will be fully autonomous with the application and will be able to progress alone. User feedback, integrated into the personal panel of the application, display the pedagogical path and evolution of the performances of the user [6],[7]. The user is given the possibility to see their timeline representing everything they have done with the application to date. They can also get details about each activity: for example, they are able to visualize the result of the sequence of biopsies they made two months ago and to watch the results on a 3D view. Charts representing the evolution of their score are also available. 1.4. Educational Content The application also contains a complete lecture about prostate biopsy, prepared by a clinical expert. It can be decomposed into two parts: A slideshow where the expert reminds and explains the basic knowledge required to practice prostate biopsy: the anatomy of the prostate and its appearance in ultrasound images, its different components and the realisation of the gesture (see Figure 5).

246 T. Janssoone et al. / A Learning Environment for Trans-Rectal Ultrasound Guided Prostate Biopsies

-

Lectures about the biopsy procedure: the method used, the checklist of all the details needed to make it successful and an explanatory text from the “Association Française d’Urologie” (French Association of Urology).

The educational part provides theoretical support to the exercises and helps the user to acquire a solid background about this medical gesture. All the resources can be transferred by the trainee to a USB pen.

Figure 5. Examples of educational content.

2. Results and discussion BiopSym allows the teaching of ultrasound guided prostate biopsies as pertinently and completely as possible. It provides theoretical support and, above all, a new way to learn the gesture using simulations. The exercises and their recommendations will help the users to progress and allow them to be totally autonomous. The assistance during simulation will help them to rapidly acquire a good mental representation of the prostate and so better master the biopsy procedure. The feedback on user performance allows the users to visualize their progress. The evaluation of BiopSym on real medical student is scheduled for 2011.

References [1] S. Sclaverano, G. Chevreau, L. Vadcard, P. Mozer and J. Troccaz. BiopSym: a simulator for enhanced learning of ultrasound-guided prostate biopsy. Proceedings of MMVR’2009 [2] S. Martin, M. Baumann, V. Daanen and J. Troccaz. MR prior based automatic segmentation of the prostate in trus images for mr/trus data fusion. In Proceedings of IEEE International Symposium on Biomedical Imaging , ISBI’2010 [3] A. Noriyoshi, N. Naoki and T.Natsuko. Human-Centered design in medical fields. Fujitsu Scientific & Technical Journal (FSTJ) Human-Centered design. 45(2), 2009-4 [4] Y. Batu Salman, J. Young Kim, A. Karahoca and H. Cheng. Participatory icon design process for medical information system. In Proceedings of IASDR 2007 [5] A. Flory, C. Verdier and S.Sassi Nouvelles interfaces pour la représentation de l’information médicale. In french. Proceedings of INFORSID 2006 [6] A. Holzinger, M. D. Kickmeier-Rust, S. Wassertheurer and M. Hessinger. Learning performance with interactive simulations in medical education: Lessons learned from results of learning complex physiological models with the HAEMO dynamics SIMulator. Computers & Education, 52(1), 292-301, 2009 [7] M.Ebner and A. Holzinger. Successful Implementation of User-Centered Game Based Learning in Higher Education – an Example from Civil Engineering. Computers & Education, 49(3), 873-890, 2007.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-247

247

Comparison of Reaching Kinematics During Mirror and Parallel Robot Assisted Movements Zahra KADIVARa, 1, Cynthia SUNGb, Zachary THOMPSON b, Marcia O’MALLEY b, Michael LIEBSCHNER c, and Zhigang DENG d a Department of Physical Medicine and Rehabilitation, Baylor College of Medicine b Department of Mechanical Engineering, Rice University c Veterans Affairs Medical Center, Baylor College of Medicine d Computer Science Department, University of Houston

Abstract. The use of robotic devices in rehabilitation allows therapists to administer the desired movement with the preferred level of assistance while expending minimum effort. Robotic devices have been used in recent years to enhance sensori-motor recovery of the impaired arm in persons with stroke. Despite recent recommendations for bimanual practice, robot-assisted bimanual activities are rarely explored and are limited to mirror image movements. We developed a novel parallel movement mode for the Mirror Image Movement Enabler robotic system and investigated trajectory error (TE) exhibited by healthy adults during parallel and mirror image motions to various target locations. TE values differed for parallel and mirror image motions and for certain target locations, suggesting the importance of considering these factors when developing robot-assisted bimanual activities. Keywords. trajectory error, robotic devices, upper limbs

Introduction Stroke is the leading cause of neurological disability in the United States[1]. Hemiparesis due to stroke is the primary cause of disability[2]. Most importantly, arm paresis is perceived as the primary cause of disability by individuals who have suffered stroke because of the limitations it creates in performing activities of daily living (ADL)[3]. Rehabilitation of the impaired limb/s is essential for improving motor function after stroke[4], yet only 30.7% of stroke survivors receive outpatient rehabilitation[5]. Therefore, effective therapy for upper-limb paresis must be addressed. Approximately 80% of all stroke survivors suffer from upper limb paresis and only 18% of these individuals gain full motor recovery with conventional treatments in the year following stroke[6-8]. Thus, continued rehabilitation of the impaired limb/s is needed. Several studies indicate that with proper treatment, arm recovery can occur years after the stroke incident[3]. For example, repetitive, task specific training of the affected limb can result in significant motor recovery more than one year after the 1 Corresponding Author: Department of Physical Medicine and Rehabilitation, Baylor College of Medicine, One Baylor Plaza, Houston, TX, 77082; Email: [email protected]

248

Z. Kadivar et al. / Comparison of Reaching Kinematics

stroke incident[8]. Experiments show that robot-assisted training of the impaired arm can be as effective as unassisted repeated practice of the impaired arm [9, 10] and more effective than neuro-developmental therapy commonly used for motor recovery after stroke[11]. Furthermore, robotic rehabilitation systems offer increased efficiency, lower expenses, and new sensing capabilities to the therapist. Taken together, the recent successes and distinct advantages offered by robotic systems have garnered much attention. Despite significant motor improvements reported after repeated practice of the impaired limb, recent neurophysiological evidence indicates greater benefits from bimanual practice[12]. These investigations report cortical reorganization in contralesional and ipsilesional brain hemispheres[12, 13] and enhanced interhemispheric activation[14] with bimanual practice. As a result, several robotic devices have been developed specifically for bimanual training, including the 6 degree-offreedom Mirror Image Movement Enabler (MIME) system[11, 15], the 2 degree-offreedom Bi-Manu-Track system[16, 17], and the 2 degree-of-freedom Bilateral ForceInduced Isokinetic Arm Movement Trainer (BFIAMT)[10]. These systems have achieved varying results. When added to conventional muscle tone normalization and range of motion exercises, resistive training with BFIAMT enhanced arm strength[18]. Bimanual mirror image training combined with unilateral arm practice with MIME led to improvements in the Fugl-Meyer scores of the proximal joints[11], while bimanual training alone using the MIME did not differ from conventional (neuro-developmental) treatments[19]. Training with Bi-Manu-Track, on the other hand, had no effects on the arm motor recovery even though it reduced wrist spasticity[16]. The abovementioned robotic systems deliver bimanual movements in a mirror image fashion and can operate in a master-slave mode where the movement of the impaired limb is directed by that of the unimpaired limb. When guided by the unimpaired limb, the affected arm experiences better joint coordination[10], so sensory afferent signals come from better coordinated movement patterns and further facilitate cortical reorganization[20]; however, use of mirror image movements alone can be criticized from the motor learning perspective. According to task specificity and variable practice principles, movements that resemble ADL and training protocols that involve a wide variety of activities result in better and more functional motor progress. To address this limitation, we developed a parallel mode for the MIME robotic system. It is reasonable to assume that parallel movements resemble more bimanual daily activities than mirror image motions (i.e., many reaching tasks) and can help attain a more variable practice for robot-assisted bimanual activities. The goal of the study was to compare the newly developed parallel movements to mirror image movements in healthy adults while reaching to various targets with the MIME robot. Findings can provide valuable knowledge for developing effective training regiments for stroke patients and other populations that can benefit from bimanual robot-assisted training.

1. Material and Methods Figure 1 shows the experimental setup using the MIME system, which consisted of a PUMA-560 robot and a position digitizer. Participants were seated in front of a low table with forearms placed in padded forearm splints at mid-position.

Z. Kadivar et al. / Comparison of Reaching Kinematics

249

Figure 1. A picture of the experiment set up. Right-handed participants were seated in front of a target array and placed their arms in forearm splints. The right splint was attached to the Puma-560 robot (Stäubli Unimation, Inc.). Optical encoders in each of the robot’s six joints allowed monitoring of the position and orientation of the splint, while a six-axis force-torque sensor (resolution 0.25N) on the end of the robot measured any forces and torques applied. The left splint was attached to the position digitizer (MicroScribe3DLX, Immersion Corp.), which could measure arbitrary trajectories for the left forearm while applying minimal resistance to movement.

Splints were supplemented with pen-like extensions to allow for precision reaching to targets. A software controller monitored the trajectory of the left arm and commanded the robot to produce corresponding positions and orientations in the right arm. For the purpose of this experiment, two modes were used: a mirror image mode (pre-existing) and a novel parallel mode. In mirror mode, the robot continuously moved the right arm to the mirror image position of the left. In parallel mode, the robot moved the right arm in the same direction as the left. Healthy right-handed participants were recruited to take part in this study. All the participants signed consent forms approved by the Institutional Review Board of each participating individual. All participants performed simple reaching exercises with the MIME robot. Targets were ping-pong balls arranged as a vertical six-by-three array and instrumented with blinking LEDs. The three rows indicated upper, middle, and lower reaching points and the six columns allowed bimanual parallel reach to the right and left and bimanual mirror reach to inner and outer targets (Figure 2). Each session consisted of performing 48 reaching trials at a comfortable speed. For all trials, the MIME system operated in the master-slave mode where the left arm was attached to the digitizer and was therefore the leading arm. Participants performed unilateral, parallel, and mirror image reaching movement in separate training sessions. The first training session was for familiarization purposes and involved unilateral reaching with the left arm. Thereafter, half of the participants performed the parallel and mirror image tasks on sessions two and three respectively, while the remaining performed these tasks in the opposite session order. All training sessions were completed in three consecutive days. This design allowed all the participants to perform parallel and mirror movements while accounting for practice order. The participants were instructed to reach to the illuminated target pairs presented in a random order as accurately as possible at their comfortable speed. Data were sampled from the MIME at approximately 110 Hz. Trajectory error (TE) was selected as the primary measure of interest. This measure represents the difference between the performed and the desired trajectories. For reaching movements

250

Z. Kadivar et al. / Comparison of Reaching Kinematics

the most natural trajectory is a straight path[21].

Figure 2. The schema of the vertical target array used for the reaching task and the associated reaching pattern. Six target pairs were illuminated for parallel (left panel) and mirror (right panel) conditions in random order and are depicted in different patterns. For the parallel condition participants reached to right or left target pairs at upper, middle and lower rows. For the mirror condition participants reached to outer or inner target pairs at the three rows.

Hence for current calculations the desired trajectory was taken to be the straight line between the starting and end points of the reaching movement. Since only the reaching portion of the task was goal oriented, analysis did not include the return path to the starting position. In addition, since the bilateral mode of MIME operates in a masterslave fashion, paths were analyzed for the left arm only. For each trial, the position at , and the direction of the the beginning of the movement was set to straight-line path was calculated by

(1) where is a unit vector, is the trajectory of the left forearm, ti is the time at the beginning of the movement, and tf is the time at the end of the movement. The error in trajectory for each point in time was then determined by taking the component of the , position vector that was perpendicular to (2) The final TE was the magnitude of this error value summed over the length of the trial and normalized over the total distance and duration of the movement to account for differences in reaching distance and speed across participants where N is the number of samples between ti and tf.

(3) Normality of the data was confirmed using a quantile-quantile plot. Repeated measures ANOVA with the main factors of condition (two: parallel, mirror) and target (six: two in each row) and repeated factor of trial was used for data analysis. Kenward

Z. Kadivar et al. / Comparison of Reaching Kinematics

251

Rodgers adjusted degrees of freedom was used to account for the small sample size. Tukeys post hoc analysis was conducted when appropriate. Alpha was set at 0.05.

2. Results Ten right-handed participants with an average age of 23.3± 2.7 years successfully completed eight repetitions for each target pair presented in random order (48 trials in each training session). Training did not result in any significant changes in the calculated TE values across trials; however, there was a significant main effects for condition (F1,818= 192.17, P < 0.0001): TE was significantly larger for the parallel condition than the mirror image condition. There was also a significant effect of target (F5,818= 44.51, P < 0.0001) where upper-row target pairs had smaller TE values. The significant condition x target interaction (F5,818= 40.59, P < 0.0001) clarified the main effect findings and is presented in Figure 3. Based on the interaction, TE values were smaller for the mirror image condition when compared to the corresponding target rows in the parallel condition. This difference was not always significant. Lower right targets in the parallel condition had the largest TE values.

3. Discussion Trajectory error is a scalar measure that is commonly used to represent the kinematic error of point-to-point movements. When performing goal directed reaching tasks, motor commands that yield the smallest kinematic error are the most optimal and are preferred by the central nervous system (CNS)[22]. Another central preference is for the two limbs to operate in spatial symmetry and in an in-phase relationship (i.e. both arms in abduction or adduction at a specific point of time) during bimanual activities [23]. This predominance is observed in healthy older adults and those affected by stroke[24]. In-phase movements are less demanding on the CNS[25] and are therefore easier to maintain, more stable[24], less variable[26], and more accurate[27] than outof-phase movements (i.e., parallel). The in-phase relationship present for the mirror condition, together with the smaller TE values for this motion regardless of the target location, indicate better performance for mirror-type motion and further support previous findings. It should be noted that there is evidence that out-of-phase movements can be mastered with repeated practice[28]. For our experiments, trajectory-error values did not change with training for the parallel and mirror conditions, indicating the potential need for longer practice. The effect of target location on reaching kinematics was also evident from the current findings. Targets at the upper level had the smallest TE values for both parallel and mirror conditions and TE was largest for the lower right target pairs in the parallel condition (Fig 3). The upper targets were at eye-level for most participants, so it is tempting to assume this is the reason behind the smaller TE for upper targets; however, previous findings suggest that the most influential factor in goal oriented reaching is the “target laterality,” the side to which reaching must occur[29]. Therefore, placing targets at eye-level is not necessarily advantageous[30]. The exact visuo-spatial integration processes are unknown and other factors such as eye dominance[31], which can further affect reaching kinematics, were not accounted for in this study. Hence, no specific

252

Z. Kadivar et al. / Comparison of Reaching Kinematics

conclusions can be drawn for the differences across the presented targets, and further research is required to clarify the current findings.

Figure 3. Mean trajectory error (TE) values are presented for the condition x target interaction. Values are ordered by row (upper (U), middle (M), lower (L)), followed by the target pairs (left (L) and right (R) for the parallel condition (P), and outer (O) and inner (I) for the mirror (M) condition). Asterisks indicate significant differences between TE values for target pairs in the same row and condition. The TE values were smallest for the upper targets of both conditions (dashed area) and largest for the lower-right target pair in the parallel condition (dotted area). Error bars represent the standard error.

In conclusion, results from the current investigation indicate the importance of considering the movement pattern and the target location when developing bimanual efficient training protocols. Protocols similar to that used in this study can be implemented prior to training to determine individual characteristics and customize training type and duration accordingly.

References [1]

[2] [3] [4] [5] [6]

[7]

P.A. Wolf, G.P. Clagett, J.D. Easton, L.B. Goldstein, P.B. Gorelick, et al., Preventing ischemic stroke in patients with prior stroke and transient ischemic attack : a statement for healthcare professionals from the Stroke Council of the American Heart Association, Stroke 30 (1999), 1991-4. R. Bonita and R. Beaglehole, Recovery of motor function after stroke, Stroke 19 (1988), 1497-500. J.G. Broeks, G.J. Lankhorst, K. Rumping, and A.J. Prevo, The long-term outcome of arm function after stroke: results of a follow-up study, Disabil Rehabil 21 (1999), 357-64. J. Liepert, H. Bauder, H.R. Wolfgang, W.H. Miltner, E. Taub, et al., Treatment-induced cortical reorganization after stroke in humans, Stroke 31 (2000), 1210-6. "Outpatient rehabilitation among stroke survivors-21 States and the District of Columbia, 2005," Centers for Disease Control and Prevention2007. S.A. Combs, S.P. Kelly, R. Barton, M. Ivaska, and K. Nowak, Effects of an intensive, task-specific rehabilitation program for individuals with chronic stroke: a case series, Disabil Rehabil 32 (2000), 669-78. M. Rijntjes, K. Haevernick, A. Barzel, H. van den Bussche, G. Ketels, et al., Repeat therapy for chronic motor stroke: a pilot study for feasibility and efficacy, Neurorehabil Neural Repair 23 (2009), 275-80.

Z. Kadivar et al. / Comparison of Reaching Kinematics [8]

[9]

[10]

[11]

[12]

[13]

[14]

[15] [16]

[17]

[18]

[19]

[20]

[21] [22]

[23] [24] [25] [26] [27] [28]

[29] [30] [31]

253

K. Wing, J.V. Lynskey, and P.R. Bosch, Whole-body intensive rehabilitation is feasible and effective in chronic stroke survivors: a retrospective data analysis, Top Stroke Rehabil 15 (2008), 247-55. L.E. Kahn, P.S. Lum, W.Z. Rymer, and D.J. Reinkensmeyer, Robot-assisted movement training for the stroke-impaired arm: Does it matter what the robot does?, J Rehabil Res Dev 43 (2006), 619-30. G.N. Lewis and E.J. Perreault, An assessment of robot-assisted bimanual movements on upper limb motor coordination following stroke, IEEE Trans Neural Syst Rehabil Eng 17 (2009), 595604. P.S. Lum, C.G. Burgar, P.C. Shor, M. Majmundar, and M. Van der Loos, Robot-assisted movement training compared with conventional therapy techniques for the rehabilitation of upperlimb motor function after stroke, Arch Phys Med Rehabil 83 (2002), 952-9. J.J. Summers, F.A. Kagerer, M.I. Garry, C.Y. Hiraga, A. Loftus, et al., Bilateral and unilateral movement training on upper limb function in chronic stroke patients: A TMS study, J Neurol Sci 252 (2007), 76-82. A.R. Luft, S. McCombe-Waller, J. Whitall, L.W. Forrester, R. Macko, et al., Repetitive bilateral arm training and motor cortex activation in chronic stroke: a randomized controlled trial, JAMA 292 (2004), 1853-61. J.H. Cauraugh, S.A. Coombes, N. Lodha, S.K. Naik, and J.J. Summers, Upper extremity improvements in chronic stroke: coupled bilateral load training, Restor Neurol Neurosci 27 (2009), 17-25. C.G. Burgar, P.S. Lum, P.C. Shor, and H.F. Machiel Van der Loos, Development of robots for rehabilitation therapy: the Palo Alto VA/Stanford experience, J Rehabil Res Dev 37 (2000), 663-73. S. Hesse, G. Schulte-Tigges, M. Konrad, A. Bardeleben, and C. Werner, Robot-assisted arm trainer for the passive and active practice of bilateral forearm and wrist movements in hemiparetic subjects, Arch Phys Med Rehabil 84 (2003), 915-20. S. Hesse, C. Werner, M. Pohl, S. Rueckriem, J. Mehrholz, et al., Computerized arm training improves the motor control of the severely affected arm after stroke: a single-blinded randomized trial in two centers, Stroke 36 (2005), 1960-6. J.J. Chang, W.L. Tung, W.L. Wu, M.H. Huang, and F.C. Su, Effects of robot-aided bilateral force-induced isokinetic arm training combined with conventional rehabilitation on arm motor function in patients with chronic stroke, Arch Phys Med Rehabil 88 (2007), 1332-8. P.S. Lum, C.G. Burgar, M. Van der Loos, P.C. Shor, M. Majmundar, et al., MIME robotic device for upper-limb neurorehabilitation in subacute stroke subjects: A follow-up study, J Rehabil Res Dev 43 (2006), 631-42. G.N. Lewis, W.D. Byblow, and R.G. Carson, Phasic modulation of corticomotor excitability during passive movement of the upper limb: effects of movement frequency and muscle specificity, Brain Res 900 (2001), 282-94. Y. Uno, M. Kawato, and R. Suzuki, Formation and control of optimal trajectory in human multijoint arm movement. Minimum torque-change model, Biol Cybern 61 (1989), 89-101. R.A. Scheidt, D.J. Reinkensmeyer, M.A. Conditt, W.Z. Rymer, and F.A. Mussa-Ivaldi, Persistence of motor adaptation during constrained, multi-joint, arm movements, J Neurophysiol 84 (2000), 853-62. W.D. Byblow, Expressions of asymmetries and anchoring in bimanual coordination Human Movement Science 13 (1994), 3-28. G.N. Lewis and W.D. Byblow, Bimanual coordination dynamics in poststroke hemiparetics, J Mot Behav 36 (2004), 174-88. P.G. Zanone, A. Monno, J.J. Temprado, and M. Laurent, Shared dynamics of attentional cost and pattern stability, Hum Mov Sci 20 (2001), 765-89. M.A. Rogers, J.L. Bradshaw, R.C. Cunnington, and J.G. Phillips, Inter-limb coupling in coordinated bimanual movement: attention and asymmetries, Laterality 3 (1998), 53-75. R.G. Carson, The dynamics of isometric bimanual coordination, Exp Brain Res 105 (1995), 465-76. J.J. Temprado and S.P. Swinnen, Dynamics of learning and transfer of muscular and spatial relative phase in bimanual coordination: evidence for abstract directional codes, Exp Brain Res 160 (2005), 180-8. J.D. Fisk and M.A. Goodale, The organization of eye and limb movements during unrestricted reaching to targets in contralateral and ipsilateral visual space, Exp Brain Res 60 (1985), 159-78. R.B. Welch and R.B. Post, Accuracy and adaptation of reaching and pointing in pitched visual environments, Percept Psychophys 58 (1996), 383-9. E. Shneor and S. Hochstein, Eye dominance effects in feature search, Vision Res 46 (2006), 4258-69.

254

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-254

Serious Games in the Classroom: Gauging Student Perceptions Bill KAPRALOSa,1, Sayra CRISTANCHOb, Mark PORTEc,d, David BACKSTEINc,d, Alex MONCLOUe, and Adam DUBROWSKIf,g a

Faculty of Business and Information Technology, and Health Education Technology Research Unit, University of Ontario Institute of Technology. Canada. b Department of Surgery and Centre for Education Research & Innovation, Schulich School of Medicine & Dentistry, University of Western Ontario. Canada. c Division of Orthopaedic Surgery, Department of Surgery, University of Toronto. Toronto, Canada. d Mount Sinai Hospital, Toronto, Canada. e Faculty of Electronics Engineering, Pontificia Bolivariana University, Bucaramanga, Colombia. f The Hospital for Sick Children Learning Institute. Canada. g Department of Pediatrics, Faculty of Medicine, and The Wilson Centre, University of Toronto. Canada.

Abstract. Serious games, or video game-based technology applied to training, learning applications, provide a high fidelity simulation of particular environments and situations that focus on high level skills that are required in the field. Given the popularity of video games, particularly with today’s generation of learners, and the growing trend of restricted resident work hours and diminsihed operating room exposure due to limited budgets increased case complexity and medicolegal concerns, serious games provide a cost-effective viable training option. To develop effective serious games, the views and perceptions of both the end users (learners) and educators regarding their use “in the classroom” must be assessed and accounted for. Here we present the results of a survey that was designed to assess students’ perceptions of serious games. Keywords. Serious games, virtual simulation, game-based learning, interactive learning.

Introduction Simulations for educational purposes range from decontextualized bench models and virtual reality-based environments to high fidelity recreations of actual training settings [1]. Although virtual reality-based technologies have been incorporated in the teaching/training curriculums of a large number of professions across various industries (including surgery) for several decades, the rising popularity of video games has seen a recent push towards the application of video game-based technologies to teaching and learning. Gaming and interactive simulation environments support learner-centered education whereby learners are able to actively work through problems while acquiring knowledge through interactive practice learning thereby allowing the player to learn via

B. Kapralos et al. / Serious Games in the Classroom: Gauging Student Perceptions

255

an active, critical learning approach [2]. Game-based learning provides a methodology to integrate game design concepts with instructional design techniques to enhance the educational experience for students [3]. Video games provide students the opportunity to learn to appreciate the inter-relationship of complex behaviors, sign systems, and the formation of social groups [4]. Games inherently support experiential learning by providing students with concrete experiences and active experimentation [5]. By designing the scenario appropriately, a problem-based learning approach can be realized [6]. Similar to a good game designer, an educator should provide trainees/learners with an environment that promotes learning through interaction [2]. Although no particularly clear definition of the term is currently available, serious games is a term that has been used to describe video games that have been designed specifically for training and education [7]. Serious games provide a high fidelity simulation of particular environments and situations that focus on high level skills that are required in the field. They present situations in a complex interactive narrative context coupled with interactive elements that are designed to engage the trainees. Goals and challenges require the trainees to solve specific problems that they may have never seen prior to engaging in the game increasing the fun factor. In addition to promoting learning via interaction, there are various other benefits to serious games. More specifically, they allow users to experience situations that are difficult (even impossible) to achieve in reality due to a number of factors including cost, time, and safety concerns. In addition, serious games support the development of various skills including analytical and spatial, strategic, recollection, and psychomotor as well as visual selective attention [8]. Further benefits of serious games include improved self-monitoring, problem recognition and solving, improved short-and longterm memory, increased social skills and increased self-efficacy [9]. Surgical training over the last 200 years has followed the master-apprenticeship model whereby the resident (trainee) acquires the required surgical techniques, skills, and knowledge in the operating room in a see one, do one, teach one manner [10]. However, such “hands-on” training of residents leads to increased resource consumption (e.g., monetary, faculty time, and time in the operating room) and has generally become more costly [11]. There is a growing trend of decreasing resident work hours in North America and globally due to political mandate [12]. This has lead to decreased training time in the operating room and hence less operative exposure, teaching, and feedback [13] and generally, traditional educational methods in surgery have come under increasing scrutiny [14]. Furthermore, with respect to technical skills development, the master-apprenticeship model is largely dependent upon the random flow of patients through the operating room thus, it fails to provide skills acquisition in an organized fashion leading to an unpredictable curriculum and educational content that the trainee is provided with [14]. It is therefore evident that given the increasing time constraints, trainees are under great pressure to acquire complex surgical cognitive and technical skills. Therefore, efforts must be made to optimize operative room exposure by devising training opportunities using artificial settings before exposure to patients. Other available alternative methods for surgical training include the use of animals, cadavers, or plastic models, each option with its share of problems [15]. We aim to use serious games to train cognitive skills by providing a platform where trainees are able to learn and practice the required steps and instruments for a particular surgical procedure. We argue that once these preliminary cognitive skills are consolidated, trainees will be able to focus solely on refining the corresponding technical skills by

256

B. Kapralos et al. / Serious Games in the Classroom: Gauging Student Perceptions

using other modalities of simulation (e.g., bench-top, part trainers, or high fidelity models). In order to develop effective virtual simulations/serious games, the views and perceptions of both the end users (learners) and educators must be initially assessed and accounted for. Here we present the results of a survey that was designed to assess the perceptions about serious games from a broad set of undergraduate students, who would eventually pursue medical training. The study was conducted across institutions in North and South America in order to explore differences in perceptions (if any) that could be related to differences in the cultural contexts.

1. Methods We conducted a study to gauge student perceptions of serious games and their use as educational tools “in the classroom”. The survey was designed and developed using the Survey Monkey survey software and questionnaire tool (http://surveymonkey.com). It was conducted in the form of an online survey and targeted students primarily from two institutions in Canada (University of Ontario Institute of Technology or UOIT), and Colombia (Pontificia Bolivariana University, Bucaramanga) across various faculties, including Health Sciences, and Engineering, during a two-month period (March 1 – April 30 2010). UOIT is a laptop-based institution and the use of technology in the classroom is emphasized and encouraged. The survey consisted of 14 questions of various types and more specifically: yes/no, multiple choice, Likert scale, and open questions. For close-ended questions, subjects were given a list of responses and asked to select their choice(s). Questions requiring a simple yes or no response were followed by an open-ended question in which participants were asked to expand/explain their response. We collected a total of 183 completed surveys (141 from Canada and 42 from Colombia). This participant pool allowed us to achieve a diversified analysis of results. The age range of the participants was 18-30. The survey was kept completely anonymous; respondents were not required to provide their name and were simply asked to provide their current faculty/program affiliation and age. The survey abided by the University of Ontario Institute of Technology’s Research Ethics Board policies for the ethics review process for experiments involving human participants.

2. Results Responses for each question were analyzed separately for the Canadian and Colombian contexts, although responses did not reveal major differences between the two groups of students. As an overview of the students’ perceptions, we present here the consolidated results for the 16 questions in the survey (see illustrations below). The results of Questions 1-2 (see Figure 1(a) and (b) respectively) revealed that the majority of respondents play video games and of those that do play video games, more than half spend over 10 hours per week doing so. The results of Question 3 (see Figure 2(a)) revealed that 67% of the respondents have never used serious games or virtual simulations for educational purposes. However, of the 33% that have used serious games, the majority believes they are beneficial to their learning experience (Question 4; see Figure 2(b)) and beneficial with

B. Kapralos et al. / Serious Games in the Classroom: Gauging Student Perceptions

257

respect to improving cognitive abilities including reaction time, concentration, problem solving, etc. (Question 5; see Figure 2(c)).

Figure 1. Results. (a) Question 1: “Do you enjoy playing video games as a past time?” and (b) Question 2: “How many hours do you play a week?”

Figure 2. Results. (a) Question 3: “Have you ever used a serious game or virtual simulation in your education?” (b) Question 4: “To what extent do you feel it was beneficial to your learning experience?” and (c) Question 5: “In general, to what extent do you feel video games are beneficial to improving cognitive abilities such as reaction time, concentration, problem solving, etc.?”

Questions 6 asked participants to rate various learning techniques including the use of PowerPoint slides, watching videos, and reading the course textbook (see Figure 3(a)) on a five-point scale ranging from “Very Beneficial” to “Not beneficial at all”, while for Question 7, participants rated a number of statements regarding learning (e.g., “Applied learning is not necessary for successful training”) once again, using the same five-point scale as in Question 6 (see Figure 3(b)).

Figure 3. Results. (a) Question 6: “Rate the following learning methods”, and (b) Question 7: “Rate the following statements about learning.”

258

B. Kapralos et al. / Serious Games in the Classroom: Gauging Student Perceptions

Questions 8-11 focused on perceptions on serious games. Overall, participants generally believed that virtual simulations/serious games may be beneficial to their learning experiences. More specifically, 71% would prefer them to traditional learning methods (Question 8; Figure 4(a); 88% believe they should be incorporated into the curriculum (Question 9; Figure 4(b); 69% would pay for them (Question 10; Figure 4(c); 83% believe universities would provide them with a higher level of education if they did incorporate virtual simulations/serious games (Question11; Figure 4(d)). Finally, the last set of questions (12, 13, and 14) further gauged participant’s perceptions on serious games for health professions/medical/surgical education and training on a five-point scale ranging from “Completely Agree” to “Completely Disagree”. Results are summarized in Figures 5(a)-(c). Overall, participants believed virtual simulations/serious games offer great potential for the health professions education field.

Figure 4. Results. (a) Question 8: “Would you prefer virtual simulation/serious game based learning over traditional styles of learning?”, (b) Question 9: “Do you think that virtual simulation/serious games should be incorporated into course materials?”, (c) Question 10: “Would you pay for serious gaming to be incorporated into your learning experience?”, and (d) Question 11: “Do you think that Universities would be offering a higher level of education if they were to incorporate virtual simulation/serious games into their courses?”

Figure 5. Results. (a) Question 12: “Rate the following questions about serious games.”, (b) Question 13: “Rate the following learning methods on the basis of medical surgery training and education.”, and (c) Question 14: “Rate the following statements about serious games from a medical perspective.”

B. Kapralos et al. / Serious Games in the Classroom: Gauging Student Perceptions

259

At the end of the survey participants were provided with a section to express their thoughts on virtual simulations/serious games. We received approximately 40 responses, which provided thorough insight into the perspective of university students. A few of the more compelling comments we received are noted below: “I believe that simulations and serious games offer a great deal of added value to an education. By being walked through a procedure, with the rules at hand and being told instantly when a mistake is made, the learning experience is magnified. However, a serious game could only very rarely be the only component of a course. When executed properly, a serious game is an invaluable supplement.”

“These technologies should be applied as learning methods only as a complement in education, we cannot rely entirely on them even when high realism levels are reached. This doesn't mean it is an area that has no worth to explore.” “I wish we had this type of learning tool in our courses. It would be interesting and fun, but could also teach. Practical hands on training is a good learning method for many individuals and specifically in the medical field would be very useful for testing techniques.” “The problem with virtual games is that the consequences are not "real" enough for a person to take seriously. Graphical representations are much harder to take serious. Despite this the possibility for games to help in education are there, it’s just hard to implement and take seriously in my opinion.” The first two comments raise an interesting issue regarding the limitations of serious games as a learning tool. Although they provide important benefits including hands on learning, and the ability to learn from mistakes, serious games would likely need to be used in conjunction with other learning methods. The first comment highlights one of the main benefits of serious games, and more specifically, their ability to allow players to learn from their mistakes. This ability is very difficult, if not impossible, with any other learning tool to achieve at such a high level of realism. The third comment highlights participant interest towards the possibility of being able to utilize this learning tool in their particular field. The last comment brings forth another limitation of serious games as a learning tool in relation to the “stress” imposed by an artificial situation vs. a real life situation. However, this does not imply that this type of training be omitted; it can be an indication that the development of serious games should be preceded by an educational needs assessment in order to decide the type of skills to be transferred with this technology (e.g., cognitive or communication skills).

3. Conclusions In this paper we have presented the results of a survey conducted at the University of Ontario Institute of Technology (Canada) and the Pontificia Bolivariana University, Bucaramanga (Colombia), to gauge students’ perceptions on the use of serious games. Students from a wide range of programs/disciplines including Health Sciences participated in the study. Results confirmed students’ appreciation for the use of serious games to complement traditional teaching and learning methods and techniques. However, serious games should be carefully designed to ensure the appropriate knowledge skills are transferred to the user. The implementation of this survey has

260

B. Kapralos et al. / Serious Games in the Classroom: Gauging Student Perceptions

served as a screening activity to support the contention that given the new generation of students (i.e., more technologically driven), teaching tools should be adapted to ensure effective learning. The results (i.e., willingness of students to use serious games) may potentially support the argument that if any effect (i.e., knowledge transfer, retention) is shown during the serious game evaluation process, then it may be attributed to the design of the serious game rather than to lack of motivation. Acknowledgments. The following UOIT undergraduate students conducted the survey

as part of their UOIT fourth year Capstone project and their help is much appreciated: Adamo Mavilla, Angela Lutrzykowski, Danielle Keller, Leonardo Colangelo, and Richard Comeau. The financial support of the Natural Sciences and Engineering Research Council of Canada (NSERC) in the form of individual Discovery Grants to Bill Kapralos and Adam Dubrowski is gratefully acknowledged.

References [1] [2] [3] [4]

[5] [6] [7] [8] [9] [10] [11] [12]

[13] [14] [15]

R. L. Kneebone. Practice, rehearsal, and performance: An approach for simulation-based surgical and procedure training. Journal of the American Medical Association, 302 (2009), 1336–1338. A. Stapleton. Serious games: Serious opportunities. In Proceedings of the 2004 Australian Game Developers Conference, pages 1-6, Melbourne, Australia, 2004. K. Kiil. Digital game-based learning: Towards an experiential gaming model. The Internet and Higher Education, 8 (2005), 13-24. D. Lieberman. Interactive video games for health promotion: Effects on knowledge, self-efficacy, social support and health. In Gold, R.L. and Manning, T. (eds.) Health Promotion and Interactive Technology, pages 103–120. Lawrence Erlbaum Associates, Norwell, NJ. USA. D. Kolb. Experiential learning: experience as the source of learning and development. Prentice-Hall Publishers, Englewood Cliffs, New Jersey: 1984. J. Savery, and T. Duffy. Problem based learning: An instructional model and its constructivist framework. Educational Technology, 35 (1999), 31-38. L. Annetta. The “I’s” have it: A framework for serious educational game design. Review of General Psychology, 14(2010), 105–112. A. Mitchell, and C. Savill-Smith, C. The use of computer and video games for learning: A review of the literature. Accessed from https://crm.lsnlearning.org.uk/user/order.aspx?code=041529 on Oct. 29 2010. D. Michael, and S. Chen. Serious games: Games that educate, train and inform. Thomson Course Technology, Boston, MA. USA, 2006. R. Wigton, See one, do one, teach one. Academic Medicine, 67 (1992):743. R. K. Reznick. Teaching and testing technical skills. American Journal of Surgery, 165 (1993), 358– 361. J. Zuckerman, E. Kubiak, I. Immerman, and P. DiCesare. The early effects of code 405 work rules on the attitudes of orthopaedic residents and attending surgeons, Journal of Bone and Joint Surgery, 87 (2005), 903–908. B. A. Weatherby, J. N. Rudd, T. B. Ervin, and P. R. Staff. The effect of resident work hour regulations on orthopaedic surgical education, Journal of Surgical Orthopaedic Advances, 16 (2007), 19–22. P. J. Gorman, A. H. Meier, C. Rawn, and T. M. Krummel. The future of medical education is no longer blood and guts, it is bits and bytes, The American Journal of Surgery, 180 (2000), 353–356. P. A. Heng, C. Y. Cheng, T. T. Wong, Y. Xu, Y. P. Chui, and S. K. Tso. A virtual reality training system for knee arthroscopic surgery, IEEE Transactions on Information Technology in Biomedicine, 8 (2004), 217–227.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-261

261

Influence of Metal Artifacts on the Creation of Individual 3D Cranio-mandibular Models a

Shintaro KASAMAa, Takumi OGAWAa, Tomoko IKAWAa, Yuko SHIGETA , a

a

b

Shinya HIRAI ,Shunji FUKUSHIMA , Asaki HATTORI , Naoki SUZUKIb. a b

Department of Fixed Prosthodontics Tsurumi University School of Dental Medicine Institute for High Dimensional Medical Imaging, Jikei University School of Medicine

Abstract. In our clinic, the four-dimensional analysis of mandibular movement has mainly been used to diagnose jaw function disorders. In present, we are considering its application for patients with occlusal issues. Consequently, an improvement in system accuracy is required. However, metal artifacts cause distortions in the 3D-cranio-mandibular model construction process, which leads to accuracy concerns. The purpose of this present study was to grasp the accuracy differences caused by the oral metal restorations in reconstructed 3D-craniomandibular models from CT data, and scanned dentition models. The accuracy of the reconstruction was confirmed from comparing the occlusal contacts in VR space and real space. The VR contact areas in the dry skulls without and moderate restoration showed a close similarity to real occlusal contacts. However, the VR contact areas in the dry skull with numerous restorations was not similar to the real contacts. From these results, it is considered that metal artifacts decrease the accuracy of model reconstruction. Keywords. occlusal contacts, 3D-cranio-mandibular model

Introduction In our clinic, the four-dimensional analysis of mandibular movement has mainly been used to diagnose jaw function disorders. In this system, the reconstructed 3D-cranio-mandibular model from CT data was driven by mandibular movement data. In present, we are considering its application for patients with occlusal issues. Consequently, an improvement in system accuracy is required. However, metal artifacts cause distortions in the 3D-craniomandibular model construction process, which leads to accuracy concerns. Therefore, the scanned dentition model data was combined to CT data for reconstructing 3D-craniomandibular model. The purpose of this present study was to grasp the accuracy differences caused by the presence of oral metal restorations in reconstructed 3D-cranio-mandibular models from CT data, and scanned dentition models. Materials and Methods The subjects were three adult dry skulls. Skull A was complete and had healthy dentitions; skull B contained one metal crown and had three missing teeth; skull C contained six metal crowns, six metal inlays and one missing tooth.

262

S. Kasama et al. / Inﬂuence of Metal Artifacts on the Creation of 3D Cranio-Mandibular Models

The accuracy of the reconstruction was confirmed from a comparison of the occlusal contacts in VR space and real space. 1.Extracting the Occlusal Contacts from the Inter Occlusal Record in Real Space The upper and lower study models were created by taking an impression form each skull, and were then mounted on an average value articulator. On the articulator, the interocclusal records were taken, and the tooth-positioner was created to decide the relation of position between cranial bone and mandible. The tooth-positioner was used to immobilize the skull’s mandibular position when takig a CT image. Interocclusal records were taken with occlusal recording material, and the condition of the occlusal contacts were identified by the modified transillumimation method1). (Fig.1) Interocclusal records were put on the Filmviwer and were taken via digital camera under (Table 1.). We then extracted data adjacent to the occlusal contact points at a 25,50,75,100,200μm thickness on the calibration curve via image analysis software (Amira 3.1.1 Mercury Computer Systems). (Fig 2.) Table1. Parameter and setting Degital Camera:FinePixS2Pro.FUJIFILM Lens:MedicalNIKKOR120mm F4.NIKON Shutter speed

1/20sec

ISO

200

Diaphragm Distance between light and object Light viewer (Average brightness)

f11 550mm 1400±300cd/m2

Figure 1.Modified transillumination method g

Figure 2.Calibration curve

2. Evaluating the Reconstruction of the Occlusal Contacts in Virtual Reality Space CT were taken via Arphard CBCT (Asahi Roentgen IND.CO.) for the dry skulls. CT data sets were automatically segmented based on Hounsfield unit, then the 3D-skull model were reconstructed using Amira3.1.1.The dentition study models were scanned by 3D scanner (Optorace,Shofu.Inc.). The 3D dentition models were reconstructed from scanned surface data via 3D-modeling software (VR Mesh,VisualGrid Company) Subsequently, we tried to supplement the data of the dentition region on CT data with

S. Kasama et al. / Inﬂuence of Metal Artifacts on the Creation of 3D Cranio-Mandibular Models

263

3D dentition data2). These data were combined with Interactive Closest Point algorithm2) Rectangular solids were put between the upper and lower dentition as a recording material for the occlusal contacts. In VR space, an occlusal record was created to remove the dentition data from the rectangle blocks via boolean operation (VR occlusal record) with modeling software (rapidform 2006,INUS Technology Inc) (Fig3). Finally, the occlusal records that had been taken in real and VR space were compared.

Results and Discussion The occlusal contacts could be reproduced in virtual reality space to supplement the CT data in the scanned dentition data. Moreover, we could evaluate the reproducibility of the occlusal contacts in VR space. The reconstructed occlusal contacts in Skull A and B showed a close similarity to the 200μm thickness adjacent to the occlusal contacts in real space. In this study we could quantify and visually evaluate the replication of the occlusal contacts in virtual reality space, using the “VR occlusal record”. The VR contact areas in Skull A and B show a close similarity to the actual points. On the other hand, the results from Skull C turned out to be different. In the case where there are many metal artifacts present, it would appear that they are extremely disruptive to image capturing. As a result, it is difficult to register CT data and the scanned dentition data. Thus, a need for reference points may arise when encountering 3D dentition models like skull C.

Figure 3. Comparison of occlusal contacts a.Extracted occlusal contacts in real space. b.VR occlusal record.

References [1] R. DeLonga, S. Knorra, G.C. Andersonb, J. Hodgesc, M.R. Pintadoa. Accuracy of contacts calculated from 3D images of occlusal surfaces,J Dent 35(6)(2007),528-534. [2] Ikawa T, Ogawa T, Shigeta Y, Fukushima S, Hattori A, Suzuki N. The reproduction of high precision 3D maxillofacial reconstruction models,Stud Health Technol Inform 142(2009), 125-7.Published by IOS Press.

264

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-264

Web-based Stereoscopic Visualization for the Global Anatomy Classroom Mathias KASPARa,b, Fred DECHb,c, Nigel M. PARSADb,c and Jonathan C. SILVERSTEIN b,c,d,1 a Department of Medical Informatics, University of Goettingen, Goettingen, Germany b Computation Institute, University of Chicago, Chicago IL, USA c Department of Surgery, University of Chicago, Chicago IL, USA d Department of Radiology, University of Chicago, Chicago IL, USA

Abstract. Many projects have focused on the improvement of virtual education. We have contributed with the global virtual anatomy course for teaching students in multiple locations with stereoscopic volume rendering, audio/video conferencing and additional materials. This year we focused on further simplifying the deployment of the classroom by using the new collaborative and web-based visualization system CoWebViz, to transfer stereoscopic visualization to the classrooms. Besides the necessary hardware installations for stereoscopy, only a web browser is necessary to view and to interact with the remote 3D stereo visualization. This system proved stable, gave higher quality images and increased ease of deployment. Its success within our classroom at the University of Chicago and Cardiff University has motivated us to continue CoWebViz development. Keywords. Virtual Reality, Web-enabling, Volume Rendering, Collaborative stereo Visualization, Education, Anatomy

1. Background Technological development around a virtual classroom can typically be grouped into two categories; (1.) Immersive sharing of the actual classroom session in real-time and (2.) Sharing primarily of course materials, which may include self-educational tools. The first category is achieved by streaming audio/video data via a unicast participant-to-participant (e.g. with H.323 supporting videoconference systems) or via a multicast group-to-group network topology (e.g. with vic/rat, http://mediatools.cs.ucl.ac.uk). Sometimes these tools come together with additional collaborative functionality such as, sharing a web-browser, screen or whiteboard (e.g. Access Grid, http://www.accessgrid.org, and EVO, http://evo.caltech.edu). These technologies may also be used to share 3D stereoscopic visualizations, which is especially interesting in a human anatomy class. Other solutions use special screen sharing applications (VNC or Virtual Network Computing) or, especially in the medical environment, specialized software implementations that are based on client side (e.g. Chili Digital Radiology, http://www.chili-radiology.com/en) or server side rendering [1]. However, specialized software, which commonly entails additional 1

Corresponding Author: Jonathan C. Silverstein, University of Chicago, Computation Institute, 5735 South Ellis Avenue, Chicago, IL 60637; E-mail: [email protected]

M. Kaspar et al. / Web-Based Stereoscopic Visualization for the Global Anatomy Classroom

265

specific hardware requirements, often has to be deployed at every participant’s site in order to share the visualization. The second category includes various web-based and readily accessible viewers such as e-textbooks that provide, among other things, images and video. More sophisticated viewers support learning by providing additional features including multiple image types, varying levels of detail and zoom [2] or multiple viewpoints for 3D objects [3,4]. Such tools are easily accessible with a web browser, making them deployable almost everywhere. A long-term goal of our global immersive virtual anatomy class is to make the first category of software as easy accessible as the second category, with no client software other than a web browser. We have taught the class [5] for the past 5 years using multilocation, multi-direction and multi-stream sharing of video, audio, desktop applications and cluster based stereo volume rendering. Access Grid was the classroom’s foundation as it provided stability and performance [6]. Nevertheless, this year we introduced CoWebViz, a new web-based collaborative visualization system to further close the gap between the aforementioned categories. CoWebViz utilizes only a web browser with no added software (e.g. no Java, Flash or plugins) as a client, for sharing 3D stereoscopic visualizations with improved quality and accessibility.

2. Methods & Materials The global anatomy class’s technical system provides two distinct functionalities: 3D stereoscopic volume rendering as well as the distribution of medical imaging data and collaborative audio/video conference capability. Both functionalities have been provided by Access Grid including a specialized vic-based distributed visualization client for MedVolViz [6]. For the spring 2010 class, we developed the web-based 3D stereo visualization system CoWebViz to replace the former 3D stereoscopic streaming component. The current overall architecture is described in the following paragraphs, the pre-existing parts in summary and the newly developed part in more detail. The architecture is also illustrated in Figure 1. We describe the current state between the original class’s architecture (using thick clients) and the replacement of all client components by a web browser. Access Grid (http://www.accessgrid.org) is a group-to-group video and audio conference system, which utilizes the multicast tools vic and rat as well as the Access Grid venue server to create virtual venues between the participating classrooms (currently two). An Access Grid node is deployed in each classroom, connected to one venue server and session. Attached are cameras, microphones and speakers. We use Access Grid’s shared browser to share online 2D medical illustrations. This browser and the participant’s videos are projected on our campus on a 16ft x 9ft screen. Real-time multisite immersive stereoscopic volume visualization is the primary classroom tool for our virtual anatomical education. It is created by MedVolViz (Medical Volume Visualization), a high performance, distributed, parallel-processing, volume rendering software that utilizes Argonne National Laboratory’s MPICH2-based volume rendering library (vl3). MedVolViz’s input data is unprocessed and anonymized computed tomography (CT) image datasets in DICOM format. The software runs on a nine-node visualization cluster where eight slave nodes individually render subsets of the input data. Of the eight, four nodes render each eye’s perspective

266

M. Kaspar et al. / Web-Based Stereoscopic Visualization for the Global Anatomy Classroom

Figure 1. The classroom’s architecture. The rounded rectangles show the distributed physical locations of class infrastructure components. The classroom at the University of Chicago and the Visualization Cluster are located at the campus of the University of Chicago. The Access Grid venue server is located at the Argonne National Laboratories. Classrooms 2 and 3 were in Cardiff and Göttingen (in tests).

respectively. These partial renderings are then assembled on the cluster’s head node and displayed locally on its X-Server. The assembled perspectives can be simultaneously streamed over the Internet from MedVolViz to a vic-based client. Streaming options include single, stereoscopic side-by-side and anaglyph views. Volume rendering is done on each slave node via commoditized hardware utilizing recent generation, consumer grade graphics cards. The visualization cluster’s hardware was upgraded before the spring class in 2010. Each of the nine nodes now has an Intel Core i7-920 quad-core processor with 6GB of DDR3 1066 RAM and two NVIDIA 275GTX GPU’s running in SLI mode. CoWebViz is a collaborative, interactive and shared visualization system with a standard web browser as client. It was developed as a remote web-based frontend for visualization applications. Currently three different visualization modes are supported: single, anaglyph and stereo. A screenshot of the stereo mode on the client side is shown in Figure 2. CoWebViz is a server application that runs on the head node. It is written in C++ with the following library dependencies: Pion Network Library, Boost C++ Libraries, Xlib, libjpeg-turbo and FFmpeg. On start up it is directly connected to the head node’s X-Server where it grabs the exact geometry of MedVolViz’s locally displayed visualization window and streams it consecutively frame by frame. Internally,

M. Kaspar et al. / Web-Based Stereoscopic Visualization for the Global Anatomy Classroom

267

each image is processed by means handling stereo frames, resizing and JPEG (Joint Photographic Experts Group) compression before being sent to an internal HTTP server. There each eye’s frame is written to a distinct TCP port in the format of a “Motion JPEG” (MJPEG) stream. A second HTTP server provides additional services: (1.) A file service to access files on a server directory (e.g. HTML files and JavaScript libraries), (2.) A visualization service (see Figure 2), (3.) A visualization control service (same as 2, but no visualization) and (4.) An event input service. Using a second HTTP server improves the parallel processing of the MJPEG streaming and the event handling. On the client side, giving one URL (Uniform Resource Locator) for each eye’s view to a browser window opens that view’s visualization. Each URL exposes one web page, which binds the corresponding MJPEG stream and aligns it to the center of the page. A layer above the MJPEG stream is added that actively obtains mouse and keyboard events with JavaScript. Every event is sent to the event input service on the server side as soon as it occurs. All events are then converted to an internal format before they are sent to the event output module and with it to the MedVolViz window. Each classroom has a polarization preserving silver screen (on our campus: a 10ft x 10ft screen) with two aligned projectors. Each is equipped with a circular polarization filter and is connected to the video output on a single desktop. The left and right eye visualization views are opened on the appropriate projector’s desktop screen. Naturally, the system also works with new consumer grade stereoscopic LCD and LED displays.

3. Results In 2010, the class was held locally at the University of Chicago and virtually to Cardiff University, in Wales. Thus far, the infrastructure using Access Grid for conference functionality and CoWebViz for the shared visualization is stable and the whole classroom procedure is established. With the running CoWebViz server application, the stereoscopic visualization is started on the client by opening two URL’s, one for the left and one for the right eye’s view, each in one browser window. Both windows are optimally set to full screen mode, with one display connected to one projector. While alignment of both projectors is obligatory using any stereoscopic visualization tool, CoWebViz does the alignment of both visualization views automatically on the screen. Thus, it is very simple to use the visualization on every computer with most operating systems and a good network connection without the issue of having to deploy a specially compiled application. CoWebViz works best on Firefox 3.5, Safari 4/5 and Google Chrome 5.0. To gain the best available configuration using CoWebViz, with high quality images and as many frames as possible, for both classrooms, we experimented with different configurations during the first sessions. After this initial period, we used two instances of CoWebViz. One instance served the classroom on our campus within the same local network as the cluster, which was configured for a resolution of 1024x768 and a JPEG quality of 85 on a scale of 0 to 100. The second instance served Cardiff with a resolution of 512x384 and a JPEG quality of 65. The lower resolution was resized to 1024x768 on the client side. With the optimized configurations, we had very fluid and high quality interactive visualization at the University of Chicago with around 10 to 13 frames per second (fps). Even with the browser scaling the images to 1024x768, the empirical opinion among our Cardiff collaborators was that CoWebViz

268

M. Kaspar et al. / Web-Based Stereoscopic Visualization for the Global Anatomy Classroom

Figure 2. Two screenshots of a CoWebViz browser window. The upper and lower window shows the left and right eye view, respectively. After typing the right URL into a web browser, the visualization is useable by interactive mouse and keyboard manipulation.

M. Kaspar et al. / Web-Based Stereoscopic Visualization for the Global Anatomy Classroom

269

gave a less compressed (in terms of artifact and colorization) visualization as compared to MedVolViz running in Access Grid specialized vic client with H.261 streams. During class, the visualization is normally manipulated by the lecturer. Therefore, we analyzed CoWebViz within a test session, where the visualization was manipulated by participants in Chicago, Cardiff and Göttingen. The visualization was streamed from a visualization cluster in Chicago sending two 1024x768 resolution eye views with a JPEG quality of 85 to the Chicago campus and sending two times a resolution of 2x512x384 with a JPEG quality of 65 to Europe. In each case, the visualization was fluid, but had a small, non-disruptive lag between the left and right eye views in Cardiff and Göttingen. However, while using the above configuration, interactive manipulation from Europe lagged, with approximately 0.3 to 2 seconds elapsing between the time a command was sent and the time an updated visualization was received. The networks round-trip time was approximately 133ms and 124ms to Cardiff and Göttingen, respectively. Whereas the Chicago classroom had a server connection with a round-trip time of approximately 0.9ms. The interactivity was improved considerably using lower resolutions and quality settings. In short, we learned the following: (1.) The frame rate with MJPEG streaming is sufficient to view the visualization; (2.) The lag among Cardiff, Göttingen and Chicago requires it to be used with interactivity at lower quality settings; (3.) CoWebViz needs more event-handling improvement in order to use it collaboratively with more than two parties. CoWebViz solved a major issue we had while using the vic-based client. The vic video streaming tool only supports streaming in CIF resolution (352x288). In order to stream at higher resolution, the visualization had to be divided into several streams on the server side and combined on the client side. Thus, possible resolutions were multiples of CIF. Using this configuration, we had sometimes an asynchronized 4-tiled view on the client. This happened whenever the stream’s frames traverse the network with different speeds, which was particularly disruptive during an interactive visualization phase. Using CoWebViz, we now have much finer configuration granularities with respect to resolution and image quality then we had while using vic (CoWebViz allows streaming with any resolution and JPEG quality). Also, the CoWebViz client’s colorization more accurately mimics the server’s on the GeoWall using JPEG compression. It should be noted that the usage of “Motion JPEG” also has a drawback. Its compression ratio is very low compared to other video compression standards. And this leads to higher bandwidth needs. During the class, we streamed with an average of 474 KBytes/sec and 58 KBytes/sec per eye view to the University of Chicago and Cardiff, respectively. Thus, we have turned the stereoscopic component of the course into requiring only a web browser on the client side. Still, a thicker client is required for collaborative audio and video conferencing and sharing of web pages.

4. Conclusions Using CoWebViz for streaming the stereoscopic visualization, we made significant progress towards a more simplified and universal client-side classroom deployment. We demonstrated the relative ease in globally streaming stereoscopic renderings with only web browsers.

270

M. Kaspar et al. / Web-Based Stereoscopic Visualization for the Global Anatomy Classroom

CoWebViz is in some respects similar to Virtual Network Computing (VNC), e.g. it uses continuously streamed JPEG images to transfer one’s computer display output to a remote client and it passes mouse as well as keyboard events from the client to the server. However, VNC sends only updates of changed segments from the entire display to the client while CoWebViz sends the entire image each frame. This makes sense, since a modification of the visualization results in a completely new image, but it also leads to higher network bandwidth usage. Use of wavelet or other compression methodologies could be used instead of JPEG compression, but current browsers do not natively support this. Another more sophisticated technique for transferring visualization to the browser would be to use efficient state-of-the-art video streaming standards with inter-frame and in the case of stereoscopic data even inter-view compression. However, we encountered issues controlling the video buffering within the browser while using HTML5 video streaming in real-time and do not want to get the disadvantages of browser added software (e.g. browser plugins or Java). Using CoWebViz, we had more configuration possibilities in terms of quality settings and resolutions then we had while using the vic-based tool. We took advantage of this greater flexibility by rendering and streaming the highest resolution and quality visualizations to our classrooms since the introduction of the course. Our future work is to make all components of the global anatomy classroom work in a web-browser.

Acknowledgment The authors would like to thank Mike W. Daley from the Cardiff School of Computer Science and Informatics of Cardiff University and Benjamin Löhnhardt from the Department of Medical Informatics of the University of Göttingen for their help and comments regarding CoWebViz. This work was supported by a fellowship within the Ph.D. Program of the German Academic Exchange Service (DAAD).

References [1] F. Lamberti, A. Sanna. A streaming-based solution for remote visualization of 3D graphics on mobile devices, IEEE Trans Vis Comput Graph, 2007, 13(2):247–60. [2] P. J. O’Byrne, A. Patry, J. A. Carnegie. The development of interactive online learning tools for the study of anatomy, Med Teach, 2008, 30(8):e260–71. [3] C. Silén, S. Wirell, J. Kvist, E. Nylander, O. Smedby. Advanced 3d visualization in student-centred medical education, Med Teach, Jun 2008, 30(5):e115–24. [4] H. Petersson, D. Sinkvist, C. Wang, O. Smedby. Web-based interactive 3d visualization as a tool for improved anatomy learning, Anat Sci Educ, Mar 2009, 2(2):61–8,. [5] J. C. Silverstein, C. Walsh, F. Dech, E. Olson, M. E. Papka, N. Parsad, R. Stevens. Immersive virtual anatomy course using a cluster of volume visualization machines and passive stereo, Stud Health Technol Inform, 2007, 125:439–44. Published by IOS Press. [6] J. C. Silverstein, C. Walsh, F. Dech, E. Olson, M. E. Papka, N. Parsad, R. Stevens. Multi-parallel open technology to enable collaborative volume visualization: how to create global immersive virtual anatomy classrooms, Stud Health Technol Inform, 2008, 132:463–8. Published by IOS Press.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-271

271

Expanding the Use of Simulators as Assessment Tools: The New Pop Quiz Abby R. KAYE, Lawrence H. SALUD, MS, Zachary B. DOMONT, MD, Katherine BLOSSFIELD IANNITELLI, MD, MS and Carla M. PUGH1, MD, PhD Northwestern University Feinberg School of Medicine, Department of Surgery Abstract. This study introduces a novel way to implement simulation in medical education. We investigated the feasibility of integrating a newly developed breast examination simulator into a breast exam technique lecture while also collecting detailed data on medical students’ breast exam skills. Results indicate that it is feasible to integrate simulation technology into the classroom environment and collect detailed performance data that can be analyzed and used for skills assessment. Keywords. Simulation, medical student education, clinical examination

Introduction Simulation, as a training tool, is most often used within the context of a simulation laboratory [1, 2]. A major focus of this study was to investigate the feasibility of using newly developed simulators to facilitate a breast lecture while also collecting detailed data on students’ breast exam skills. 1. Methods We designed breast examination simulators that are reconfigurable to reflect various clinical findings and are instrumented internally with sensors to detect palpation characteristics. The sensors interface with data acquisition software on a connected laptop allowing us to collect and store performance data. For this study, three simulators represented three different clinical presentations. The Station A model had a hard mass in the upper inner quadrant (UIQ). The Station B model had no palpable masses. The Station C model had a cystic mass in the upper outer quadrant (UOQ) and a firm mass in the lower inner quadrant (LIQ), Figure 1. Prior to classroom implementation, a convenience sample of experienced clinicians (N=100) attending a breast cancer meeting evaluated the simulators as teaching tools. These simulators were further modified based on these evaluations.

Figure 1. Stations A, B, and C configurations. Station A has hard mass in UIQ. Station B has no palpable masses. Station C has cystic mass in UOQ and firm mass in LIQ. 1 Corresponding Author: Carla M. Pugh, MD, PhD, Northwestern University Feinberg School of Medicine, 676 N. St. Clair Street - Suite 650, Chicago, IL 60611-2908; E-mail: [email protected].

272

A.R. Kaye et al. / Expanding the Use of Simulators as Assessment Tools: The New Pop Quiz

Surgery clerkship students (N=63) performed a breast exam on each simulator and documented their findings while their performance data was collected directly from the simulators and stored. After the students’ performed the examinations, faculty members demonstrated examinations on the simulators while they discussed their findings and the work-up of a breast mass. Faculty presenters modified their traditional surgery clerkship lectures by splitting their information into a pre-simulation lecture and a post-simulation demonstration and discussion. This allowed time in the middle of the allotted clerkship session for the students to test their breast examination and documentation skills on three different breast simulators.

2. Results On a 5-point Likert Scale (1=not useful, 5=extremely useful), clinicians rated the initial simulator configurations useful for teaching (3/5). After upgrading the simulators, they were successfully integrated into the lecture series without increasing the allotted time period of 120 minutes. Detailed performance data were collected and revealed that 85% (52/61) of students correctly identified, localized and characterized a 2 cm hard mass in the UIQ of Simulator A. For Simulator B, 46.8% (29/62) correctly noted that there were no palpable masses. For Simulator C, 65.6% (40/61) correctly identified, localized and characterized a 3 cm cystic mass in the UOQ and a 2 cm firm mass in the LIQ. Using the sensor data collected from each student, we compared student examination techniques (up-down vs. circular techniques) and performance across the three simulators. In comparing examination techniques, there were significant (p < 0.05) differences in the time spent examining each simulator, amount of breast tissue palpated, and the depth and frequency of palpations. Figure 2 shows a breast exam technique diagram, color-coded sensor map, and the resulting pressure versus time graph generated for the up-down examination technique (“the strip method”). Figure 3 shows a breast technique diagram, color-coded sensor map, and the resulting pressure versus time graph generated for the circular breast examination technique.

Figure 2. (A) The up-down breast examination technique. (B) Color-coded sensor placement map. (C) Pressure vs. time for up-down breast examination technique.

A.R. Kaye et al. / Expanding the Use of Simulators as Assessment Tools: The New Pop Quiz

273

Figure 3. (A) The circular examination technique. (B) Color-coded sensor placement map. (C) Pressure vs. time for circular breast examination technique.

3. Conclusion It is feasible to integrate simulation technology into a standard clerkship lecture and to collect detailed performance data on medical students’ clinical skills that can be stored for offline analysis. This type of integration represents a formative assessment of clinical skill and can be used in many different ways to guide medical student training and assessment.

References [1] [2]

McGaghie, WC, Issenberg, SB, Petrusa, ER, & Scalese, RJ. (2010). A Critical review of simulationbased medical education research: 2003–2009. Medical Education, 44, 50-63. Hammoud, MM, Nuthalapaty, FS, Goepfert, AR, Casey, PM, & Emmons, S. (2008). To the Point: medical education review of the role of simulators in surgical training. American Journal of Obstetrics & Gynecology,199(4), 338-343

274

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-274

Validation of Robotic Surgery Simulator (RoSS) Thenkurussi KESAVADASa,1, Andrew STEGEMANNb, Gughan SATHYASEELANa, Ashirwad CHOWRIAPPAa, Govindarajan SRIMATHVEERAVALLIa and Stéfanie SEIXAS-MIKELUS b , Rameella CHANDRASEKHAR b, Gregory WILDING a, Khurshid GURUb a University at Buffalo, Buffalo, NY b Roswell Park Cancer Institute, Buffalo, NY

Abstract. Recent growth of daVinciTM Robotic Surgical System as a minimally invasive surgery tool has led to a call for better training of future surgeons. In this paper, a new virtual reality simulator, called RoSS is presented. Initial results from two studies – face and content validity, are very encouraging. 90% of the cohort of expert robotic surgeons felt that the simulator was excellent or somewhat close to the touch and feel of the daVinci console. Content validity of the simulator received 90% approval in some cases. These studies demonstrate that RoSS has the potential of becoming an important training tool for the daVinci surgical robot. Keywords: Robot assisted surgery, virtual training, face validity, content validity

Introduction The daVinci™ Surgical System (dVSS) offers all the technological features that an experienced surgeon needs to ensure equivalent or superior outcomes to conventional/laparoscopic surgery. Despite this, there has been criticism and doubt regarding the clinical outcomes of robotic surgery and the competency of surgeons to operate these consoles [1]. The training program [2] offered by Intuitive Surgical is generally limited and the Halstedian model of “learning by doing” fails because of the limited availability for necessary training with the dVSS. Hence, a ‘flight simulator’ for robot-assisted surgery can be beneficial for improving outcome on dVSS. We believe that current challenges can be met with advances in simulation and virtual reality technologies. We propose the development of a trainer that would closely simulate all key components of dVSS as a way of achieving this training. To prove its usefulness, such a system must undergo multi-faceted validation that would evaluate it for face validity of the simulated physical interface and content validity of the simulated training software and tasks. This paper reports the construction of one such system, called RoSS, and its subsequent evaluation by leading robotic surgeons.

1. Methods & Materials The Robotic Surgical Simulator (RoSS™), developed by the University at Buffalo and Roswell Park Cancer Institute, is a training platform to familiarize and prepare surgeon

T. Kesavadas et al. / Validation of Robotic Surgery Simulator (RoSS)

275

trainees to use dVSS. RoSS is marketed by Simulated Surgical Systems LLC, a spinoff company from the two institutions. RoSS consists of a hardware interface that closely replicates the kinematics and feel of working on the dVSS console, and virtual reality software that simulates the operation of the patient side slave. RoSS was developed over multiple design iterations and the current version is presented in Figure 1. The figure compares each component of RoSS console with the corresponding key feature of the dVSS console. The look and feel of the system was validated through a face validity study that recruited thirty surgeons of varying robotic experience. The surgeons worked on simulated tasks on RoSS and evaluated its overall likeness to working on dVSS. They also rated individual components of the hardware interface [3].

Figure 1. Comparison of RoSS and dVSS (Courtesy – Simulated Surgical Systems LLC and Intuitive Surgical, Inc.)

Figure 2. Sample of tasks attempted by cohort of surgeons for evaluation

The training courses currently available on RoSS are (i) Orientation module, which introduced the trainee to the basic operation of dVSS and its control of its hardware components; (ii) Motor skills module, that honed the fine motor control of the trainee in conjunction with the robotic instruments and (iii & iv) Surgical skills module, which trained for basic and elementary surgical skills such as safe blunt dissection of tissue and suture handling. The RoSS software includes a database and user management system that records up to eight key metrics (time, tool-tool collisions, tool flight path etc.) for every training task undertaken on the system. These metrics are then harvested to establish gold standards for every training task in the software. This curriculum of training tasks was then evaluated (content validity) using a second cohort of expert robotic surgeons (n=42) [4]. The surgeons tried four tasks (Figure 2) and evaluated them for their didactic potential and training efficacy.

2. Results An analysis of the cohort from the face validity study showed that eighty percent of participants had at least 4 years of experience with robotic surgery and 77% had performed an average of 340 cases on the dVSS as primary console surgeons. Study subjects responded that RoSS was realistically close to the dVSS console in terms of virtual simulation and instrumentation. Fifty-two percent of respondents rated RoSS somewhat close and 45% rated RoSS very close to the dVSS console. For the similarity in movement of arms and tool control, 43% rated it somewhat close, 40% rated it very close, and 7% felt that it was just like the dVSS. Compared to the dVSS, camera

276

T. Kesavadas et al. / Validation of Robotic Surgery Simulator (RoSS)

movement and clutch functions were rated as follows - somewhat close (57%), and very close (32%) (Figure 3). From the 42 subjects participating in the content validity study, eighty percent had an average of 4 years of experience with robot-assisted surgery with the expert group of 17 (41%) surgeons averaging 881 (160-2200) robot-assisted cases. Considering the training tasks, this expert group rated the “clutch control training module” as a good (71%) or excellent (29%) teaching tool. Seventy-eight percent rated the “ball placement motor skill” task as good or excellent, while 22% rated it as poor. Twenty-seven percent rated the “needle removal” task as an excellent teaching tool, while 60% rated it good and 13% poor. Ninety-one percent rated the “fourth arm tissue removal” task as good or excellent (Figure 4). Ninety-four percent responded that RoSS would be useful for training purposes. Eighty-eight percent felt that RoSS would be a useful training and testing format before operating room experience for residents and 79% responded that RoSS could be used for certifying trainees in robotic surgery.

Figure 3. Results from face validity study [3]

Figure 4. Results from content validity study [4]

3. Discussion The results from the content and face validity studies indicate that RoSS is capable of closely simulating activities on the dVSS. The experts opined that this has the potential of becoming an evaluation system, which can be used for certification of robotic surgeons. Looking ahead, additional studies must be performed using RoSS to establish construct and predictive validity of the system. Disclosure T. Kesavadas and K. Guru are co-founders of Simulated Surgical Systems LLC. References [1] The New York Times, Results Unproven, Robotic Surgery Wins Converts, http://www.nytimes.com/2010/02/14/health/14robot.html, accessed July 2010. [2] Intuitive Surgical, http://www.intuitivesurgical.com/services/producttraining/process.aspx, (July 2010.) [3] Seixas-Mikelus S., Kesavadas T., Srimathveeravalli G., Chandrasekhar R., Wilding G., Guru K., Face Validation of Novel Robotic Surgical Simulator, Urology (Gold), 76(2):357-60, Aug 2010. [4] Seixas-Mikelus S., Stegemann A., Kesavadas T., Srimathveeravalli G., Sathyaseelan G., Chandrasekhar R., Wilding G., Peabody J., Guru K., Content Validation of a Novel Robotic Surgical Simulator, to appear in British Journal of Urology International (online published Oct 2, 2010).

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-277

277

Practical Methods for Designing Medical Training Simulators Thomas KNOTT a,1 , Sebastian ULLRICH a and Torsten KUHLEN a a RWTH Aachen University, Germany Abstract. We reviewed several approaches in literature used in the design process of medical training simulators. We have collected a set of useful practical methods which should help to eﬃciently derive a well-founded design for a speciﬁc surgical intervention in a structured manner. Keywords. Medical Simulation, Medical Training, Survey, Task Analysis, Task Decomposition

1. Introduction Training simulators get increasing importance in medical education to provide hands-on practice. There are several simulators enabling trainees to learn entire procedures, isolated steps or surgical core skills [1]. Training applications should be designed to ensure content validity and learning eﬃciency. Therefore, the design process should incorporate a structured task analysis, learning theory, and cognitive sciences. As there are a wide range of methods and theories in these areas, this paper provides a collection of eﬃcient techniques which are tailored for medical training simulator design and could be used by clinicians as well as technicians.

2. Theoretical Background Humans have only a limited attentional capacity [2]. Gallagher et al. [3] describe impacts of this on the learning of complex skills: if a novice trains a complex procedure without having internalized basic skills, most of her attentional resources are occupied for basic psychomotor or spatial tasks. As a result, less or no capacity is left to comprehend instructions or extract additional knowledge. Therefore, the authors propose, that surgeons should train necessary core or basic skills to automatism, before moving on to more complex tasks. It has been shown that simulator training of core skills already improves the trainees performance in real surgery [4]. Nevertheless, it is not suﬃcient to be skilled in subtasks to ensure performance of the encompassing complex task, as the ability to coordinate and integrate the parts has to be learned to [5]. In summary, optimal training envi1 Corresponding

E-Mail Address: [email protected].

278

T. Knott et al. / Practical Methods for Designing Medical Training Simulators

ronments consider basic tasks, core skills as well as complex tasks and eventually the whole procedure.

3. Methods To fulﬁll the above described requirements a design process has to include the following steps: collection of data about the procedure, decomposition of the procedure into sub tasks and analysis which surgical core skills are needed to accomplish them. In the following we will present useful practical methods for these activities. 3.1. Data Collection In case results of task analysis or related descriptions in the literature are missing, insuﬃcient, imprecise or outdated, new data has to be collected. This could be done by consulting subject matter experts or directly observing the actual procedure. For the latter [6] use a clearly predeﬁned procedure in combination with a video annotation software. An inter-observer reliability check is utilized in this process to guarantee consistent and reproducible descriptions. While this procedure is done in a post-processing step, [7] suggest to annotate directly during the intervention. The proposed software, Surgical Workﬂow Editor, enables the recording of interventions and can be used by the surgeon to directly comment the material with information about important decisions, structures or abnormalities. The tool also facilitates the aggregation of information from multiple recorded similar interventions. 3.2. Task Decomposition Cao [6] propose the hierarchical decomposition as framework for structuring surgical procedures. An intervention is broken down into surgical meaningful events, which are then further subdivided into action-sequences, and hereafter into basic surgical tasks. Finally, a Motion analysis [8] could be used to analyse which basic motions are actually done to solve a basic surgical task. Mnchenberg et al. [9] also use increasing levels of detail to structure their descriptions, but instead of a serial decomposition they utilize instruction graphs, which allow to model alternative operation courses. As intervention descriptions should be unambiguous and serve as tool for communication, used terms should be clear and well-deﬁned. For that purpose, Heinrichs et al. [10] draw on historical deﬁnitions of fundamental surgical manipulations to declare a structured vocabulary for the purpose of describing surgical interventions. 3.3. Task Analysis Grunwald et al. [11] describe how cognitive task analysis can be used during the development of surgical simulators. The technique helps to acquire knowledge about, e.g., task goals or performance standards. The applied cognitive task anal-

T. Knott et al. / Practical Methods for Designing Medical Training Simulators

279

Figure 1. Structured overview of practical methods used in the design process of medical training simulators.

ysis [12] was especially developed and validated to help practitioners who are not trained in cognitive psychology. The technique provides interview methods as well as templates for knowledge representation and is considered to be not resource intensive.

4. Conclusions/Discussion We believe that this collection of practical methods can help in the design process of medical training simulators. Future work will incorporate case studies to validate the approaches and further extend the overview.

References [1] Liu A, Tendick F, Cleary K, Kaufmann C. A survey of surgical simulation: applications, technology, and education. Presence: Teleoperators & Virtual Environments. 2003;12(6):599–614. [2] Broadbent D. Selective and Control Processes. Cognition. 1981;10:53–58. [3] Gallagher AG, Ritter EM, Champion H, Higgins G, Fried MP, Moses G, Smith CD, Satava RM. Virtual reality simulation for the operating room: proﬁciency-based training as a paradigm shift in surgical skills training. Annals of surgery. 2005 Feb;241(2):364–72. [4] Seymour NE, Gallagher AG, Roman SA, O’Brien MK, Bansal VK, Andersen DK, Satava RM. Virtual Reality Training Improves Operating Room Performance. Annals of surgery. 2002 Oct;236(4):458–63; discussion 463–4. [5] Merri¨ enboer JV, Clark R, De M. Blueprints for complex learning: The 4C/ID-model. Educational Technology. 2002;50(2):39–64. [6] Cao CGL, MacKenzie C, Ibbotson J. Hierarchical decomposition of laparoscopic procedures. Medicine Meets Virtual Reality. 1999;p. 83–89. [7] Burgert O, Neumuth T, Audette M, Possneck A, Mayoral R, Dietz A, Meixensberger J, Trantakis C. Requirement speciﬁcation for surgical simulation systems with surgical workﬂows. Studies in Health Technology and Informatics. 2007;125:58–63. [8] Cao C, MacKenzie C, Payandeh S. Task and motion analyses in endoscopic surgery. Proceedings ASME Dynamic. 1996;p. 583–590. [9] M¨ unchenberg J, W¨ orn H, Brief J, Hassfeld S, M¨ uhling J. A pattern catalogue of surgical interventions for computer-supported operation planning. Studies in Health Technology and Informatics. 2000;70:227–229. [10] Heinrichs WL, Srivastava S, Montgomery K, Dev P. The Fundamental Manipulations of Surgery: A Structured Vocabulary for Designing Surgical Curricula and Simulators. The Journal of the American Associacion of Gynecologic Laparoscopists. 2004;11(4):450–456. [11] Grunwald T, Clark D, Fisher SS, McLaughlin M, Narayanan S, Piepol D. Using cognitive task analysis to facilitate collaboration in development of simulator to accelerate surgical training. Studies in health technology and informatics. 2004 Jan;98:114–20. [12] Militello L, Hutton R. Applied Cognitive Task Analysis (ACTA): A practitioner’s toolkit for understanding cognitive task demands. Task analysis. 2000;41(11):90–113.

280

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-280

The Minnesota Pelvic Trainer: A Hybrid VR/Physical Pelvis for Providing Virtual Mentorship Vamsi KONCHADAa,1 Yunhe SHENb,c Dan BURKEc Omer B. ARGUNb,c Anthony WEINHAUSd Arthur G. ERDMANe,f and Robert M. SWEETb,c a Department of Computer Science and Engineering, University of Minnesota b Department of Urologic Surgery, Medical School, University of Minnesota c Center for Research in Education and Simulation Technologies, Univ. of Minnesota d Department of Integrative Biology and Pathology, University of Minnesota e Department of Mechanical Engineering, University of Minnesota f Medical Devices Center, University of Minnesota

Abstract. Obtaining accurate understanding of three dimensional structures and their relationships is important in learning human anatomy. To leverage the learning advantages of using both physical and virtual models, we built a hybrid platform consisting of virtual and mannequin pelvis, motion tracking interface, anatomy and pathology knowledge base. The virtual mentorship concept is to allow learners to conveniently manipulate and explore the virtual pelvic structures through the mannequin model and VR interface, and practice on anatomy identification tasks and pathology quizzes more intuitively and interactively than in a traditional self-study classroom, and to reduce the demands of access to dissection lab or wet lab. Keywords. Anatomy, motion tracking, pelvis, virtual reality.

Introduction Medical virtual reality (VR) applications improve medical cognition in a highly engaging environment [1]. Traditionally, anatomical knowledge is first imparted through textbooks and plastic models, and then through cadaver dissection guided by a tutor. Recently, computer-based visualization and animations have improved the ability to depict anatomical complexities [2][3]. The implementation of VR-based applications for education and training will fundamentally change and improve the way to learn human anatomy. The objective of this work is to create a pelvic trainer as a proof-ofconcept application of the aforementioned virtual mentorship for medical education.

1

Corresponding Author: Vamsi Konchada; Email: [email protected]

V. Konchada et al. / The Minnesota Pelvic Trainer

281

1. Requirements • • • •

Performance: System must be able to track and identify anatomical details in real time (30 frames per second) without occlusion/intermission or noticeable interference from general environment. Usability: Easy for medical community to operate; includes effortless set up, calibration, and intuitive interaction with the system. Cost: Under $5,000 cost to build per unit. Compact profile: Contains multiple units in limited work space.

Figure 1. The Process

2. Methods We captured 3-D CT dataset of a female pelvis and scanned a human pelvic skeleton. Using MIMICS™, we converted the DICOM dataset into a 3D reference mesh. The pelvis mesh model was rebuilt using Maya™, and then textured using Z brush™ and Photoshop™. A plastic model of the structure was printed via Rapid Prototyping [4].We experimented with a variety of optical and magnetic tracking solutions, and chose magnetic tracking for this application. Anatomy identification and quiz software is developed to register user input with the virtual models and give feedback according to a curriculum and knowledge base for teaching pelvic anatomy, physiology, pathophysiology and clinical treatments.

Figure 2. Physical and Virtual interaction of pointing device with bony pelvis (left), virtual pelvis is segmented into different regions and color coded (right)

282

V. Konchada et al. / The Minnesota Pelvic Trainer

3. Results • •

•

Modeling: Figure 1 shows examples of realistic models and textures created for this application. Tracking: Our tests show that magnetic tracking system meets the requirements and is preferred to optical, for an optical camera may encounter occlusion problems in tracking optical markers on the pelvic complex as it is being continuously rotated by a user. Software and the hybrid interface: We successfully completed and integrated the platform whose mannequin and virtual models and graphic user interface (GUI) are shown in Figure 2. An example of anatomy quiz is in Figure 3.

Figure 3. GUI representation of Pelvic anatomy quiz. Quiz application extended to muscles, organs and vessels with bony pelvis as reference

4. Future Work The next step is to integrate more clinically relevant content including orthopedics, obstetrics, urology and general surgery cognition. This application will also be tested in a new web-based online framework design with cost-efficient terminal interfaces.

Acknowledgements This work was supported in part by the Institute for Engineering in Medicine (IEM) at the University of Minnesota. We acknowledge Minnesota Supercomputing Institute (MSI) for visualization facility support.

References [1] M. Jeannerod, Motor Cognition: What Actions Tell the Self, Oxford University Press, 2006. [2] J. D. Brooks, W. Chao, J. Kerr. Male Pelvic Anatomy Reconstructed from The Visible Human Data Set, Journal of Urology (1998), 868-872. [3] K. Hinckley, R. Pausch, J. H. Downs, D. Proffitt, N. Kassell. The props-based interface for neurosurgical visualization, Stud Health Technol Inform (1998), 552-562. Published by IOS Press [4] Rapid Prototyping Machine, Univ. of Minnesota: http://www.me.umn.edu/info/desres/rapid/

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-283

283

Registration Stability of Physical Templates in Hip Surgery Manuela KUNZa,b, John F RUDANb, Gavin CA WOODb, and Randy E ELLISa,b,c,1 a School of Computing, Queen’s University, Kingston, ON, Canada K7L3N6 b Department of Surgery, Queen’s University c Department of Mechanical and Materials Engineering, Queen’s University

Abstract. We tested the registration stability of individualized templates in a consecutive study with 80 patients undergoing hip-resurfacing surgery. These templates physically encode registration and navigation parameters but do not require a computer during the actual surgery. The surgical target was the placement of the femoral guidance pin during hip resurfacing, which is a difficult and highly variable task using conventional instruments. The drill trajectory for the guidance pin of the femoral component was planned on a 3D computer model of the femur derived from a preoperative computed tomography (CT) scan. A surface-matched drilling template was designed to perform mechanical registration on the bone surface and had a hole for the drill guide; the template was created using a rapid prototyping machine. Intraoperatively, the individualized template was positioned on the patient anatomy and the pin was drilled into the femoral neck. The final achieved pin orientation and position were measured using an optoelectronic CTbased navigation system. The measured mean deviation between planned and actual central pin alignment of 0.05° in valgus and 2.8° in anteversion shows that the proposed individualized templates for hip resurfacing have reliable registration. Keywords: Image-Guided Surgery, Patient-Specific Instrument Guides, Hip Resurfacing, Orthopaedic Surgery

Introduction In 1998, Rademacher et al. [1] first described the use of individual templates as an easy-to-use and cost-effective alternative to computer-assisted orthopaedic surgeries. The principle of the individualized templates was to customize surgical templates based on virtual 3D reconstruction of patient-specific bone structures. Small reference areas of these bone structures were integrated into the template. By this means, the planned position and orientation of the tool guide in spatial relation to the bone was stored in a structural way, which could then be reproduced intraoperatively by adjusting the contact faces of the template until an exact fit to the bone was achieved. Other authors subsequently published the results of various applications of individualized templates [24]. Research in the area of individual templates, also known as patient-specific guides, has increased in the last two years (due in part to more accessible prototyping technologies). Advantages of using individualized templates compared to conventional 1

Corresponding Author: Randy E Ellis, School of Computing, Queen’s University, Kingston, ON, Canada K7L 3N6; E-mail: [email protected]

284

M. Kunz et al. / Registration Stability of Physical Templates in Hip Surgery

image guided surgical systems (such as optoelectronic, or electromagnetic navigation) include: elimination of technical equipment in the operating theater; increase of user friendliness; and reduction of cost. On the other hand, the success of the guided surgical action depends on the preoperatively defined structural registration stability of the individualized templates. There is no widely accepted means of intraoperatively verifying the registration, and in the most cases it is “feeling the fit” by the surgeon. Furthermore, there are only limited options to correct the fit of the templates intraoperatively, either by manipulating of the registration surface on the anatomy (e.g., removal of soft tissue and osteophytes), or by manipulating the templates (e.g., breaking or cutting). The mechanical instrument guidance that is integrated in the template also limits the number of surgical actions that can be navigated. In this study, we investigated in a consecutive series of 80 patients, to determine whether individual templates provide sufficient accuracy to be a valuable alternative to optoelectronic navigation systems for hip resurfacing. Hip resurfacing has recently gained greater acceptance as a treatment for osteoarthritis, especially for younger and more active patients [5-7]. There are, however, several disadvantages reported in the literature because of difficult surgical exposure and the technical challenge of the intraoperative procedure [5,7-9]. Surgical errors, such as notching of the femoral neck, tilting of the prosthesis in excess varus, or improper prosthesis seating, can result in early failure of the femoral component. Various studies [10-13] have shown that computer-assisted systems can help to prevent such problems, but come with added complexity, time and cost to the surgery. Individualized templates for placement of the femoral guide pin would be an interesting alternative for this application. However, because of the nature of hip deformities normally seen with patients requiring hip resurfacing, accurate segmentation can be challenging. In many patients the cartilage is damaged and the joint gap partially or fully missing. Furthermore, these hip deformities are often accompanied by osteophytes, which are not always detectable in a CT scan. We developed a method for navigation of guide-pin placement during hip resurfacing using individual templates, aiming to optimize the design of the template to provide stable and precise registration.

1. Methods and Materials We tested the registration stability of individualized templates for the drilling of the femoral central-pin on a consecutive series of 80 patients (28 left, 52 right) undergoing hip resurfacing. Seventeen-two patients were operated with the anterolateral approach, 8 patients with a posterior approach. Articular surface replacement (ASR) components from DePuy (a Johnson&Johnson Company, Warsaw, USA) was used for all patients. One to four weeks prior to surgery a computer tomography (CT) scan of the affected acetabulum and proximal femur was performed. All scans were obtained with a LightSpeed CT (GE Healthcare, Waukesha, USA) in helical mode, with a slice thickness of 2.5mm. 3D virtual surface models of the proximal femur and the acetabulum were created using commercially available software (Materialise, Leuven, Belgium).

M. Kunz et al. / Registration Stability of Physical Templates in Hip Surgery

285

1.1 Preoperative Planning The preoperative planning was performed in two steps: first surgical planning followed by the creation of the individualized template. Surgical Planning: At this stage, the position and orientation of the central-pin was defined and the femoral and acetabular component size chosen. The planning was performed using a CT-based planning system. After loading the femoral and acetabular 3D models into the system, a translucent cylinder was superimposed on the femoral model. To register orientation and diameter of the femoral neck axis, the surgeon modified the position, orientation and size of this cylinder until the optimal alignment between cylinder and femoral neck was achieved. Femoral neck areas outside the cylinder indicated potential notching risk during the reaming process. To verify bone coverage, component position and sizing, virtual models of femoral and acetabular components were superimposed on the femur and acetabulum model. Creation of individualized drill templates: Custom-made software was developed, based on the free visualization library Coin3d (www.coin3d.org). The virtual femoral model and the planned central-pin position and orientation were loaded and displayed to the user. In a semi-automatic procedure, the user planned the location and size of the individualized template. The registration template was constructed using two subcomponents. The first subcomponent was oriented along the femoral neck and head, which ensured stable position and orientation along the neck axis. Because of the segmentation uncertainty in the articular surface of the head, this region was eliminated from the registration surface, as shown in Figure 1. The second subcomponent was a region oriented perpendicular to the neck axis, and positioned on the lateral aspect of the femoral neck; this ensured rotational stability about the neck. To increase the overall stability of the template, this second registration surface enveloped roughly 130° of the lateral femoral neck, with respect to the femoral neck axis (Figure 1). During the planning the user was able to spin the registration region around the neck axis to ensure that it was positioned in areas accessible during the surgical approach. The user also selected height and depth of the registration surfaces to account for the size of the femur. A custom algorithm was used to intersect the two template subcomponents along the femur bone surface. Both template subcomponents were united and a drill-guidance component was automatically attached to the medial side of the template. A guide channel was inserted into this guidance component, which was oriented along the determined central-pin trajectory as shown in Figure 1. After the design of the individualized drilling template was completed, the computer model data was saved in a stereolithographic (STL) format and a physical model of the template was created using a rapid-prototyping machine (dimension SST, Stratasys, Inc., USA). The material used during this 3D printing process was thermo-plastic acrylonitrile butadiene styrene (ABS). Finally, the individualized drill template and the femur plastic model were gassterilized and labeled before being sent to the operating theater.

286

M. Kunz et al. / Registration Stability of Physical Templates in Hip Surgery

Figure 1. Individualized template for hip resurfacing. Left side: Registration surface is constructed from two parts. Part 1 is oriented along the femoral head and neck; Part 2 is perpendicular to the neck axis on the lateral aspect of the neck. This second part enveloped 130 degree around the neck (right side).

1.2 Intraoperative Procedure The surgeon performed a conventional approach, dislocating the hip and removing extensive osteophytes as well as performing any necessary soft-tissue cleaning. After the proximal femur was exposed, the surgeon fitted the individualized template to the corresponding bone surface. The conventional drill sleeve for the central-pin was inserted into the guide channel of the template (Figure 2). The central-pin was then drilled using a conventional power drill, after which the metal drill sleeve and template were removed. The rest of the surgery was performed using the conventional technique.

Figure 2. During the intraoperative procedure the individualized template is registered to the bone surface and a metal drill sleeve is inserted into the guidance part of the template. The central pin is drilled using this drill sleeve as guidance.

M. Kunz et al. / Registration Stability of Physical Templates in Hip Surgery

287

1.3 Analysis To measure the 3D accuracy of the proposed method, we used an optoelectronic CTbased navigation system. A Polaris tracking system (Northern Digital Inc., Waterloo, Canada) was installed in the operating theater. After the central-pin hole was drilled using the individualized template, an optoelectronic marker was attached to the proximal femur. For an initial registration between anatomy and the preoperative virtual femur model a pair-point matching of 3-4 anatomical landmarks was used. This initial estimation was refined by point-to-surface registration of an additional cloud of 12-20 bony points collected from accessible anatomical areas. After successful registration, the central-pin was removed from the femur and a tracked axial pointing device was aligned inside the central-pin hole and its 3D orientation was stored with respect to the coordinate system of the virtual femur model. After this measurement the central-pin was returned in place and the final steps of surgery was performed conventionally. Custom software was developed which allowed determination of the deviation between planned and navigated central-pin alignment. A graphical user interface allowed the user to define the anatomical shaft and neck axis of the femur based on the virtual 3D model. Using these anatomical values, varus/valgus and anteversion/retroversion deviation of the planned and navigated central pin alignment were calculated. Student’s t-test was used to investigate significant differences in mean values, and Levene’s test was used to determine if statistical differences existed in the variances between these two groups. All tests were performed using a significance level of α = 0.05, which is a 95% confidence interval.

2. Results Statistical results used the mean value μ, standard deviation σ, and significance p. For each of the 80 patients, central-pin drilling was performed using individualized templates. We measured a deviation between the planned guide-pin orientation to the final placed pin orientation of μ = 0.05° (σ = 3.3°) in the valgus direction and μ = 2.8° (σ = 5.5°) in anteversion. The entrance position for the central pin was on average μ = 0.47mm (σ = 1.86mm) superior to the planned entrance point and μ = 2.6mm (σ = 3.6mm) anterior. In Figure 3 results are presented according to the selected surgical approach (anterolateral or posterior). Because of the small number of posterior approaches, statistical significance was not achieved. However, the results showed a tendency for the individualized templates to be more retroverted and in valgus for the posterior approach and anteverted for the anterolateral approach. Also the entrance point seemed to be more superior for the posterior approach and more anterior for the anterolateral approach. For anterolateral cases with an alignment error greater than 3° anteversion, we measured on average μ = 4.03mm (σ = 3.4mm) anterior displacement of the entrance point, compared to an average anterior displacement of μ = 1.68mm (σ = 3.0mm) for cases with a smaller alignment error in the anteversion direction. This difference in mean values was statistically significant (p = 0.004).

288

M. Kunz et al. / Registration Stability of Physical Templates in Hip Surgery

Figure 3. Evaluation of position and orientation error for final pin placement with respect to surgical approach. Each box plot shows on the left side the evaluation for the anterolateral approach, and on the right side the posterior approach evaluation.

3. Conclusions Correct femoral component alignment has been identified by many authors as an important factor for long-time success of a hip resurfacing procedure [5, 7-9]. The final alignment and location of the femoral component is guided by a central-pin which is placed into the femoral neck. This makes the correct placement of this pin an essential step for the success of the procedure. In this study, we investigated the accuracy of individualized templates to guide the central-pin placement. In the frontal plane, we found a mean error of 0° and standard deviation of 3.3°, which is comparable to other studies using optoelectronic computerassisted hip resurfacing methods. Specifically, Davis et al.[13] reported an average of 1.7° varus deviation (σ = 2.9°) of the navigated axis compared to the planned orientation using an image-free navigation system on cadavers. Hess et al. [10] reported a mean deviation between planned and postoperatively measured prosthesis-femur angle of 2.6° (±0.89°), where Hodgson et al. [12] measured varus/valgus deviations of 2.5° (σ = 2.2°) between planned and navigated central-pin alignment. Our results show higher error values for translation and angle deviation in the transverse plane compared to the frontal plane, especially for the anterolateral approach. In this approach we found a correlation between an error in anteversion and an anterior displacement of the central pin entrance point at the femoral head. This correlation suggests that the errors were the result of inaccurate registration of the individualized templates in the medial part of the femoral neck and/or parts of the lateral head. To avoid registration errors due to segmentation uncertainties, we eliminated the articular surface of the femoral head from the registration area of the template. However, on the junction between head and neck there are often osteophytes which can increase the chance of registration errors. To avoid such errors, osteophytic regions may also need to be removed from the registration surface.

M. Kunz et al. / Registration Stability of Physical Templates in Hip Surgery

289

To evaluate the intraoperative accuracy of the individualized drilling template we used a CT-based computer-assisted system to measure the outcome of the central-pin alignment. By using this well-established tracking technology, we were able to measure and compare the intraoperative central-pin alignment with the preoperatively planned central-pin alignment in a common frame of reference without the need for a postoperative CT scan. Our results were not affected by variation of projection directions between preoperative and postoperative conventional 2D radiographs. In conclusion, individualized templates for placement of the central-pin during a hip resurfacing surgery provide the surgeon with an accurate instrument, with results comparable to conventional computer-assisted solutions. In contrast to conventional solutions, individualized templates are easy to integrate into the surgical workflow and do not demand additional intraoperative equipment.

Acknowledgments The authors are grateful to Paul St. John for his technical support, and to the surgical and perioperative teams of the Kingston General Hospital, Kingston, ON, Canada for their assistance. This work was supported in part by the Canadian Institutes for Health Research, the Canada Foundation for Innovation, and the Natural Sciences and Engineering Research Council of Canada.

References [1]

[2]

[3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13]

K. Radermacher, F. Portheine, M. Anton, A. Zimolong, G. Kaspers, G. Rau, H. Staudte, Computer assisted orthopaedic surgery with image based individual templates, Clin Orthop and Rel Res 354 (1998), 28–38. K. Birnbaum, E. Schkommodau, N. Decker, A. Prescher, U. Klapper, K. Radermacher, Computerassisted orthopedic surgery with individual templates and comparison to conventional operation method, Spine 26(4) (2001), 365–370. G. Brown, K. Firoozbakhsh, T. Jr. DeCoster, M. Moneim, Rapid prototyping: the future of trauma surgery? J Bone and Joint Surg 85-A suppl. (2003), 49–55. B. Owen, G. Christensen, J. Reinhardt, T. Ryken: Rapid prototyping patient-specific drill template for cervical pedical screw placement, Comput Aided Surg 12(5) (2007), 303–308. M. Mont, P. Ragland, G. Etienne, T. Seyler, T. Schmalzried, Hip resurfacing arthroplasty, J Am Acad Orthop Surg 14(8) (2006), 454–463. H. Amstutz, S. Ball, M. LeDuff,F. Dorey, Resurfacing tha for patients younger than 50 year: results of 2- to 9-year followup, Clin Ortho Rel Res 460 (2007), 159–164. H. Amstutz, Hip resurfacing arthroplasty, J Am Acad Orthop Surg 14(8) (2006), 452–453. H. Amstutz, P. Campbell, M. LeDuff, Fracture of the neck of the femur after surface arthroplasty of the hip, J Bone Joint Surg 86-A(9) (2004), 1874–1877. T. Siebel, S. Maubach, M. Morlock, Lessons learned from early clinical experience and results of 300 asr hip resurfacing implantations, Proc Inst Mechl Engin - Part H 220(2) (2006), 345–353. T. Hess, T. Gampe, C. Koettgen, B. Szawlowski, Intraoperative navigation for hip resurfacing. methods and first results [article in german], Orthopade 33(10) (2004), 1183–1193. P. Belei, A. Skwara, M.D.L. Fuente, E. Schkommodau, et al, Fluoroscopic navigation system for hip resurface replacement, Comput Aided Surg 12(3) (2007), 160–167. A. Hodgson, K. Inkpen, M. Shekham, C. Anglin, J. Tonetti, B. Masri, C. Duncan, D. Garbuz, N. Greidanus, Computer-assisted femoral head resurfacing, Comput Aided Surg 10(5-6) (2005), 337–343. E. Davis, P. Gallie, K. Macgroarty, J. Waddell, E. Schemitsch, The accuracy of image-free computer navigation in the placement of the femoral component of the Birmingham hip resurfacing: a cadaver study, J Bone Joint Surg 89(4) (2007), 557–560.

290

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-290

Real-Time 3D Avatars for Tele-rehabilitation in Virtual Reality Gregorij KURILLO a,1 , Tomaz KORITNIK b Tadej BAJD b and Ruzena BAJCSY a a University of California, Berkeley, USA b University of Ljubljana, Slovenia Abstract. We present work in progress on a tele-immersion system for telerehabilitation using real-time stereo vision and virtual environments. Stereo reconstruction is used to capture user’s 3D avatar in real time and project it into a shared virtual environment, enabling a patient and therapist to interact remotely. Captured data can also be used to analyze the movement and provide feedback to the patient as we present in a preliminary study of stepping-in-place task. Such tele-presence system could in the future allow patients to interact remotely with remote physical therapist and virtual environment while objectively tracking their performance. Keywords. Teleimmersion, Rehabilitation, Telerehabilitation, Lower extremities

Introduction One of the major goals in rehabilitation is to make quantitative and qualitative improvements in daily activities in order to improve quality of independent living [15]. Rehabilitation process often includes task-oriented training and repetition of different motor activities involving impaired neuromuscular or musculoskeletal system [10]. In traditional rehabilitation approach, patient is guided by a trained physical therapist who observes and assists the patient to perform the tasks correctly. This process, however, is labor intensive, time consuming and often very subjective. Patient, on the other hand, often perceives the repetitive tasks as dull and non-engaging, which is consequently reducing patient’s level of involvement in the rehabilitation. Several studies have shown importance of patient’s psychological response, which greatly depends on the cognitive feedback associated with the performance of the tasks and affects success of the rehabilitation [6][7].

1. Related Work Virtual reality (VR) as such can enrich the visual feedback associated with the performance of rehabilitation tasks [14]. In the past, many of the VR-based rehabilitation systems relied on custom-built devices to be used for input or feedback in virtual environ1 Corresponding Author: Gregorij Kurillo, University of California, Berkeley, #736 Saturdja Dai Hall, CA 94720, Berkeley; E-mail: [email protected].

G. Kurillo et al. / Real-Time 3D Avatars for Tele-Rehabilitation in Virtual Reality

291

ments. Such devices, however, were usually associated with high costs, low reliability, low accessibility, and poor ergonomic design, making them unsuitable for clinical use. More recently, some of the commercial systems for gaming have been adopted for use in rehabilitation applications, such as Wii Remote and Wii Fit by Nintendo [3][11]. Several studies have also included live video-feedback of the patient to be integrated with the virtual environment to enhance patient’s feeling of presence in the interactive space [3][8]. In these applications, the captured video was used to provide the visual feedback (as a virtual mirror) of the patient immersed in the environment. The video data was also used to interact with the graphics environment (e.g. popping bubbles with your hand) [3]. The technology behind it, however, does not allow for a full three-dimensional (3D) interaction in the virtual space as the users are only captured by a regular (2D) video camera.

2. Tele-immersion In our framework (Fig. 1) we address some of the issues associated with creating the immersive feedback and data acquisition when using such video systems for rehabilitation and training of motor skills. We employ one or more stereo cameras to capture 3D avatar of the user/patient in real-time [16]. In contrast to traditional video cameras, the data produced preserves geometry of the movement with respect to the body and the environment allowing for accurate mapping of the movement into the virtual environment. The 3D video stream can be sent remotely or displayed locally while being seamlessly integrated with the virtual scene. Generated 3D mesh can be enhanced with dynamic texture to improve the visual quality of the video. In addition the 3D data captured by the cameras can be analyzed in real time to provide feedback on the screen while posing no restrictions on user’s movements, such as in the case of motion capture systems with markers. This tele-immersive technology in connection with a virtual reality can provide a feeling of remote presence (i.e. tele-presence). A shared virtual environment can host several individuals from mutually distant locations and enable them to interact with each other in real time via a system of video cameras, audio communication, and fast network-based data transmission. Such approach can represent a basis for tele-rehabilitation practices that we are addressing in our current and future work. In the past we have demonstrated beneﬁts of immersive training in teaching of TaiChi movements [1], dance [14] and remote interaction. Similar tele-immersive technology has also been applied by one of our collaborators in the coaching of basketball players in wheelchairs [2]. Recently we have made signiﬁcant advancements in the stereo algorithms to allow for real-time (25+ FPS) capture of 3D videos from one or more stereo cameras. Application of our technology in the area of telemedicine is in the early stage; however, we feel it is important to share the experience of this new emerging technology with the VR-based medical and rehabilitation community. In this paper we present a pilot study on a group of healthy individuals using stepping-in-place (SIP) task which has a long history in evaluation of movement patterns in lower extremities [4]. In our work we apply 3D video in two ways, (a) to generate lifelike visual feedback of the remote therapist and patient as their reﬂection in a virtual mirror and (b) to measure the hip angles during task performance directly from the data without using markers. Finally, we outline our future work in the tele-rehabilitation and full body tracking using the tele-immersion technology and VR.

292

G. Kurillo et al. / Real-Time 3D Avatars for Tele-Rehabilitation in Virtual Reality

Figure 1. Diagram of the proposed setup for tele-immersive rehabilitation in system using a shared virtual environment and 3D avatars.

3. Methods & Materials 3.1. Real-time 3D Video To generate patient’s real-time 3D avatar for the VR rehabilitation task, a stereo camera is needed to capture two slightly displaced images. Our stereo algorithm with an adaptive meshing scheme [16] allows for fast and robust stereo matching on the triangle nodes. The stereo reconstruction can achieve the frame rate of more than 25 FPS on a pair of re-sampled images with 320x240 pixels or about 10 FPS on images with the resolution of 640x480 pixels. Result is a 3D mesh of the surface, which can be further mapped with a high resolution dynamic texture to achieve better visual ﬁdelity. Further details on the algorithm can be found in [16]. The accuracy of the depth reconstruction depends on several factors related to the camera arrangement and typically ranges from 1 cm to 3 cm in our setup. Several stereo views can be combined by externally calibrating the cameras to a common coordinate system to increase user workspace or to generate 360-degree avatars. 3D video is then streamed through a gateway to the local renderer or remote location enabling telepresence by linking two or more remote sites. The captured 3D data is also used to perform body tracking and extract simpliﬁed kinematics which is used in the feedback loop for augmenting the video with virtual objects. Since user’s body is captured by a calibrated stereo camera, the body movement is accurately mapped into the virtual environment and the geometry of the workspace is preserved. The generated data is also suitable for display on a 3D screen. 3.2. Stepping-in-Place (SIP) Task The SIP task consists of guided performance of rhythmic movement of the lower extremities. It allows for the assessment of basic temporal parameters closely related to gait, such as stance and swing phase, double-stance phase, and step frequency. In the previous studies in connection with VR, the SIP task was considered as a modality of lower-

G. Kurillo et al. / Real-Time 3D Avatars for Tele-Rehabilitation in Virtual Reality

293

extremities training for rehabilitation [10]. In this study, a virtual mirror was applied, displaying a generic avatar driven by a motion capture system to provide feedback during the training. The subjects were asked to track the steps performed by a virtual teacher. One of the concerns reported was the use of the same human ﬁgure model for all subjects as compared to using a personalized avatar as a form of feedback. Our tele-immersion framework enables exactly that by real-time 3D capture of a patient and a therapist. The therapist guiding the rehabilitation tasks can be located in the same room, at different geographical location or pre-recorded to replicate the same movements in every session. In our experiments we chose the latter to ensure a consistent reference motion pattern. In addition, the 3D data generated by the stereo system was also applied for the lower extremity kinematics extraction, thus allowing for markerless capture of hip angles. As mentioned above, the 3D video of the ’therapist’ was pre-recorded for our task to achieve consistency across subjects. We enabled visualization of two persons at the same time in the same virtual environment. The subjects observed themselves on the screen with the virtual ’therapist’ rendered next to them (Fig. 2). They were instructed to track the ’therapist’ avatar’s movement as closely as possible. Two scenarios were tested: (1) 3D video only and (2) 3D video with overlaid virtual tracking targets to enhance visual feedback. In the ﬁrst scenario, the left and right leg was shaded with two different color tones. The tracking targets in the second scenario marked the location of the knee joints for both persons. The targets were scaled and superimposed on the therapist’s avatar. 3.3. Experimental Setup The hardware setup consisted of one stereo camera Bumblebee2 (Point Grey, Inc.), with the resolution of 1024x768 pixels and the focal length of 3.8mm, was positioned above 65-inch LCD screen in front of the subject. The subject was standing upright at a marked position about 3m from the display and the camera. During the execution of the tasks, the subject was instructed to keep the arms close to his/her body to allow the algorithm to perform the segmentation based on the body symmetry along the sagittal plane. The stereo reconstruction was performed by a two dual core 2.33 GHz machine with the connected camera while the rendering and segmentation was performed on a dual quad core 2.00 GHz graphics server with GeForce GTX 8800.

Figure 2. Stereo reconstruction is used to capture user’s 3D avatar in real time (left) and project it into a shared virtual environment (right), enabling a patient and therapist to interact remotely. Color shading of the legs assists with focus and orientation within the virtual mirror projection.

294

G. Kurillo et al. / Real-Time 3D Avatars for Tele-Rehabilitation in Virtual Reality

3.4. Trials We have performed preliminary experiments on the group of 12 healthy individuals with the average age of 26.7 +/- 5.7 years (minimum age 20 years; maximum age 37 years). None of the subjects had a medical history of signiﬁcant lower limb injuries or any other known medical condition that would impair movement. All subjects gave informed consent to participate in the experiment. Subjects performed the task in each of the two scenarios three times, starting with the 3D video only feedback. The reference recording required the subjects to exert hip angles of about 30◦ during the SIP task. 3.5. Data Analysis To obtain the hip angles and knee joint positions from the stereo data, a simpliﬁed 3D video kinematics analysis was performed online on the renderer side (Fig. 4). The algorithm ﬁrst segmented the data into left and right body half, assuming the symmetry of the body with respect to the camera coordinate system. Position of the lower part of the body was calculated from the ergonomics table while the segmented left and right leg was projected onto a plane aligned with the sagittal body plane. From the projection, hip and knee angles were calculated using a line ﬁtting algorithm (using least squares method). Only the hip angle was used for the feedback and the analysis. The hip angle of 0◦ was deﬁned in an upright standing position with the angle increasing to 90◦ as the leg was lifted from the ﬂoor. The accuracy of the method for the hip angle calculation was evaluated using a motion capture system and was within 10-degree error margin. The results of the measured hip angles were analyzed using correspondence algorithm presented in [5] and variance analysis of the spatial and temporal adaptation [9].

Figure 3. Block diagram of the segmentation and angle extraction process with intermediate results.

4. Results The results of the experiments were analyzed for the two different scenarios by calculating the error of the aligned signals between the teacher and each subject. Fig. 4 (left) shows the output of the right and left hip angles as acquired in one of the subjects as compared to the reference recording. The shown output was captured for the 3D video only scenario. The subject closely followed the reference, with only small delays ( 200ms). The result in Fig. 4 (left) shows more precise tracking with the left leg as compared to the right one where larger deviations can be observed.

G. Kurillo et al. / Real-Time 3D Avatars for Tele-Rehabilitation in Virtual Reality

295

Fig. 4 (right) shows the distribution of the tracking results for the spatial and temporal adaptation in all subjects (n=12) with (1) and without (2) superimposed targets. The box diagram presents 5.0, 25.0, 50.0, 75.0, and 95.0 percentiles of the distribution. The spatial and the temporal adaptation was statistically different between the two scenarios (p < 0.001, ANOVA) suggesting that the augmented feedback (with superimposed tracking targets) helped subjects perform the task with greater accuracy. 50

Subject Reference

Left Hip Angle (°)

40 30 20 10 0 −10 30

35

40

45

50 t (s)

55

60

Right Hip Angle (°)

50

65

70

Subject Reference

40 30 20 10 0 −10 30

35

40

45

50 t (s)

55

60

65

70

Figure 4. Sample output of the measured left (above) and right (below) hip angle trajectories during the trial. The angles were extracted in real time with a markerless method from the captured 3D video. Mean distribution (n=12) of the error for spatial and temporal adaptation during the two conditions: (1) 3D video with superimposed tracking targets and (2) 3D video only.

5. Discussion & Future Work The focus of this exemplar study was to investigate the feasibility of the tele-presence system for use in tele-rehabilitation. The group of healthy individuals successfully performed the tracking task of stepping in place. We compared the tracking results of 3D video-only with 3D video enhanced by virtual targets, and concluded that better spatial and temporal adaptation was achieved when additional tracking targets were displayed. Our ongoing work is directed towards development of tele-immersive framework for use in different application areas that would beneﬁt from collaborative aspect of real-time 3D video, such as in tele-rehabilitation, medical evaluation, sports medicine, teaching of dancing and several areas of collaborative work. In many of the application, full-body segmentation and tracking is crucial for extraction of kinematic parameters which can be used to provide online feedback (such as in the presented example), perform gesturebased interaction or to drive computer-generated avatars. In this preliminary study, the extraction of the kinematics was simpliﬁed and closely related to the task. Our goal is to achieve more general human body segmentation and tracking from the 3D data in real-time (e.g. [12]) which could be applied to various tasks in VR-based (tele-) rehabilitation and applications of markerless motion capture. The performance of the current algorithms for full-body tracking is limited due to real-time constraints and sensitivity to outliers in the stereo data. In this study and our past research work [1][14] we showed that current technology provides the user with feeling of tele-presence, suitable for remote teaching of body mo-

296

G. Kurillo et al. / Real-Time 3D Avatars for Tele-Rehabilitation in Virtual Reality

tion (such as in tele-rehabilitation), using relatively affordable equipment. The system also produces data which can be used online for feedback or ofﬂine for analysis which can quantify patient’s performance during different motor activities. In this way, the patients could in the future participate in rehabilitation process from their homes or smaller medical ofﬁce without the need to travel to large urban rehabilitation centers. Acknowledgements Research work on stepping in place task was supported by Slovenian Research Agency. Development of the Tele-immersion framework was partially supported by NSF (grants: 0703787, 0724681, 0937060), HP Labs, EADS and CITRIS at University of California, Berkeley. References [1] [2]

[3]

[4] [5]

[6] [7] [8] [9] [10] [11]

[12] [13] [14]

[15] [16]

J.N. Bailenson, K. Patel, A. Nielsen, R. Bajcsy, S. Jung, G. Kurillo. The effect of interactivity on learning physical actions in virtual reality, Media Psychology 11 (2008), 354-376. P. Bajcsy, K. McHenry, H.J. Na, R. Malik, A. Spencer, S.K. Lee, R. Kooper, M. Frogley. Immersive environments for rehabilitation activities, Proceedings of ACM International Conference on Multimedia, Beijing, China, (2009). J.W. Burke, M.D.J. McNeill, D.K. Charles, P.J. Morrow, J.H. Crosbie, S.M. McDonough. Serious games for upper limb rehabilitation following stroke, Proceedings of Games and Virtual Worlds for Serious Applications, Coventry, (2009), 103-110. T. Fukuda. The stepping test: two phases of the labyrinthine reﬂex, Acta Oto-Laryngol 50 (1958), 95108. M.A. Giese, T. Poggio. Synthesis and recognition of biological motion patterns based on linear superposition of prototypical motion sequences, Proceedings of the 1999 IEEE Workshop on multi-view modeling and analysis of visual scene, Fort Collins, CO, USA, (1999), 73-80. R.L. Hewer, Rehabilitation after stroke, Neurological Rehabilitation, Blackwell Scientiﬁc Publications, Inc., Oxford, UK, (1994), 157-166. M.K. Holden, T. Dyar. Virtual environment training: a new tool for neurorehabilitation, Neurology Report 26 (2002), 62-71. R. Kizony, N. Katzand, P.L. Weiss. Adapting an immersive virtual reality system for rehabilitation, J. Visual. Com. Anim. 14 (2003), 261-268. T. Koritnik, T. Bajd, M. Munih. Virtual environment for lower-extremities training, Gait & Posture 27 (2008), 323-330. J.W. Krakauer. Motor learning: its relevance to stroke recovery and neurorehabilitation, Curr. Opin. Neurol. 19 (2006), 84-90. R.S. Leder, G. Azcarate, R. Savage, S. Savage, L.E. Sucar, et al. Nintendo Wii Remote for computer simulated arm and wrist therapy in stroke survivors with upper extremity hemipariesis, Proceedings of Virtual Rehabilitation, Vancouver, BC, (2008), p. 74. J.M. Lien, G. Kurillo, R. Bajcsy. Multi-camera tele-immersion system with real-time model driven data compression, The Visual Computer 26, (2010), 3-15. J. Moline. Virtual reality for health care: a survey, Virtual Reality in Neuro-Psycho-Physiology, IOS Press, Amsterdam, Netherlands, (1998). K. Nahrstedt, R. Bajcsy, L. Wymore, R. Sheppard, K. Mezur. Computation model of human creativity in dance choreography, Proceedings of Association for the Advancement of Artiﬁcial Intelligence (AAAI) Spring Symposia, (2008). H. Sveistrup. Motor Rehabilitation using virtual reality, J. NeuroEngineering & Rehabilitation 10, (2004), 1-10. R. Vasudevan, Z. Zhou, G. Kurillo, E. Lobaton, R. Bajcsy, K. Nahrstedt. Real-time stereo-vision system for 3d tele-immersive collaboration, Proceedings of IEEE International Conference on Multimedia & Expo, Singapore, (2010).

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-297

297

Fundamentals of Gas Phase Plasmas for Treatment of Human Tissue Mark J. KUSHNER and Natalia Yu. BABAEVA University of Michigan, Dept. of Electrical Engineering and Computer Science 1301 Beal Ave., Ann Arbor, MI 48109-2122 USA [email protected]

Abstract. The use of gas phase plasmas for treating human tissue is at the intersection of two disciplines – plasma physics and engineering, and medicine. In this paper, a primer will be provided for the medical practitioner on the fundamentals of generating gas phase plasmas at atmospheric pressure in air for the treatment of human tissue. The mechanisms for gas phase plasmas interacting with tissue and biological fluids will also be discussed using results from computer modeling. Keywords. Gas electroporation

phase

plasma,

plasma-medicine,

electrical

discharges,

Introduction The use of gas phase plasmas for treating human tissue is an emerging area of research and clinical studies.[1-4] Gas phase plasma treatment of tissue has been shown to be an effective therapy for wound healing, blood clotting and wound sterilization, in addition to sterilization of skin and instruments. Gas phase plasmas in the context of plasma medicine are partially ionized gases in which a small fraction of the gas molecules are ionized, perhaps only a few ppm (partsper-million).[5,6] (Unless noted otherwise, the term plasma in this paper refers to a gas phase ionized gas and not the biological fluid.) The neutral gas in an atmospheric pressure plasma typically remains at or near room temperature while the electrons in the gas-phase plasma are heated by applied electric fields to many electron volts (1 eV = 11,600 K). These plasmas are often called low temperature plasmas to distinguish them from the high temperature plasmas that are employed in fusion experiments. If the electrons have low temperature compared to fusion plasmas, they are still hot with respect to room temperature. However, since the electrons have such a small mole fraction in the atmospheric pressure gas, the thermal energy of the plasma is small. As a result, the plasma is not thermally damaging to tissue in contact with the plasma. Since the electrons are thermally hot, they are able to activate gas molecules by electron impact reactions. When these energetic electrons collide with atoms and molecules, they are able to produce additional electrons through ionization processes and create gas phase radicals by electron impact dissociation. These chemically active radicals diffuse to surfaces, such as tissue, where reactions may take place.[7] The fact that gas phase plasmas contain electrons, positive ions and negative ions which produce an electrical charge density, plasmas are capable of modifying the electric fields applied to the plasma. Plasmas are also able to transfer those electric

298

M.J. Kushner and N.Yu. Babaeva / Fundamentals of Gas Phase Plasmas

fields to materials bounding the plasma. The typical treatment method in plasma medicine is to generate a gas phase plasma in air in the vicinity of the tissue to be treated. The ionized molecules, gas phase radicals and modified electric fields produced by the plasma are then in contact with the tissue, modifying surface and intracellular processes, which lead to therapeutic effects. In this paper, a primer will be provided for medical practitioners on the fundamentals of gas phase plasmas, the production of plasmas in atmospheric pressure air and how these plasmas may interact with tissue and biological fluids.

1. Gas Phase Plasmas A gas phase plasma is initiated by applying an electric field to a gas. The electric field accelerates randomly appearing seed electrons, as might be produced by cosmic rays, to energies of many eV. When the seed electrons collide with gas molecules, such as the N2 and O2 in air, the electrons can ionize, excite or dissociate the molecules. The electrons can also be attached to the molecule to form a negative ion. Excited states of molecules or atoms then emit photons. For example, in an air plasma a subset of the electron impact reactions that may occur include:

These electron impact processes include sources (ionization processes that produce additional electrons) and sinks (recombination and attachment processes that consume electrons). If the applied electric field is of sufficient magnitude and the electrons are accelerated to high enough energies, the sources will be larger than the sinks, and the initial electrons will avalanche to produce a plasma that is composed of a few ppm to a few percent of electrons and ions. Due to the loss of a small fraction of electrons to boundaries, the total charge in the plasma is slightly positive, and so these plasmas are called electropositive. The plasma and electrical circuitry providing the accelerating electric fields will modify the applied electric fields so that in the steady state, the sources of electrons are equal to the sinks. Following these electron impact events, further ion-molecule reactions may occur to form different ions (e.g., N2+ + N2 + N2 → N4+ + N2.) The neutral radicals also undergo further reactions (e.g., O + O2 + N2 → O3 + N2). In state of the art models of air plasma chemistry, there are often hundreds of reactions and tens of neutral and charged species required to accurately describe the plasma.[8] The end result is the production of ion fluxes and neutral radical fluxes to surfaces in contact with the plasma, such as human tissue. In an air plasma, these fluxes contain reactive oxygen species (ROS), such as O atoms, O2(1Δ) (electronically excited oxygen), OH, and O3; and reactive nitrogen species (RNS), such as N, N2(A) (electronically excited nitrogen), and NO. The magnitude and variety of these fluxes, in addition to the photons fluxes produced by the plasma, then determine the therapeutic effects of plasma treatment. In addition to these reactive fluxes, plasmas can deliver impulses of large electric fields to the tissue – large enough to produce electroporation.[9,10] These electric fields can also accelerate ions into the surface being treated. The ions can attain

M.J. Kushner and N.Yu. Babaeva / Fundamentals of Gas Phase Plasmas

299

energies as large as 10-20 eV or more, though over only a few to tens of ns duration, which is also the duration of the ion fluxes. The fluxes of neutral radicals are sometimes orders of magnitude higher than those of ions, and have lifetimes of many ms or longer in the case of selected ROS and RNS such as O3 and NO. It is likely that many plasma medicine treatments are dominated by the far larger and longer lasting neutral radical fluxes. The electric fields which create these plasmas are externally applied by electrodes connected to power supplies or launched into the plasma by antennas or coils. Through separation of negative and positive charge due to their drift in opposite directions in the electric field or by diffusing to boundaries at different rates, the plasma can modify the applied electric fields. This process is described by Poisson’s equation,∇⋅ε∇Φ = -ρ, where Φ is the electric potential, ε is the electrical permittivity of the material through which the electric fields propagate and ρ is the net charge density (the difference in density between negative and positive charges). This modification of the applied electric potential by the plasma enables relatively modest voltages to avalanche air (that is, create an ionization cascade) at atmospheric pressure, and so make plasma treatment of tissue possible in an open room environment.

2. Typical Treatment Scenario A typical treatment device for plasma medicine is similar the dielectric-barrierdischarge (DBD) shown in Fig. 1.[11] A DBD is a plasma generating device where an alternating high voltage [V(t) in the figure] is applied to a metal electrode that is encased in a dielectric. A grounded electrode is similarly encased in a dielectric. The plasma is formed between the dielectrics in the form of a forest of small filaments or streamers (a few hundred to a few thousand per cm2) having diameters of a few hundred microns. (See Fig. 2.) The electron densities in the filaments are up to 1015 cm-3, which produce fluxes of ROS and RNS to surfaces up to 1020 cm-2s-1. The charging of the dielectrics by the plasma filaments produces opposing electric fields that terminate the current flow in the filament and prevent thermal damage to the surface. As such, each filament may last only a few to 10 ns. The plasma filaments are produced with each cycle of the alternating voltage applied to the DBD, with frequencies of up to tens of kHz. In plasma medicine applications, the tissue being treated acts as a floating electrode, also shown in Fig. 1. The DBD device then has only one dielectric covered electrode and the filaments terminate on the tissue. The surface of the tissue is charged as in a conventional DBD. Although these sometimes randomly appearing filaments or streamers have implications with respect to the uniformity of treatment, the fact that they are randomly appearing and short in duration insures that the tissue in contact with the filaments is not damaged. The neutral radical fluxes to the tissue produced by the filaments are far more uniform.

300

M.J. Kushner and N.Yu. Babaeva / Fundamentals of Gas Phase Plasmas

Figure 1. Schematic of (left) a conventional dielectric barrier discharge (DBD) and (right) a DBD filament intersecting with tissue that acts as a floating electrode.

Figure 2. A dielectric barrier discharge (DBD) sustained in room air. (top) Plasma filaments of a DBD resulting from many hundreds of voltage pulses. (bottom) DBD in contact with a human thumb. (From Ref. 4.)

3. Modeling Plasma Treatment of Tissue Sophisticated computer models are being used to investigate the fundamental processes of plasma generation and treatment of tissue in plasma medicine. One such example is the DBD plasma treatment of skin and prediction of electroporation capable electric fields in the skin.[12] In this 2-dimensional model, called nonPDPSIM, conservation equations are solved for the density, momentum and energy of electrons, ions and neutral species, in concert with Poisson’s equation for the electric potential. A numerical mesh is constructed to represent the structure of the DBD electrodes and the skin surface, in this case the thumb shown in Fig. 2. The numerical mesh is able to resolve many orders of magnitude in spatial scale – and so the cellular structure in the epidermis is resolved on the micron level, as shown in Fig. 3. The cell membranes, cytoplasm and nucleoplasm are separately assigned electrical permittivities and conductivities so that the generation of electric fields within the cells can be assessed. Electrical currents and electric fields are computed within the tissue using these permittivities and conductivities by requiring current conservation. The gas phase plasma is modeled using a full accounting the plasma chemistry of humid air. A typical calculation (initiation and propagation of the plasma filament, and its intersection with the skin) takes approximately 1 day on a high speed computer workstation. Examples of results from the modeling of plasma filaments intersecting skin are shown in Figs. 4 and 5. The electron density during the propagation of three filaments from the DBD electrode to the surface of the thumb is shown in Fig. 4. The voltage in the DBD applicator is -30 kV. The plasma filaments require only 2-3 ns to propagate across a 2 mm gap between the DBD electrode and the thumb surface, with electron densities approaching 1015 cm-3. This represents a fractional ionization in the filaments of 0.01%. The widths of the filaments are a few hundred microns. Note that the filaments attempt to orient themselves perpendicular to the curved skin surface. This results from the skin having a finite electrical conductivity. The reorientation of the

M.J. Kushner and N.Yu. Babaeva / Fundamentals of Gas Phase Plasmas

301

filaments may produce different treatments across the skin surface – and is one of the challenges of designing plasma devices that produce uniform treatments. As the filament approaches the skin, displacement currents from the filament produce charge separation in cells within a few hundred microns to a mm of the surface. Upon intersection of the plasma filament with the skin, the surface of the skin is charged negatively, and electric fields are launched into the sub-surface layers. Since the duration of the current pulse is commensurate with the dielectric relaxation time of the cells, these electric fields are able to penetrate into the conductive cells. Electric fields of 150 kV/cm are sustained across the cell membranes and electric fields of 5060 kV/cm are sustained across the more conductive cytoplasm and nucleoplasm, as shown in Fig. 5. These values of electric fields across the cell membranes correspond to voltage drops of a few tenths of a volt, which are sufficient to initiate electroporation.

Figure 3. Representation of skin cells in nonPDPSIM for modeling plasma treatment of tissue. (From Ref. 12.)

The duration of these large electric fields imparted to the cells is short, only a few to perhaps 10 ns. However, DBDs can operate at many tens of kHz, and so cells can be repetitively treated in this manner. A current topic of research is determining the repetition rate with which these pulses must be applied to enable their effects on the cells to be cumulative.

3. Concluding Remarks Low temperature plasmas sustained in room air are now being investigated for the therapeutic treatment of human tissue. Atmospheric pressure plasmas have been commercially developed for other purposes (e.g., plasma functionalization of polymers), and so their use for directly treating human tissue can leverage these existing knowledge bases. Sophisticated computer models, originally developed for plasma materials processing of organic and inorganic substances are being adapted to investigate the plasma treatment of living tissue. The insights provided by these investigations will speed the development of plasma sources and protocols in plasma medicine.

Acknowledgments This work was supported by the Department of Energy Office of Fusion Energy Sciences.

302

M.J. Kushner and N.Yu. Babaeva / Fundamentals of Gas Phase Plasmas

Figure 4. Results from nonPDPSIM for the propagation of plasma filaments from the plasma applicator to the skin of a thumb. The color scale shows the density of electrons in the streamer filament on a log scale over 2 decades. The filaments reorient themselves to intersect the skin surface at nearly a perpendicular angle. (From Ref. 12.)

Figure 5. Results from nonPDPSIM for electric fields across skin cell membranes and cell interiors produced by the intersection of a plasma filament generated in a dielectric barrier discharge sustained in air. Electric fields exceed 100 kV/cm across cell membranes and can exceed 50 kV/cm across cell interiors. (From Ref. 12.)

M.J. Kushner and N.Yu. Babaeva / Fundamentals of Gas Phase Plasmas

303

References [1]

D. Dobrynin , G. Fridman, G. Friedman and A. Fridman, “Physical and biological mechanisms of direct plasma interaction with living tissue” New J. Phys. 11, 115020 (2009). [2] M. G. Kong, G. Kroesen, G. Morfill, T. Nosenko, T. Shimizu, J. van Dijk and J. L. Zimmermann “Plasma medicine: an introductory review” New J. Phys. 11, 115012 (2009). [3] G. Fridman, G. Friedman, A. Gutsol, A. B. Shekhter, V. N. Vasilets and A. Fridman “Applied plasma medicine” Plasma Process. Polym. 5, 503 (2008). [4] G. Fridman, M. Peddinghaus, H. Ayan, A. Fridman, M. Balasubramanian, A. Gutsol, A. Brooks and G. Friedman, “Blood coagulation and living tissue sterilization by floating electrode dielectric barrier discharge in air”, Plasma Chem. Plasma Process. 26, 425 (2006). [5] M. A. Lieberman and A. J. Lichtenberg, “Principles of Plasma Discharges and Materials Processing” (Wiley-Interscience, New Jersey, 2005). [6] A. Fridman and L. A. Kennedy, “Plasma Physics and Engineering” (Taylor and Francis, New York, 2004) [7] A. Fridman, “Plasma Chemistry”, (Cambridge Univ. Press, New York, 2008). [8] I. A. Kossyi, A. Yu Kostinsky, A. A. Matveyev and V. P. Silakov, Plasma Sources Sci. Techol. 1, 207 (1992). [9] K. H. Schoenbach, S. J. Beebe and E. S. Buescher, “Intracellular effect of ultrashort electrical pulses”, J. Bioelectromagn. 22, 440 (2001). [10] S. J. Beebe, P. M. Fox, L. J. Rec, K. Somers, R. H. Stark and K. H. Schoenbach, “Nanosecond pulsed electric field (nsPEF) effects on cells and tissues: apoptosis induction and tumor growth inhibition”, IEEE Trans. Plasma Sci. 30, 286 (2002). [11] U. Kogelschatz, Plasma Chem. Plasma Proc. 23, 1 (2004). [12] N. Yu. Babaeva and M. J. Kushner, J. Phys. D: Appl. Phys. 43, 185206 (2010).

304

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-304

VR-Based Training and Assessment in Ultrasound-Guided Regional Anesthesia: From Error Analysis to System Design Erik LÖVQUISTa,1, Owen O'SULLIVANb, Donnchadh OH’AINLEa, Graham BAITSONa, George SHORTENb and Nick AVIS c a National Digital Research Centre, Ireland b Cork University Hospital, University College Cork, Ireland c Cardiff School of Computer Science, Cardiff University, UK

Abstract. If VR-based medical training and assessment is to improve patient care and safety (i.e. a genuine health gain), it has to be based on clinically relevant measurement of performance. Metrics on errors are particularly useful for capturing and correcting undesired behaviors before they occur in the operating room. However, translating clinically relevant metrics and errors into meaningful system design is a challenging process. This paper discusses how an existing task and error analysis was translated into the system design of a VR-based training and assessment environment for Ultrasound Guided Regional Anesthesia (UGRA). Keywords. Virtual reality, ultrasound guided regional anesthesia, training, assessment, error analysis, system design.

Introduction Since “To Err is Human” [1] was published, there has been greater attention to patient safety [2]. Many regard simulation-based training as a potential route to safer healthcare [3]. This is particularly true for the training of procedural skills, for which trainee doctors can gain certain proficiencies prior to attempting them on a patient. Specifically, the identification and correction of errors in this pre-clinical setting is argued to be the most valuable way to maximize patient safety [4]. If a Virtual Reality (VR)-based medical training system measures clinically relevant errors, it can be used to correct undesirable behaviors, potentially minimizing errors involving patients in the operating room. However, many medical training systems focus on improving general technical performance [4]. It is not clear if such performance measures alone correlate with greater patient benefit and safety. The development of VR-based medical training systems that can i. improve technical performance and ii. support recognition of errors is a challenge. We speculate that such a system will result in long term improvements in patient safety. In this paper we discuss how an existing task and error analysis was applied during the design of a VR-based training and assessment environment in

1

Corresponding Author: Dr. Erik Lövquist, National Digital Research Centre, Ireland. Email: [email protected]

E. Lövquist et al. / VR-Based Training and Assessment in Ultrasound-Guided Regional Anesthesia 305

Ultrasound Guided Regional Anesthesia (UGRA). This environment is intended to be used by the trainee unattended, i.e. without expert or consultant guidance.

1. Training Analysis An existing task and error analysis of a specific UGRA procedure, Ultrasound guided Axillary Brachial Plexus Blockades (USgABPB) [5], was utilized to direct the design of a VR-based medical training and assessment environment. The analysis consisted of a Hierarchical Task Analysis (HTA) [6], a Systematic Human Error Reduction and Prediction Approach (SHERPA) [7] and a Failure Modes, Effects, and Criticality Analysis (FMECA) [8]. The HTA provided a description of all significant operations and sub-operations involved in the USgABPB procedure. In this analysis, the procedure was decomposed into progressively smaller and more precise sub-operations each of which contributes to achieving the desired behavior. The operations were arranged in a hierarchical relationship, such that each operation was associated with a goal and a plan. In total, the analysis covered more than 200 HTA operations and suboperations. The HTA provides a comprehensive and structured description of the procedure. The SHERPA was based on an error taxonomy consisting of probability, criticality, recovery potentials and remedial strategies. This taxonomy was used to generate predictions of potential human errors within the HTA hierarchy. The SHERPA provides detailed information about the errors that potentially occur during the performance of USgABPB in novice hands. The FMECA generated a weighting of the errors identified in the SHERPA based on probability, severity and detectability. These three factors were combined to calculate a criticality index (CI). The magnitude of the CI value determined the significance of each error. The FMECA provides an ordering of the SHERPA errors based on their CI.

2. System Design In order to utilize the HTA, SHERPA and FMECA in a training and assessment environment for USgABPB, they had to be “translated” into system requirements and design. 2.1. Participatory Design The design of the VR-based training and assessment environment involved system developers, medical experts and a psychologist collaborating closely throughout the development process. Participatory Design [9] was applied to ensure that the medical experts were able to participate constructively as part of the development team, providing continuous input to the design of the training and assessment environment. For instance, the medical experts were regularly exposed to functioning prototypes, which allowed them to provide feedback on the system’s design. From seeing the prototypes, they were able to validate the technical implementations, provide new design ideas and control how the metrics and errors were to be applied in the environment. Their active participation and systematic input effectively directed the system developers in how to translate the task and error analysis into system design.

306 E. Lövquist et al. / VR-Based Training and Assessment in Ultrasound-Guided Regional Anesthesia

2.2. Translation of Metrics and Errors The SHERPA and FMECA helped to ensure that the training and assessment environment covered the most significant errors which can occur during USgABPB. They provided the development team with a list of weighted errors based on probability, severity and detectability. It was decided to single out the twenty most significant errors based on their CI value. By selecting the top twenty errors, it ensured that the environment covered the most important errors and focused specifically on clinically relevant training and assessment (see Table 1 for example errors). Metrics were assigned to each aspect of the HTA, initially based on expert perceptions. The metrics were derived from the HTA in order to capture performance of the defined operations and sub-operations. During prototype development metrics were further refined based on the opinions of experts, through exposure and refinement as part of an iterative process. Error states were then characterized by identifying the metrics which defined them. The FMECA showed that a majority of the most significant errors occurred during tracking of nerves, needle insertion and injection of anesthetic solution. The outcome of the analysis verified that a system based on VR and haptics was a potential option for training and assessing the parts of the procedure where the most significant errors occur. Table 1. Example of four significant errors targeted in the training and assessment environment. Example errors In the event that the needle is poorly or not visualized while advancing it towards the target, the anesthetist continues to advance the needle. In finding the best needle trajectory to perform the block, the anesthetist fails to check the risk of the possible trajectory to cause neural/other injury or vascular puncture. In attempting to optimize the image of a needle which is poor/lost, the anesthetist moves needle rather than the probe. Prior to depositing a bolus of local anesthetic, the anesthetist fails to confirm that the needle tip is visualized.

2.3. Design of Training and Assessment Based on Clinically Relevant Metrics and Errors The HTA, SHERPA and FMECA provided detailed information on performance measures and error identification. However, how to utilize this information in a meaningful training and assessment environment for USgABPB required careful consideration. Specific learning guidelines proposed by Cannon-Bowers et al. [10] were adapted with the aim of providing trainees in USgABPB with optimal learning experiences based on clinically relevant metrics and errors. Cannon-Bowers et al. suggest that patient scenarios should be incorporated into simulation-based training in order to ensure that relevant learning objectives are covered. The USgABPB training and assessment environment was designed to handle

E. Lövquist et al. / VR-Based Training and Assessment in Ultrasound-Guided Regional Anesthesia 307

unique patient scenarios containing events that potentially result in the errors identified in the SHERPA analysis. Cannon-Bowers et al. also suggest that process feedback should be provided during a training session in the initial stages of skill acquisition. However, it has to be designed to minimize cognitive load and gradually fade into outcome feedback as the trainee progresses. Feedback should also be provided to the trainee when an error is made. Based on these guidelines, different means of presenting feedback to a trainee were designed in order to maximize the VR-based environment’s ability to decrease the number of errors and improve trainee performance. The VR-based training and assessment system was designed not to directly alert the trainee when an error is made, but allow them to continue without disruption to minimize cognitive load. For instance, if a critical error has occurred, error-related events are triggered, and the trainee has to deal with a potential crisis situation. This scenario is assessed based on the trainee’s immediate actions. The detected errors are then reported at the end of a training session. At this debriefing stage, the environment provides the trainee with information about the error and its potential consequences. Through the use of a severity score, identified through the FMECA, a hierarchy of errors is generated by the environment. Immediate error feedback is limited to a selection of the most severe errors. This will allow trainees to target their previously performed errors during the next practice attempt, without overwhelming them with a long list of errors should they occur. Where specific errors are repeatedly uncorrected an additional weighting may be applied. The training and assessment was also designed to provide graphical feedback based on performance measurements and errors. For instance, trainees’ improvements over time (multiple training sessions) can be visualized by plotting performance metrics and error rates in 2D graphs. Performance measurements and errors are plotted as bar charts for each previous attempt together with desired proficiency levels and acceptable error rates. Such graphs are intended to direct and motivate the learner by providing clear and simple visualizations of progress over time, indicating any potential need for improvement. Thus, progress made is easily identifiable, by both trainees - stimulating repeated deliberate practice [11] to a mastery level, and also trainers - allowing those with a slower learning curve, in need of targeted tuition, to be identified. The training environment also provides specifically designed feedback on the errors identified as most significant in the SHERPA and FMECA analysis. For example, one significant error is a poorly visualized needle while advancing towards a target (nerve). After a training session, a graphical representation illustrates how well the trainee visualized the needle tip and shaft on the ultrasound image throughout the duration of the procedure (plotted as needle distance from ultrasound plane over time, see Figure 1). The environment was designed to track a trainee’s errors over an extended period of time. By weighting the severity of an error in relation to the frequency of previously performed errors, the environment will determine what operations and sub-operations require further practice. Hence, the environment adaptively customizes each session depending on the skill level of an individual trainee.

308 E. Lövquist et al. / VR-Based Training and Assessment in Ultrasound-Guided Regional Anesthesia

Figure 1. Indicative plot of how the error “poorly visualized needle” is provided as feedback to a trainee. A distance above 1.5 mm is considered out of plane, i.e. needle is poorly visualized.

2.4. Use Case – Novice Trainee A novice trainee logs onto the training and assessment environment and is given a general introduction on how it is intended to support their learning of the procedure. The trainee is directed to an introductory training session. They are initially guided through a patient scenario and are presented with pre-procedural problems up to the point at which ultrasound imaging begins. The trainee then performs the procedure in the VR-based environment with metrics and error flags constantly monitoring the progress. After each session, formative feedback (such as in Figure 1) is provided to aid the trainee during the learning process. The environment will adapt to their performance and errors by deliberately choosing patient scenarios based on the skill level of the trainee. When the trainee shows a steady increase in performance and error reduction (i.e. meets certain predefined proficiency parameters), the environment will present more challenging scenarios, adapting to the trainee’s skill level. As the trainee progresses, the training and assessment environment provides feedback based on performance over time. This feedback is intended to motivate and help the trainees to systematically align them with the USgABPB’s training objectives through deliberate practice. In parallel with the training program, formal assessment scenarios are performed regularly to determine the progress of the trainee. The outcome of these scenarios is provided to the trainer(s) which will allow them to identify trainees that require targeted tuition.

E. Lövquist et al. / VR-Based Training and Assessment in Ultrasound-Guided Regional Anesthesia 309

3. System Platform The system design is currently being implemented on a VR-based training and assessment platform. The platform utilizes the open-source graphics and haptics API the H3D API (www.h3dapi.org). The H3D API supports a wide range of haptic devices, which allows the platform’s fidelity to be customized based on training needs. In order to present learning material and feedback in the training environment, the user interface (UI) solution Qt (http://qt.nokia.com) is utilized. The haptic viscosity model for the tissue sensations is implemented using the Volume Haptics Toolkit (VHTK) [12]. The platform currently incorporates a simplistic ultrasound representation for core skills acquisition, but is designed to import and utilize pre-acquired ultrasound data from systems such as “The Stradwin 3D Ultrasound Acquisition and Visualization System” (http://mi.eng.cam.ac.uk/~rwp/stradwin) or MedaPhor Ltd’s “ScanTrainer” solution (www.scantrainer.com). The VR-based platform in itself covers most of the significant SHERPA-identified errors. However, some parts of the HTA, e.g., the positioning of patient and positioning of equipment, are not covered by the VR technology due to technical limitations. Instead, the concept of Virtual Patients [13] is applied to train and assess additional parts of the clinical procedure and to contextualize the training around patient scenarios. A virtual patient player extends the VR-based platform. The player is used to render patient scenarios in the form of text and pictures and to handle interactive decision-making by the trainee. This is done before and after technical skills, such as tracking of nerves and needle insertion, are trained and assessed on the VR-based system.

4. Discussion This paper discusses the design of an unattended VR-based training and assessment environment intended to automatically detect and appropriately report clinically significant errors. It is our view that such an environment, if coupled with the ability to improve technical skills using appropriate metrics, will ultimately translate into improved patient benefit and safety. The training and assessment environment has been designed to 1) identify clinically relevant metrics and errors, 2) provide useful feedback based on actual performance and errors and 3) individualize training by adapting to a trainee's previous performance and errors. The existing training analysis (HTA, SHERPA and FMECA) was validated with five independent experts in the procedure. The experts ensured that the overall training analysis covered clinically relevant tasks, errors and remedial strategies. An alternative approach to errors analysis includes retrospective analysis of errors related to adverse patient outcomes [14]. We chose to utilize tools which proactively analyze processes to identify potentially weaknesses where errors are likely to occur, or where the consequences of such errors may be significant. The Institute of Medicine highlighted the use of such techniques to increase the chances of preventing errors and adverse events [15]. The medical experts on the development team informally validated the environment’s face and content validity throughout the design process. The resulting system is currently being evaluated in a randomized controlled trial. The trial will examine if the system’s capacity to capture errors results in a decrease in error

310 E. Lövquist et al. / VR-Based Training and Assessment in Ultrasound-Guided Regional Anesthesia

frequency in the operating room. If the outcome of this trial is positive, the VR-based training and assessment environment for USgABPB will be integrated into formal, curriculum-based training, with the associated benefits of decreasing the incidence of clinical errors and consequently improving patient safety. The integration of a training system into a formal curriculum is crucial for its usefulness as a training tool [16]. Currently, a curriculum for UGRA has been suggested jointly by the American Society of Regional Anesthesia and the European Society of Regional Anaesthesia [17]. The VR-based training and assessment environment is targeted for such curricula.

References [1] L. T. Kohn and M. S. Donaldson (eds), To Err Is Human: Building A Safer Health System, Washington, DC: National Academy Press, 2000. [2] H. R. Clinton and B. Obama, Making Patient Safety the Centerpiece of Medical Liability Reform, New England Journal of Medicine, 354 (2006), 2205-2208. [3] R. Satava, Emerging Trends that Herald the Future of Surgical Simulation, Surgical Clinics of North America , 90:3 (2010), 623-633. [4] A. G. Gallagher, E. M. Ritter, H. Champion, G. Higgins, M. P. Fried, G. Moses, C. D. Smith and R. M. Satava, Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training, Annals of Surgery, 241 (2005), 364-372. [5] Analysis performed by Dr. O’Sullivan and colleagues at the Cork University Hospital (CUH), Ireland, in Spring 2010 as part of the Irish research project “Haystack”. Haystack is funded by the National Digital Research Centre (NDRC), Ireland. [6] J. Annett, K. D. Duncan, R. B. Stammers and M. J. Gray, Task Analysis, Department of Employment Training Information Paper 6, HMSO, London, 1971. [7] D. E. Embry, SHERPA: a systematic human error reduction and prediction approach, Paper presented at the International Topical Meeting on Advances in Human Factors in Nuclear Power Systems, Knoxville, Tennessee, April 21-24, 1986. [8] E. Williams, The use of failure mode effect and criticality analysis in a medication error subcommittee, Hospital Pharmacy, 29, (1994), 331–337. [9] P. M. Asaro, Transforming society by transforming technology: the science and politics of participatory design, Accounting, Management and Information Technologies, 10:4 (2000), 257-290. [10] J. A. Cannon-Bowers, C. Bowers and K. Procci, Optimizing Learning in Surgical Simulations: Guidelines from the Science of Learning and Human Performance, Surgical Clinics of North America, 90:3 (2010), 583-603. [11] K. A. Ericsson, Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains, Academic Medicine, 79 (2004), S70–81. [12] K. Lundin, B. Gudmundsson and A. Ynnerman, General proxy-based haptics for volume visualization, Proceedings of the World Haptics Conference, Pisa, Italy, (2005), 557–560. [13] R. Ellaway, T. Poulton, V. Smothers and P. Greene, Virtual Patients Come of Age, Medical Teacher, 31:8 (2009), 683-684. [14] J. B. Cooper, R. S. Newbower and R. J. Kitz, An analysis of major errors and equipment failures in anesthesia management: considerations for prevention and detection, Anesthesiology, 60:1 (1984), 3442. [15] P. Aspden, J. M. Corrigan, J. Wolcott and S. M. Erickson (eds), Patient Safety: Achieving a New Standard for Care, Washington, DC: The National Academic Press, 2003. In chapter: “Adverse event analysis”, 200-225. [16] S. B. Issenberg, W. C. McGaghie, E. R. Petrusa, et al., Features and uses of high- fidelity medical simulations that lead to effective learning: a BEME systematic review, Medical Teacher, 27:1 (2005), 10–28. [17] B. Sites, V. W. Chan, J. M. Neal, et al., The American Society of Regional Anesthesia and Pain Medicine and the European Society of Regional Anaesthesia and Pain Therapy Joint Committee Recommendations for Education and Training in Ultrasound-Guided Regional Anesthesia, Regional Anesthesia and Pain Medicine, 34 (2009), 40-46.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-311

311

Real-Time Electrocautery Simulation for Laparoscopic Surgical Environments Zhonghua LU a, b, Venkata Sreekanth ARIKATLAa, Dingfang CHENb, Suvranu DEa,1 a Rensselaer Polytechnic Institute, Troy, NY b Wuhan University of Technology, Wuhan, China

Abstract. We present a novel real-time technique for cutting during electrocautery procedures in surgical training. Our algorithm is based on cauterizing the part of the tissue that exceeds the critical vaporization temperature. The resulting topology changes due to cutting are accounted for in real-time. Results presented for the overall electrocautery cutting algorithm show that the real-time costs are minimal and thus allow interactive simulation. Keywords. Electrocautery, surgery simulation, Cutting simulation

Introduction Minimally invasive surgery (MIS) is increasingly becoming the choice for many surgical procedures due to the short hospitalization and quick recovery it implies. However, performing these surgeries requires significant training to develop good hand-eye coordination and dexterity in tool manipulation. Electrocautery, one of the most commonly used surgical procedures in MIS, is used to burn tissue through localized heating with a specialized tool that is heated by high-frequency electric current [1]. In this paper, we propose an algorithm for cutting soft tissue, a part of the electrocautery process. A volumetric tetrahedral elemental mesh is used to simulate the deformation and the heat conduction problems through a finite element method. During cutting, the burnt parts are removed in successive time steps, which look like a progressive tissue cauterizing process. In addition, as the tissue is cauterized, new nodes are created and some of the elements are removed. We have presented the online manipulation of matrix data structures to account for these topology changes in real time. The algorithm has been successfully implemented in a surgical simulation environment.

1. Tools and Methods Electrocautery procedures in MIS require bimanual interaction with the tissue. We model the deformation and heat transfer problems separately. The temperature 1

Corresponding Author: Dr. Suvranu De, Department of Mechanical, Aerospace and Nuclear Engineering, Rensselaer Polytechnic Institute, Email: [email protected]

312

Z. Lu et al. / Real-Time Electrocautery Simulation for Laparoscopic Surgical Environments

information is used to cut or remove some parts of the tissue. Sections below describe the techniques employed at various stages to simulate this electrocautery procedure. 1.1. Deformation and Heat Transfer Modeling As mentioned in the previous section, we decouple the deformation and the heat transfer problems in order to reduce the computational costs. We adopt a co-rotational formulation with linear material model for the deformation update. This results in the following simultaneous set of equations: (1) Where is the displacement, F is the load vector, is the consistent mass is the stiffness matrix and is the damping matrix. matrix, For simulating heat transfer we use the bio-heat equation. After identifying metabolic heat generation rate, determining blood perfusion to be negligible and discretizing using linear tetrahedral elements, we arrive at this simultaneous set of equations: (2) Where

is the heat capacity matrix,

is the heat conductivity matrix, P is the

heat supply vector, T is the temperature vector, and

is the time derivative of T.

1.2. Time Stepping Since both the deformation and the heat transfer simulations are time-dependent we have to use appropriate time stepping schemes. In case of deformation update we adopted Newmark beta time stepping, while for heat simulation, we used central difference. 1.3. Electrocautery Cutting In the present simulation the central idea is to remove the part of the organ geometry that exceeds the critical vaporization temperature of the tissue at each time step. In a dynamic simulation, this is an ongoing process i.e., if the cautery tool is in action, we need to remove some part of the geometry at every time step. This amounts to remeshing locally and forming a global stiffness matrix at every time step. Obviously, this increases the load at every time step making it prohibitive in real-time interactive simulation. In this work we overcome this obstacle by separating the rendering and physics related geometries near the cut area. The main idea is to render the cut surface by removing all the geometry above the vaporization temperature. For display purposes, we need to find the isotherm for the vaporization temperature around the cautery tool tip. Since we use a linear tetrahedral element in modeling heat transfer, we can find the along the edges. Therefore if and are the nodal points on the edges that have temperatures on two ends of the edge and if

,

then the point on the

Z. Lu et al. / Real-Time Electrocautery Simulation for Laparoscopic Surgical Environments

isotherm

along

the

edge

is

313

where

.

Electrocautery generally requires temperatures of 400°C to 600°C to cause a thermal cutting effect [2]. In our simulation the threshold temperature is chosen to be 500°C. For a tetrahedral element, there are five different ways to form an isotherm based . Those cases in which on the number of nodes whose temperatures are higher than either none or all nodal temperatures are higher than in which all the nodal temperatures are above

are complementary. In the case , that elemental contribution is

removed from the heat and 3D deformation global matrices. If either one or three nodes have temperatures higher than , then one of the tetrahedrons or pentahedrons is burnt and is removed for display purposes (see Figure 1). In this case there are three , then the nodes defining the isotherm. If two nodes of any tetrahedron exceed isotherm is defined by four nodes as shown in Figure 1. Similarly, this isotherm is used for display purposes while the elemental contribution is retained.

Figure1: Definition of the isotherms for cases in which one and two nodes exceed critical temperature

If the electrocautery tip moves away, some of the node’s temperature will decrease will move backward. However, due to conductive heat loss, and the isotherm for the cutting procedure is irreversible. To prevent the threshold isothermal plane from moving back, we record the highest temperature of every node in the electrocautery process and use that temperature to get the interpolating points. For real time rendering we render the surface of the model. At the initialization, we set the face property to be either inside or outside for each tetrahedron. If a tetrahedron is cut by the threshold plane, the new faces derived from the outside face should be added to the rendering buffer, and the isothermal plane should also be added as part of the incision to be displayed. Through the aforementioned method, we cannot see any incisions being taken into . This effect in the deformation even when the nodal temperatures are higher than might create some visual artifacts in the simulation. Therefore, we split the nodes if . their temperature is higher than

314

Z. Lu et al. / Real-Time Electrocautery Simulation for Laparoscopic Surgical Environments

Figure 2: Overview of the overall simulation

1.4. Modifying Sparse Structures As described above, during the electrocautery process new nodes might be created by node splitting or some nodes might be removed based on the temperature isotherm. When a new node is added by the splitting procedure, some of the old connections need to be deleted and the new connections or nodes must be established. The FEM matrices are block-sparse matrices; we apply a Block Compressed Row (BCR) format to save their storages. If the node is split and if one of the edges linking nodes i and j is deleted, then the 3x3 blocks Aij and Aji need to be removed in the BCR structure. Similarly, if a new edge is added, we have to expand the arrays to add an additional 3x3 block. An overview of the overall procedure is shown in Figure 2. A total of four threads were used; one each for rendering, collision detection, deformation and temperature update, and force rendering.

2. Results In the present work, the simulations were performed on an Intel Core 2 Quad Q9550 CPU machine with NVIDIA GTX 280 graphic cards. Two PHANTOM® OmniTM haptic interface devices were used for force feedback. In the present simulation, we use 2.7x105Pa for Young’s modulus, 0.4 for Poisson’s ratio, 0.512 for thermal conductivity and 3600 J/ kg·K for specific heat [3].

Process Deformation Heat transfer Matrix rebuilding

Time (milliseconds) 12.54 1.41 67.51 (Direct method) 26.64 (Our method) 2.53 (Speed-up ratio)

Table 1: Timings from various stages of the electrocautery process during the cutting of a ligament.

Z. Lu et al. / Real-Time Electrocautery Simulation for Laparoscopic Surgical Environments

315

During runtime we solved the deformation and heat transfer problems separately using a conjugate gradient algorithm with a tolerance of 10-4. The deformation update consumes more time compared to the heat solution as the degree of freedom is three times as much. All our simulations ran at 50Hz for physics, 60Hz for rendering. The cutting through cautery is implemented using the algorithm described in previous section. The main cost in the cutting procedure is in nodal splitting and the resulting modification of the sparse structures. Table 1 shows the costs involved in various components of the simulation. For comparison, the time for the stiffness matrix reevaluation using the direct method is also shown. It can be seen that using our method. The speed was more than doubled. Figure 3 shows the screenshot and the corresponding color map of temperatures during the cutting of a ligament using electrocautery.

(a)

(b) Figure 3: (a) Snapshot of the electrocautery cutting simulation (b) Color map of the corresponding temperature distribution

316

Z. Lu et al. / Real-Time Electrocautery Simulation for Laparoscopic Surgical Environments

3. Conclusion/Discussion We have developed a paradigm to simulate electrocautery procedures for surgical simulation. We avoided the expensive modification of sparse structures at every time step by separating geometries for rendering and the underlying physics. Finally, it should be noted that the detailed electro-thermo mechanics of the electrocautery procedure is highly complex and tissue dissection involves prediction of damage initiation and final rupture of the tissue. A complete physics-based description of this process must be developed through further experimentation and modeling before it can be implemented in a surgical simulation framework.

Acknowledgement This work was supported by grant R01 EB005807 from NIH/NIBIB.

References [1] [2] [3]

Pearce, J. A., 1986, Electrosurgery, Wiley, New York. Kaplan, L. Uribe, J. W. Sasken, H. Markarian, G. The Acute Effects of Radiofrequency Energy in Articular Cartilage: An In Vitro Study. ARTHROSCOPY, 2000, VOL 16; PART 1, pages 2-5 Carter FJ, Frank TG, Davies PJ, et al. Measurements and modelling of the compliance of human and porcine organs. Medical Image Analysis 2001; 5:231-236.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-317

317

Guidewire and Catheter Behavioural Simulation Vincent LUBOZa,1, Jianhua ZHAIb, Tolu ODETOYINBOc, Peter LITTLERc, Derek GOULDc, Thien HOWb, and Fernando BELLOa a Department of Surgery and Cancer, Imperial College London, UK {v.luboz, f.bello}@imperial.ac.uk b University of Liverpool, School of Clinical Sciences, Liverpool, UK. c Royal Liverpool Hospital, Liverpool, UK

Abstract. Guidewire and catheter manipulation is a core skill in endovascular interventional radiology. It is usually acquired in an apprenticeship on patients, but this training is expensive and risky. Simulation offers an efficient alternative for core skills training, though the instrument complex behaviour requires accurate replication. This paper reviews the mass-spring model used to simulate seven guidewires and three catheters, and the matching with their real world counterparts by tuning our model’s bending coefficient, which allows replication of the instrument flexibility. This coefficient was matched through computed tomography imaging of a vascular phantom in which each instrument was inserted and manipulated. With an average distance of 2.27mm (standard deviation: 1.54) between real and virtual instruments, our representation showed realistic behaviour. Keywords. Guidewire/catheter, mass-spring model, endovascular simulation

Introduction Interventional Radiology (IR) consists of minimally invasive diagnostic and therapeutic procedures guided by medical imaging. This anatomical imaging provides a road map of complex organ or vascular systems to allow accurate navigation and positioning of instruments (needles, catheters, and guide-wires) to treat a range of complex pathologies. In this paper, we focus on vascular interventional radiology. A high level of proficiency is required for safe, effective performance of these IR procedures, which use fine motor skills for instrument navigation within a patient’s vascular anatomy. These skills rely on key tactile and visuo-spatial cues, and are traditionally learnt during deliberate practice in an apprenticeship in patients. The past decade has seen developing interest in the benefits of using medical simulation for training in a range of specialities that includes IR [1-4]. The exacting and unforgiving nature of vascular IR procedures in patients requires physically accurate, real-time, virtual instrument models for ‘real world’ replication of the behaviour and response of instruments within virtual blood vessels. A wide range of IR instruments is available in existing commercial vascular simulators (e.g. Mentice; Simbionix) along with a range of pathologies and datasets. 1

Corresponding Author.

318

V. Luboz et al. / Guidewire and Catheter Behavioural Simulation

Nevertheless, perfection has yet to be attained and current simulations lack ultimate realism of instrument behaviour with potentially inaccurate core skills training [3]. Improving the accuracy of instrument behaviour is the focus of several research groups. A non linear deformable beam model resulting in an accurate simulation was introduced in [2]. [5] makes use of Cosserat models to simulate a guidewire as a set of straight, non-bendable, incompressible beams with perfect torque control, resulting in accurate but not real time simulation. Recently, we proposed a mass-spring model to simulate the instruments’ behaviour [6], showing the possibility to navigate a whole guidewire in real time and with visually correct accuracy in a very detailed vasculature. The model used in our paper is inspired by [6] due to its simplicity and its computational speed. This model has been validated for several guidewires in [7]. Here we focus on improving the realism of the simulation by integrating the real shape of the instruments and parameterizing them to fit their real behaviour in terms of flexural modulus. The two other main contributions of this paper in comparison to [7] are the integration of several catheters, and the improvements of the tests evaluating the difference between real and virtual instruments, now based on a CT scan analysis.

1. Methods Training of the core skills involved in vessel access using the Seldinger technique, followed by the manipulation of catheters and guidewires, is the ultimate goal of our framework. In this paper, we concentrate on guidewire and catheter interactions only. 1.1. Instrument Modeling The instruments are represented by a mass-spring model similar to the guidewire model of [6]. The same model is used to simulate the catheters. This model is composed of a set of particles (X0..XN) connected by stiff springs of equal length (λ). Its tip consists of the first (X0 ..Xtlength) particles of the instrument whereas the body is formed using subsequent particles (Xtlength+1..XN). The tip is distinguished from its body to model different tip shapes between instruments. When the user moves the instrument, the motion detected by the haptic device (or the keyboard) is sent to the simulator. Two different motions are possible: translation or rotation. When the instrument is pulled, the translation is applied along the body axis to all the body particles starting with the last one (at the proximal end) and then propagated to the tip particles, in an intuitive “bottom to tip” approach. A “tip to bottom” approach is used when the instrument is pushed to enhance the mass-spring model stability. The rotation is done in an intuitive “bottom to tip” approach. In either case, a collision detection step is performed to ensure that the instrument is not going through the vasculature. This verification is computed every time step for every particle of the instrument to provide a realistic rendering of the scene. We did not modify the algorithm presented in [6] since it proved to be efficient and accurate. At each time step, four forces are applied to all the particles of the instrument: external force, spring force, and 2D and 3D bending forces. It ensures that all the particles stay inside the vasculature, especially in case of a collision. The external force moves the particle away from the vascular wall triangles. This force is repeatedly applied until the particle is no longer colliding. For any particle, it is given by the following equation:

V. Luboz et al. / Guidewire and Catheter Behavioural Simulation

319

(1) is the average normal of the colliding Where ke is the external force coefficient, triangle, d is the distance from the closest colliding triangle, and ε is a constant minimum displacement applied to the particle to resolve the collision. The spring force ensures spring length remains close to λ and is given by:

(2) Where ks is the spring force coefficient, l{i-1, i+1} is the distance between the particle i and its neighbours, and v{i-1, i+1} is the vector between i and its neighbours.

Figure 1. The 2D bending force applied at Xi leads to a translation of Xi-1 along u{i-1}.

The instrument shape is conserved by the 2D bending force, using a bias angle conservation constraint. For the ith particle, it is defined as: (3) Where kb is the bending factor translating the instrument flexibility, θi is the bending angle for the ith particle, θb is the bias bending angle for the ith particle, and u{i1} is the vector perpendicular to the v1 vector lying on the same surface as vector v2 (Figure 1). The instrument’s body being initially straight, its particles’ bias angle is null. This force therefore contributes to straighten them. On the contrary, the bias angle will tend to keep the initial shape of the different guidewires since it is not null. Because of the nature of the 2D bending force, its effect is only along the plane defined by v1 and v2. Since external forces might create a deviation of the tip particles out of their original plane, a fourth force was added to the collision response: the 3D bending force. It constrains all tip particles in the same plane as defined in the following equation for the ith particle: (4) Where kb3D is the bending factor, α is the bending angle between the body plane π2 and the tip plane π1, w{i-1} is the vector whose direction is the same as n2, π2 plane’s normal, but which is attached at the i-1th particle. The body plane is determined by the first two particles of the tip (Xtlenght and Xtlenght-1) and the first body particle (Xtlenght+1). The tip plane is composed by the particle where the 3D bending force is being defined and the two neighbouring particles.

320

V. Luboz et al. / Guidewire and Catheter Behavioural Simulation

1.2. Instrument Interactions During an IR procedure, an operator manipulates catheters and guidewires such that they interact with each other to perform vascular navigation, branch vessel selection and catheterisation. Such interactions were not considered in [6], which focused only on the guidewire. To realistically simulate these interactions we have set the following priorities. For each motion of the catheter, the guidewire relocates according to catheter position, as it slides in its hollow tube. The only exception is when the guidewire has part of its particles outside the catheter. In this case, a motion of the catheter will make the guidewire particles inside the catheter follow it, but the guidewire particles outside the catheter are just updated to satisfy the collision response forces, especially the spring and bending forces. When the guidewire is moving, it is also influencing the catheter’s position depending on the catheter’s flexibility. This complex set of interactions provides realistic navigation of the instruments within vascular anatomy, and allows an operator to select any target or branch vessels. 1.3. Instrument Selection There are a number of different types of guidewires and catheters currently used in IR procedures [8]. Instruments vary widely in length, size, surface coating, material, stiffness, tip shape, etc. In general, guidewires can be classified into three main groups based on their mode of use at different stages of an interventional procedure: access, selection and exchange. Catheters can be separated into two groups, for either diagnostic or therapeutic use. Based on these considerations, seven commonly used guidewires were chosen for building the guidewire simulation models: Fixed Core Straight (Cook Medical Inc., USA), Fixed Core Safe-T-J-Curved (Cook), RosenCurved (Cook), Amplatz Super Stiff (Boston Scientific, USA), Bentson (Boston), Terumo Angled and Terumo Stiff Angled wires (Terumo Corp., Japan). Among these guidewires, the Cook Fixed Core Straight, Cook T-J-curved and Bentson wires were considered representative of access guidewires. Examples of selection guidewires comprised the Angled and Stiff Angled Terumo wires. Amplatz Super Stiff, Cook Rosen-curved and Bentson wires covered properties inherent for exchange purposes. Three diagnostic catheters were chosen: 5F Beacon, 4F and 5F Terumo. 1.4. Evaluation Methodology We first set the values for the four coefficients of our mass-spring model. Because ke, the external force coefficient, is applied equally on each particle, a value of 1 was chosen for the body particles. For the tip particles, a slight attenuation was introduced and their ke was set to 0.9. The spring coefficient, ks, is the same all along the wire, and is set to 1 for every particle. The 3D bending force being applied only to the tip, a kb3D of 1 was applied to the tip and 0 to the body. The last coefficient, kb, is relative to the 2D bending forces. It represents the flexibility of the instruments and has a major impact on their behaviour. To evaluate the bending coefficients for the tip and the body, each guidewire and catheter was inserted by the same operator and at room temperature, in three different positions (common iliac artery bifurcation, aortic bifurcation, left renal artery origin) in a silicone rubber vascular phantom (Elastrat, Geneva, Switzerland; Figure 2) and scanned in a 128 slice, multidetector CT scanner with a resolution of 0.53x0.53x1 mm3.

V. Luboz et al. / Guidewire and Catheter Behavioural Simulation

321

Figure 2. The vascular silicon phantom used for the validation. Three main comparison positions are recorded: common iliac artery bifurcation, aortic bifurcation, left renal artery origin.

The phantom models real patient vessels (abdominal aorta, iliac and common femoral arteries), providing their real dimensions (2mm to 15mm diameter). The 3D position of the real instrument was obtained by threshold segmentation of the CT data sets. The simulated virtual phantom geometry was reconstructed from a CT scan of the real phantom without instrumentation using ITK-snap snake segmentation [9]. Virtual instruments were then inserted up to the same points as attained within the physical model and the positions of their particles were recorded. The Euclidean distance between corresponding points in the real and virtual instruments at 2mm intervals starting from the tip was then computed. Bending coefficients for tip and body were optimized to minimize the distance between real and virtual instruments.

2. Results For each instrument, the best parameters were defined according to the above process. Table 1 summarizes the chosen kb for the body and tip particles, average distance, std dev, and max and min distance, averaged for the three instrument positions (common iliac artery bifurcation, aortic bifurcation, left renal artery origin). Numerical and graphical data demonstrated good correlation between the real and virtual environments. On average, the distance between the real and virtual instruments is 2.27mm with a standard deviation of 1.54 (min 0.44mm, max 5.71mm). A typical example (Figure 3) shows the average distance between the real and virtual Terumo angled guidewires is 2.21mm, with a std dev of 1.3mm (min 0.43mm, max 5.59mm).

Figure 3. Comparison of the path taken by the real (black) and virtual (different colors) Terumo angled guidewires. The insertion point is on the left and the tips on the right. The virtual wire particles are blue for very good agreement (< 0.5mm), green for good agreement (> 0.5, < 1mm), yellow for intermediate agreement (<1, > 2mm), orange for poor agreement (>2, < 3mm) and red for very poor agreement (> 3mm).

322

V. Luboz et al. / Guidewire and Catheter Behavioural Simulation

Table 1. Comparison between the real and virtual instruments based on the distance between corresponding pairs of points. It shows the average distance, the standard deviation and the minimum and maximum distances between the two instruments. All distances are in mm. kb tip

kb body

Average distance)

Standard deviation

Minimum distance

Maximum distance

Terumo angled

0.8

0.8

2.21

1.3

0.43

5.59

Terumo stiff

0.8

1.1

2.63

1.91

0.54

7.45

Amplatz

0.8

1.2

2.43

1.43

0.56

5.48

Bentson

0.7

1

2.88

2.02

0.59

6.68

Cook J

0.9

1

2.05

1.18

0.4

4.93

Cook Straight

0.9

1

2.24

1.48

0.24

5.92

Rosen

1

1.05

2.43

2.37

0.33

7.29

Beacon 5Fr

0.9

1

1.75

1.39

0.46

4.11

Terumo ST 4Fr

0.9

1

2.09

1.12

0.52

4.69

Terumo ST 5Fr

0.9

1

1.94

1.19

0.3

4.93

Instrument name Guidewires

Catheters

Table 2. Distance agreement for each instrument along their length in term of number of particles. The distance range is the same as in Figure 3. The number of particles (Nb. parti.) for each anatomical location is also presented. Nb parti. at iliac artery

Nb parti. at aortic bifurc.

Nb parti. at renal artery

% Very good

% Good

Terumo angled

90

122

173

3

18

35

17

27

Terumo stiff

91

120

170

2

19

34

11

34

Amplatz

83

95

165

1

11

47

16

25

Instrument name

% Interm

% Poor

% Very poor

Guidewires

Bentson

88

119

167

2

16

41

8

34

Cook J

94

127

171

3

14

45

19

20

Cook Straight

89

119

167

6

20

32

15

27

Rosen

93

119

169

2

16

43

13

26

Catheters Beacon 5Fr

67

115

165

5

10

61

15

8

Terumo ST 4Fr

85

119

168

1

12

48

20

18

Terumo ST 5Fr

81

118

168

2

14

59

8

18

Table 2 shows the distance agreement for each instrument along their length and the number of particles at each instrument position. It ranges from very good agreement (< 0.5mm), good agreement (> 0.5mm, < 1mm), intermediate agreement (>1mm < 2mm), poor agreement (>2mm, < 3mm) and very poor agreement (> 3mm). On average, 3% of the particles are in very good agreement, 15% of the particles are in

V. Luboz et al. / Guidewire and Catheter Behavioural Simulation

323

good agreement, 44% of them are in intermediate agreement, 14% are in poor agreement, while 24% are in very poor agreement. 3. Discussion and Conclusion This paper presents the design of realistic IR instruments and their integration within our training environment. Use of realistic physical parameters such as flexural modulus may significantly improve the relevance of the behaviour of virtual IR instruments and, indeed, of the overall training experience, to actual use of real instruments in real patients. Seven guidewires and three catheters were implemented to match the shape and behaviour of their real counterparts. The shape is modelled by the position of the tip particles of each instrument. The behaviour depends essentially on the 2D bending coefficients for the instrument tip and body. The three other coefficients, related to the external force, the spring force and the 3D bending force, are constant for all instruments and therefore have only a minor influence on their flexibility. The instruments’ 2D bending coefficients were estimated through a set of experiments matching performance of these real instruments, in three dimensions within a silicone rubber vascular phantom, with their virtual representations. Results show good correlation with an average distance of 2.27mm between the real and virtual instruments and standard deviation of 1.54mm. These figures highlight the accuracy and realism of our virtual instruments. Some large errors can still be observed, especially at the instrument tip (up to 7.45mm for the Terumo stiff guidewire), but overall there is at least an intermediate agreement for most of the sampled points (on average 62%). The errors may in part relate to minor involuntary rotations that cannot be controlled in the real environment but are ignored or suppressed in the virtual world. Some error will also occur due to frictional forces of the silicon rubber material of the model being higher than those of the intimal lining of real vessels. Finally, the fact that the same bending coefficient is given to all the tip particles or all the body particles may not be realistic enough since the flexibility of the instrument changes along its whole length. Future work should address these sources of error. References [1] [2] [3] [4] [5] [6] [7] [8] [9]

F. Wang, L. Duratti, E. Samur, U. Spaelter and H. Bleuler, A Computer-Based Real-Time Simulation of Interventional Radiology, Eng. in Medicine and Biology Society (2007). C. Duriez, S. Cotin, J. Lenoir, and P. Neumann, New Approcahes to Catheter Navigation for Interventional Radiology Simulation, Comput Aided Surg. (2005), 11(6):300-8. D.A. Gould, J.A. Reekers et al., Simulation Devices in Interventional Radiology: Validation Pending, J. of Vascular and Interventional Radiology (2006), Vol. 17, Iss. 2, 215-216. J. Dankelman, M. Wentink, et al., Does Vitual Reality Training Make Sense in Interventional Radiology? CardioVascular and Interventional Radiology (2004), 27(5), 417-421. T. Alderliesten et al. , Modeling Friction, Intrinsic of Curvature, and Rotation Guide Wires for Simulation of Minimally Invasive Vascular Interventions. IEEE Trans. Biomedical Eng.(2007), 54 (1). V. Luboz, R. Blazewski, D. Gould and F. Bello, Real-time Guidewire Simulation in Complex Vascular Models, The Visual Computer (2009), vol. 25/9, 827-834. V. Luboz, J. Zhai, P. Littler, T. Odetoyinbo, D. Gould, T. How, F. Bello, Endovascular guidewire flexibility simulation. International Symposium on Biomedical Simulation (2010), 171-180. P. Schneider, Endovascular Skills: guidewire and catheter skills for endovascular surgery, 3rd ed, 2008. P.A. Yushkevich, J. Piven, et al., User-guided 3D active contour segmentation of anatomical structures: significantly improved efficiency and reliability. NeuroImage (2006), 31 (3), 1116-1128.

324

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-324

Design and Implementation of a Visual and Haptic Simulator in a Platform for a TEL System in Percutaneuos Orthopedic Surgery a

Vanda LUENGOa , Aurelie LARCHER a and Jérôme TONETTI b Laboratoire Informatique de Grenoble, Université Joseph Fourier, Grenoble, France b CHU de Grenoble, Service d’Orthopédie-Traumatologie, Grenoble, France

Abstract. Within a research project whose aim is to promote the learning of percutaneous operation in orthopedic surgery we design a Technological Enhanced Learning (TEL) system. This project belongs to a multidisciplinary field including computer, orthopedic surgery, medical imaging, didactic and cognitive sciences. The article presents the design principles of TEL with a particular interest in the development of a simulator. This simulator allows a virtual exercise interacting with the learner in visual, temporal and haptic dimension. Keywords. Technology Enhanced Learning, Simulation, orthopedic surgery.

Introduction The TELEOS Project (Technology Enhanced Learning Environment for Orthopaedical Surgery) is principally aiming to promote the learning of percutaneous orthopaedic surgery. There are three types of knowledge that are at stake during this learning activity: declarative, pragmatic (often empirical) and perceptivo-gestural. The objective of the system is to let the learner train himself freely on a percutaneous orthopaedic surgery in order to give him an epistemic feedback according to his actions. The feedback accompanies the subject in the learning process, by provoking reinforcements, destabilisations, hints, scaffolding, etc.

Figure 1. Architecture of the TELEOS Project.

These research works involve the developing of a TEL platform (Figure 1). Trails are produced during the problem solving activity in order to analyse the learner's behaviour.

V. Luengo et al. / Design and Implementation of a Visual and Haptic Simulator

325

The gathered data processing is made by another application entity: the diagnosis agent. This agent consults Bayesian networks in order to make a knowledge diagnosis of the learner's activity. A didactic decision agent is also involved to make an epistemic feedback to the learner in accordance with the diagnosis. Depending on the diagnosis, the feedback can consist in another exercise to do with the simulator, in lessons to consult online or in clinical cases to study. These three knowledge entities were based in a cognitive and didactic analysis of the surgeon activity formalised in one cognitive theoretical model [7]. In this article, we are going to explain how the simulation agent has been developed, this agent being trying to assimilate some characteristics of virtual reality: interaction, immersion, and autonomy.

1. Description of the System In order to encourage the extensibility and the re-usability of the software, we have chosen to take into account two kind of percutaneous operations in the simulator. These operations need a singular gesture and concern a distinct anatomic part: the vertebroplasty and the sacro-iliac screw. The first one is a spinal column operation which consists in injecting cement into a broken or a shrunken vertebra. The second one is a pelvis operation consisting in reducing and fixing the sacrum to the hip-bone. In these two situations, the act can be done if the surgeon manages to locate his tool in the patient's body. To do so, two indicators are available: the radiographies and the pressures felt during the introduction of the tool in the body. These radiographies have needed to be validated after some adjustment on the fluoroscope. Regardless of the simulated operation, the TEL system gives to the learner the opportunity to train himself to practise a surgical operation thanks to several functionalities: Choose the type of patient and the type of operation; Visualize in 3D the tool and the patient's model; Adjust the position and the incidence of the fluoroscopic image intensifier; Draw the cutaneous marking off on the body of the patient's model; Produce and visualize radiographies; Manipulate the surgical tool through a mouse or through haptic interface; Verify the trajectory when it was validated.

Figure 2. Simulator's graphical interface when the user adjusts the fluoroscopic, during the insertion and when the trajectory is validate by the user.

We can see in some of the presented interfaces, two 2D images representing the last two radiographies produced by the user, the 3D model of the patient, and the surgical tool and some graphical interface components such as a button or a cursor, to make some adjustments for the exercise.

326

V. Luengo et al. / Design and Implementation of a Visual and Haptic Simulator

1.1. The 3D Patient's Model and the 2D Patient's Radiography Our objective, in a learning point of view, is to propose a variety of patients’ models which represented different cases. Therefore, for the patient's model, the data used to create the volume comes from data files of real patients that had a scan before being operated. As a consequence, the simulator's 3D model will have a fractured, compressed, or slipped bone. The snapshots in DICOM format are gathered and used for 3D modelling. For the model to reach the highest quality, the scanner type must be a bodyscan, that is to say that all the body must be showed on the snapshots, and no injection must have been made to the patient because it would alter the tissue's aspect. Using these transverse sections and taking the spaces between intra and inter sections into account, the 3D model can be created. To do so, the C++ VTK library (Visualization ToolKit) [5] offers many classes to help designing 3D models. By giving the same contrast to all snapshots from all models, we make sure that there is just one correspondence between the scalar value of the image's pixel and the density. The use of a ray tracing algorithm [4], [2] coupled with an isosurface algorithm allows us to examine the volume and to assign a graphical property to all the voxels whose scalar value corresponds to the founded values, the other voxels not being displayed. From the same model, we can obtain, in the same way, a volume with the entire cutaneous surface or the entire bony surface visible. In the two situations, we assign a particular coloration according to the scalar value corresponding to the searched density. A radiographic image is a grayscale image with a number of transparency scales according to the tissues' density. Ray tracing algorithm [2] particularly meets the requirements to produce a radiographic image. But, contrary to rendering surface skin or bone, the use of an isosurface algorithm is not suitable. We are going to use a composition algorithm, considering different scalar values in order to assign distinct graphic rendering. The radiographic reproduction obtained is a volume, as the model with the aspect of skin or bone also is. In order to obtain the frontal, inlet, outlet, and profile snapshots, we are simply going to change the camera's position, and the camera's focal position from the 3D scene, being the place where the radiographic volume and the tool are, and then we are going to record the rendering image. It is this image which is going to be displayed in the 2D image of the simulator.

2. Towards an Immersive Interaction In our system, pseudonatural immersion consists in setting the learner standing in front of the graphic and haptic interfaces. With the mouse he can modify directly the graphical interface, for example by activating some buttons, or by shifting cursors. The Qt library is required as its signal and slot system simplifies the factual management. The user can also change the 3D scene using the VTK library to calculate in real time the 3D objects' transformations. The graphical interface modifications can alter the 2D and 3D result, like, for example, when the user makes a 2D radiography. With the haptic interface, the user physically modifies the stylus' position and orientation, as he would do manipulating a surgical instrument during an operation. The rotations and translations undergone by the stylus are directly retransmitted visually in the 3D graphic rendering of the simulator. At the same time, according to the body's density, a force feedback is produced by the device. As a consequence, we differentiate the

V. Luengo et al. / Design and Implementation of a Visual and Haptic Simulator

327

graphic rendering from the haptic rendering. A timer is in charge of the realization of these two renderings. The visual and tactile result seems fluent for the user and does not imply visible latency. The user's autonomy rests in his capacity to coordinate his perceptions and his actions during the interaction with others entities' process [6]. Our goal being to enable the learner to train himself freely and to allow him to make mistakes, the concept of autonomy is a crucial characteristic of our simulator.

3. Towards an Epistemic Retroaction The original approach of our TEL is based on the interest of producing epistemic retroactions [3]. From knowledge elicitation, experimentations are developed which generates different types of experimental data. This data is processed through Bayesian network. The goal is to inform the Bayesian networks, which is used by the diagnosis agent to set up a report according to the learner's exercise and by the didactic decision agent to evaluate the feedback method and content. From the simulator's conception point of view, it is essential to take the concept of trace into account for the diagnosis to be made. We have seen that the 3D model was made from a real patient's DICOM files. This step belongs to a unique procedure for each model of patient. It also includes segmentation steps used to find traces. The knowledge model underlines the anatomic parts that need to be segmented in subvolume or in interest point in order to record the interest point's position in a text file. The goal is to make some areas identifiable and to take them into account in the kinematics of the patient's gesture. For each new exercise, whether it is Sacro-iliac screw or vertebroplasty, the user selects the 3D model that is going to be used for the exercise. When a user realizes an exercise, a file containing the generated traces is created. The traces include information about the model chosen, and about the user's data, all dynamically generated by some events. Each significant action of the user will generate traces. We consider two kinds of events, the direct events, directly related to the user's action on the graphical interface, and the indirect events, which are related to the model's state. For the direct events, we associate the user's interaction on the graphical interface with an event generating traces. Most of the characteristic actions can be identified as such thanks to a direct action of the learner on the graphical interface. In other situations, the state of the collision between the tool and the model will be studied, particularly the cutaneous and the bony entry point. To locate these two events, we use the voxel's scalar value corresponding to the collision point between the tool and the volume. This scalar value is known all along the exercise. For example, the first time this value exceed zero, the tool get in contact with the body, this corresponds to the cutaneous entry point. In the same way, the first time this value is equal or superior to the value corresponding to a cortical bone's density, this is the bony entry point. When a significant action from the user is detected, data relative to the user is recorded. These are elements that could have been altered by the user. It can be adjustments, force exerted, speed, but also, more generally, chronological order or redundancy of the actions. For example, the learner can adjust the fluoroscope's position and its incidence rays, this action altering directly the radiographies. Thus, for each radiography, the simulator records the position of the camera and the focal. All the information, with the model's data about some interest points could be used by the

328

V. Luengo et al. / Design and Implementation of a Visual and Haptic Simulator

diagnosis agent to determine the knowledge required during the modifications made by the learner.

4. Evaluation and Discussion The current version of the TEL system includes a finalised version of the simulator and intermediate versions of Bayesian networks, diagnosis and didactic decision agents. Our architecture and methodology allow us a progressive validation. We specify and validate our computer models in interaction with two kinds of experts (medical and didactical).The simulator's functions reliability has been demonstrated by the methods used for its conception: To train oneself using real data, the used data's coherency is guaranteed by the data acquisition process for the construction of 3D models. To evaluate intelligible radiographies, the principle of radiography generation guarantees the homogeneity of the 2D rendering, in spite of the variability of the models. To insure a perceptual coherence, the graphic rendering is in accordance with the haptic rendering. Our first validations, with the medical’ experts, show that, for the mouse mode, the usability and correctness properties seem in accordance with the expected ones, i.e. for learning use. For the haptic mode, these fits test shows some correctness problems due our physical device. Indeed, the needle added is not perfectly aligned with the Ommi’ pen. Consequently there are a gap between the physical model and the 3D model. Otherwise, the preliminary evaluations with the didactical experts show that the completeness factor seem in accordance with the learning scenarios proposed. These validations allow us to install the stable version in the hospital. The system was used in June 2010 be used by ten students and currently we evaluate it in ergonomic and learning points of views. The first results are in concordance with the medical experts validations, i.e. the visual feedback are acceptable but it is necessary to work in an improved version of the hardware device in order to have better results with the added needle.

References [1] Ceaux, E., L. Vadcard, M. Dubois, et V. Luengo. «Designing a learning environment in percutaneous surgery: models of knowledge, gesture and learning situations.» European Association for Research on Learning and Instruction. Amsterdam, 2009. [2] Levoy, M. « Efficient ray tracing of volume data,,.» ACM Transactions on Graphics. Stanford University, 1990. 245-261. [3] Luengo, V. «Take into account knowledge constraints for TEL environments design in medical education.» International Conference on Advanced Learning Technologies. Santander: Springer, 2008. 5 pages. [4] Roth, S.D. « Ray casting for modeling solids.» Computer Graphics and Image Processing, 1982: 109144. [5] Schroeder, W., K. Martin, et B. Lorensen Schroeder. The Visualization Toolkit: An Object Oriented Approach to3D Graphics 3rd Edition. Kitware, Inc. Publisher., 2003. [6] Tisseau, J. Réalité virtuelle : autonomie in virtuo. Rennes, France: Thèse en informatique, Université de Rennes I, 2001. [7] Vadcard, L., et V. Luengo. «Réduire l'écart entre formation théorique et pratique en chirurgie : conception d'un EIAH,.» Environnements informatiques pour l'Apprentissage Humain. Montpellier: INRP, 2005. 129-140.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-329

329

Computational Modeling of Human Head Electromagnetics for Source Localization of Milliscale Brain Dynamics Allen D. MALONYa,1, Adnan SALMAN b, Sergei TUROVETS c, Don TUCKER c, Vasily VOLKOV d, Kai LI c, Jung Eun SONG c, Scott BIERSDORFF b, Colin DAVEY c, Chris HOGE b, and David HAMMOND b a Dept. Computer and Information Science, University of Oregon b Neuroinformatics Center, University of Oregon c Electrical Geodesics, Incorportated d Dept. Mathematics and Mechanics, Belarusian State University

Abstract. Understanding the milliscale (temporal and spatial) dynamics of the human brain activity requires high-resolution modeling of head electromagnetics and source localization of EEG data. We have developed an automated environment to construct individualized computational head models from image segmentation and to estimate conductivity parameters using electrical impedance tomography methods. Algorithms incorporating tissue inhomogeneity and impedance anisotropy in electromagnetics forward simulations have been developed and parallelized. The paper reports on the application of the environment in the processing of realistic head models, including conductivity inverse estimation and lead field generation for use in EEG source analysis. Keywords. Electromagnetics, head modeling, brain dynamics, EEG, localization.

Introduction Advances in human brain science have been closely linked with new developments in neuroimaging technology. Indeed, the integration of psychological behavior with neural evidence in cognitive neuroscience research has led to fundamental insights of how the brain functions and manifests our physical and mental reality. However, in any empirical science, it is the resolution and precision of measurement instruments that inexorably define the leading edge of scientific discovery. Human neuroscience is no exception. Brain activity takes place at millisecond temporal and millimeter spatial scales through the reentrant, bidirectional interactions of functional neural networks distributed throughout the cortex and interconnected by a complex network of white matter fibers. Unfortunately, current non-invasive neuroimaging instruments are unable to observe dynamic brain operation at these milliscales. Electromagnetic measures (electroencephalography (EEG), magnetoencephalography (MEG)) provide high temporal resolution (≤1 msec), but their spatial resolution lacks localization of neural source activity. Hemodynamic measures (functional magnetic resonance 1

Corresponding Author.

330

A.D. Malony et al. / Computational Modeling of Human Head Electromagnetics

imaging (fMRI), positron emission tomography (PET)) have good 3D spatial resolution 1mm3, but poor temporal resolution on the order of seconds. Our research goal for the last six years has been to create an anatomicallyconstrained spatiotemporally-optimized neuroimaging (ACSON) methodology to improve the source localization of dense-array EEG (dEEG). Anatomical constraints include high-resolution three-dimensional segmentation of an individual's head tissues, identification of head tissue conductivities, alignment of source generator dipoles with the individual's cortical surface, and interconnection of cortical regions through the white matter tracts. Using these constraints, the ACSON technology constructs a fullphysics computational model of an individual's head electromagnetics and uses this model to map measured EEG scalp potentials to their cortical sources.

1. Methods Modern dense-array EEG (dEEG) technology, such as the Geodesic Sensor Net [19] from Electrical Geodesics, Inc. (EGI) shown in Figure 1(left), can measure micro-volt potentials on the human scalp at up to 256 sensors every 1 msec or less. EEG signals are the consequence of current dipoles associated with postsynaptic activities of neuronal cells. A single postsynaptic potential produces a current-dipole moment on the order of 20 fAm (femtoampere × meter) [9]. A 10 mm2 patch of the cortex surface contains approximately 100,000 neurons with thousands of synapses per neuron. At least 10 nAm is required to detect extracellular fields, and measurable EEG signals with a good signal-to-noise ratio require tens of millions of simultaneously activated synapses. As seen in Figure 1 (right), cortical neurons are arranged parallel to each other and point perpendicular to the cortical surface. It is this structural arrangement that allows currents from groups of thousands of neurons to accumulate and generate an equivalent current dipole for a cortex surface region. Therefore, scalp potentials measured by dEEG can be modeled by the combined electrical potentials (called lead fields) produced by up to 10,000 or more cortex patches. That is the good news. The bad news is that the scalp potentials are a linear superposition of all the distributed source lead fields and the individual EEG contributors (i.e., the distribute source dipoles) must be disentangled to determine the dynamics of each brain region.

Figure 1. (Left) EGI 256-channel Geodesic Sensor Net for dEEG recording and topographical potential maps showing epileptic spike wave progression between 110-310 msec with 10 msec samples. (Right) Neuronal current flows perpendicular to the cortex and creates dipole fields. Because of cortex folding, these fields can be radial, tangential, and oblique in orientation.

Localization Model. The general distributed source localization problem can be stated as follows: Φ = KS + E, where Φ=[φ1,...,φNt] are Ne measured EEG signals over

A.D. Malony et al. / Computational Modeling of Human Head Electromagnetics

331

Nt time (NexNt), K is the lead field matrix (LFM) linking Ns current sources to their electrical potential (NexNs), S=[s1,...,sNt] are the current source values over time (NsxNt), and E is error over time. Since the only variables are the source dipole magnitudes S, their solution is a classic linear inverse problem obtained by inverting Φ. Unfortunately, NsNe, making the problem ill-posed. Methods for solving the underdetermined distributed source inverse problem apply minimum norm estimates and their generalization with various regularization schemes to overcome the ill-posed nature of the problem [8,13,14]. No matter how sophisticated the inverse technique, they all depend on determining the forward projection of current dipoles with unit magnitudes to scalp electrical potentials at known sensor locations (i.e., the lead field matrix K). Building K requires a model of the head electromagnetics. Electromagnetics Model. Given a volume conductor Ω with an arbitrary shape and ΓΩ as its boundary, a current density within the volume induces electric and magnetic fields E and B that can be measured on the conductor surface. If the conductivities σ and the electrical current sources S are known, the electric and magnetic fields inside the volume are fully described by Maxwell’s equations. Thus, the electrical forward problem for the human head can be stated as follows: given the positions and magnitudes of neuronal current sources (modeled as distributed dipoles), as well as geometry and electrical conductivity of the head volume Ω, calculate the distribution of the electrical potential on the surface of the head (scalp) ΓΩ. Mathematically, it means solving the linear Poisson equation: ∇ · σ(x, y, z)∇φ(x, y, z) = S in Ω with no-flux Neumann boundary conditions on the scalp: σ(∇φ) · n = 0. Here n is the normal to ΓΩ, σ = σij (x, y, z) is an inhomogeneous tensor of the head tissues conductivity and S is the source current; if the head tissues are considered to be isotropic, σ is a scalar function of (x, y, z), and — when they are orthotropic, σ is a diagonal tensor with off-diagonal — components σij =0, i≠j. Conductivity Inverse Model. If the head tissue conductivities are not known, it is necessary to solve the conductivity inverse problem by applying a general tomographic structure with a known current source, in this case current injected into the head at the scalp surface (this substitutes for neuronal current sources). From an assumed set of the average head tissue conductivities, σij, and given an injection current configuration, S, it is possible to predict the set of potential measurement values, φp, given a forward model, F, of head electromagnetics as the nonlinear functional by solving the Poisson equation above: φp = F(σij(x,y,z)). Once an appropriate objective function describing the difference between the measured scalp potentials, V, and the predicted potentials (at the sensor locations), φp, is defined (e.g., least square norm), and a search for the global minimum is undertaken using advanced nonlinear optimization algorithms [10,15]. When head tissue conductivities are determined, the forward model can be used to create the lead field matrix K by individually activating each current dipoles with unit magnitude and calculating the scalp electrical potentials at the sensor locations. With the LFM formed, it is then possible to solve for the spatiotemporal source dipole magnitudes S given a dEEG waveform.

2. ACSON Design The most critical component for source localization of dEEG measurements is the computational modeling of the electromagnetics of each subject. To build an

332

A.D. Malony et al. / Computational Modeling of Human Head Electromagnetics

electromagnetics head model of the highest quality for an individual requires accurate anatomical constraints and biophysical parameters: High-resolution segmentation of head tissues. Various imaging methods (e.g., magnetic resonance imaging (MRI) and computerized axial tomography (CAT)) can provide volumetric data of the human head. Since the biophysical properties of each tissue are different and we want to employ quantitative (as opposed to qualitative pixel-to-pixel) piece-wise constant tomographic reconstruction, image segmentation is necessary for modeling. The physical geometry of the segmented tissues forms the basis for the 3D computational model. Determination of tissue conductivities. The human head tissues are inhomogeneous (different tissues have different conductivities) and anisotropic (conductivity can change with respect to orientation and other factors). None of the internal head tissues can be measured directly and noninvasively. They must be determined through bounded electrical impedance tomography (bEIT) and inverse modeling [4,15,16,17,20,21,22]. Cortex surface extraction and tesselation. To build a lead field matrix, dipole generators must be place at locations normal to the cortex surface. Cortex tesselation creates regions for dipole placement. Our research has produced methods and technologies to address these requirements. The ACSON environment shown in Figure 2 integrates the tools in a processing workflow that inputs head imagery (MRI, CT), bEIT data, and EEG sensor registration information and generates automatically accurate LFMs for use in source localization

Figure 2. The ACSON framework supports a workflow of MRI/CT image processing and electromagnetics modeling to deliver a lead field matrix for a single individual to use in source localization. The brain images on the right portray scalp EEG source-mapped to cortex locations.

3. Results The ACSON environment implements all the head modeling capabilities necessary for high-resolution source localization, but it has never been used until now to produce a

A.D. Malony et al. / Computational Modeling of Human Head Electromagnetics

333

real head model and LFM for an individual that can be applied in source localization. We selected Dr. Colin Holmes (a.k.a. “colin27” in the Montreal Neurological Institute (MNI) BrainWeb database [2]) for this purpose. The MNI wanted to define a brain representative of the standard adult male population. They took 250 normal MRI scans, scaled landmarks to equivalent positions on the Talairach atlas [18], and averaged them with 55 additional registered images to create the “MNI305” dataset. In addition, one of the MNI lab members (Dr. Holmes) was scanned 27 times, and the scans were coregistered and averaged to create a very high detail MRI dataset of one brain. When compared to MNI305, it turned out that Dr. Holmes’ brain was (is) very close to the average head standard! While colin27 provides the necessary MRI data for segmentation, ACSON also requires bEIT scans. Luckily, Dr. Holmes has been a longtime collaborator with our group. Last year, he agreed to have 64 bEIT scans made. 3.1. Head Electromagnetics Forward Solver. The ADI and VAI forward solution methods for electromagnetic should first be validated with respect to a known solution. The source localization field has long used a concentric k-shell sphere model (k=3,4) as a theoretical standard of reference (each shell represents a head tissue), since analytical solutions are known for the isotropic and anisotropic case [3,5]. We created a 4-sphere testcase with 100x100x100 voxels and achieved a near-perfect correspondence between the theoretical isotropic and ADI results for a set of shell conductivities. Analytical solutions for spherical anisotropic models [3] are also available for VAI validation. We achieved very good accuracy with respect to the spherical model in both cases, lending strong confirmation that the algorithm is working properly. Based on these findings, the colin27 MRI dataset was segmented at (2mm)3 and 1mm3 resolutions into five tissue: scalp, skull, CSF, gray matter, and white matter. We built ADI and VAI head models and computed a forward solution for each resolution case for known conductivities and current sources. These models were evaluated relative to each other and then used for conductivity inverse and lead field calculations. 3.2. Conductivity Inverse Solution The ADI and VAI forward solvers for electromagnetic head modeling are the core computational components for the conductivity inverse and lead field matrix calculations. The conductivity inverse problem will need to process the bEIT measurements for up to 64 current injection pairs in the general case. Depending on the number of conductivity unknowns, each conductivity search for a single pair will require many thousands of forward solutions to be generated. Placement of current injection points is important to maximize the bEIT measurement value. Running the full complement of pairs enables the solution distribution to be better characterized. For all of our experiments, we set the number of tissue conductivity parameters to three: scalp, skull, and brain. Using the 1mm3 colin27 head model, a simulated annealing optimization process was applied to search for optimum values for all 64 EIT pairs. Histogram plots of conductivity solutions for all pairs were fitted with a normal distribution to determine mean and standard deviation. While other groups have reported research results for human head modeling and conductivity analysis (see [1,6,11,12]), our results are impressive because they are the first results in the field determined by dense array bEIT scanning, high-resolution subject-specific MRI/CT

334

A.D. Malony et al. / Computational Modeling of Human Head Electromagnetics

based FDM of the human head, and simultaneous 3D search in the space of unknown conductivities. The derived brain/skull resistivity ratio is confirmed to be in the 1:20 to 1:30 range reported by other research groups [7,23]. 3.3. Lead Field Matrix Generation Once tissue conductivity estimates are determined, they can be used to calculate the lead field for all current dipoles of interest. Because the ACSON methodology is based in finite difference modeling, it is necessary to represent the dipoles normal to the cortex surface as vector triplets in x, y, z whose weighted combination determines the normal vector. The consequence is that three forward solves must be run, one for each axis orientation, for every dipole in three-space. We created an isotropic LFM and an anisotropic LFM for colin27 based on 4,836 axis dipoles. This required 9,672 forward solutions to be computed (half for ADI, half for VAI) by activating only one dipole and calculating the scalp projection. For each projection, we capture the value for 1,925 potential sensor locations. Thus, each LFM is 4836 x 1925 in size. 3.4. Source Localization Our efforts at building the most accurate electromagnetics head model culminate in the use of the LFM for source localization. We created an anisotropic LFM from a 1mm3 head model for 979 dipoles at 8mm spacing (2937 axis dipoles). For each dipole, we chose the LFM column representing that dipole’s scalp EEG projection at 1925 potential sensors locations and input the values for source localization. Magnitudes for all the dipoles were computed using sLORETA [14] and the one with the maximum intensity was determined and the 3D distance from the “true” dipole measured. Even with a noise level of 10%, the maximum magnitude dipole source localized with a anisotropic LFM is within 6.37mm of a 8mm spaced target dipole. The isotropic LFM is significantly worse. The bottom line is that modeling anisotropy in human head electromagnetics simulation is important for improving the accuracy of linear inverse distributed source solutions.

4. Conclusion We have created the ACSON methodology and environment to address one of the most challenging problems in human neuroimaging today – observing the high-resolution spatiotemporal dynamics of a person’s brain activity noninvasively. If such a capability existed, it would significantly advance neurological research and clinical applications, providing a powerful tool to study neural mechanisms of sensory/motor and cognitive function and plasticity, as well as improving neuromonitoring and neurorehabilitation for epilepsy, stroke, and traumatic brain injury. Our work provides an initial demonstration of the utility of full-physics modeling of human head electromagnetics and accurate head tissue conductivity assessment in improving the accuracy of electrical source localization. The ACSON modeling methods have been validated with analytical solutions and experimental results confirming prior research findings in the field.

A.D. Malony et al. / Computational Modeling of Human Head Electromagnetics

335

Acknowledgment. This work was supported by a contract from the Department of Defense, Telemedicine Advanced Technology Research Center (TATRC).

References [1] M.Clerc, G.Adde, J.Kybic, T.Papadopoulo, J.-M.Badier, In vivo conductivity estimation with symmetric boundary elements, International Conference on Bioelectromagnetism, May 2005. [2] C. Cocosco, V. Kollokian, R. Kwan, G. Pike, A. Evans, Brainweb: Online interface to a 3D MRI simulated brain database, NeuroImage, 5:425, 1997. [3] J. de Munck, T. Faes, A. Hermans, R. Heethaar, A parametric method to resolve the ill-posed nature of the EIT reconstruction problem: a simulation study, Annals of the New York Academy of Sciences, 873:440–453, 1999. [4] B. Esler, T. Lyons, S. Turovets, D. Tucker, Instrumentation for low frequency studies of the human head and its validation in phantom experiments, International Conference on Electrical Bioimpedance, April 2010. [5] T. Ferree, J. Eriksen, D. Tucker, Regional head tissue conductivity estimation for improved EEG analysis, IEEE Transactions on Biomedical Engineering, 47(12):1584–1592, 2000. [6] S.Goncalves, et al., The application of electrical impedance tomography to reduce systematic errors in the EEG inverse problem: a simulation study, Physiological Measurement, 21(3):379–393, 2000. [7] S. Goncalves, et al., In vivo measurement of the brain and skull resistivities using an EIT-based method and realistic models for the head, IEEE Transactions on Biomedical Engineering, 50(6):754–767, June 2003. [8] R. Greenblatt, A. Ossadtchi, M. Pieger, Local linear estimators for the bioelectromagnetic inverse problem, IEEE Transactions on Signal Processing, 53(9):3403–3412, Sept. 2005. [9] M. Hamaläinen, J. Sarvas, Realistic conductivity geometry model of the human head for interpretation of neuromagnetic data, IEEE Transactions on Biomedical Engineering, 36:165–171, Feb 1989. [10] S. Kirkpatrick, C. Gelatt, M. Vecchi, Optimization by simulated annealing, Science, 4598:671–680, May 1983. [11] J. Meijs, O. Weier, M. Peters, A. van Oosterom, On the numerical accuracy of the boundary element method, IEEE Transactions on Biomedical Engineering, 36:1038–1049, 1989. [12] T. Oostendorp, J. Delbeke, D. Stegeman, The conductivity of the human skull: results of in vivo and in vitro measurements, IEEE Transactions on Biomedical Engineering, 47(11):1487–1492, 2000. [13] R. Pascual-Marqui, Review of methods for solving the EEG inverse problem, International Journal of Bioelectromagnetism, 1(1):75–86, 1999. [14] R. Pascual-Marqui, Standardized low resolution brain electromagnetic tomography (sloreta): Technical details, Methods and Findings in Experimental and Clinical Pharmacology, 24(5):22612, 2002. [15] A. Salman, A. Malony, S. Turovets, D. Tucker, Use of parallel simulated annealing for computational modeling of human head conductivity, In Y.S. et al., editor, International Conference on Computational Science, LNCS 4487:86–93. Springer-Verlag, 2007. [16] A.Salman, S.Turovets, A.Malony, Computational modeling of human head conductivity, In V. S. et al., editor, International Conference on Computational Science, LNCS 3514:631–638, Springer-Verlag, May 2005. [17] A. Salman, et al., Noninvasive conductivity extraction for high-resolution EEG source localization, Advances in Clinical Neuroscience and Rehabilitation, 6:27–28, 2006. [18] J. Talairach and P. Tournoux., Co-planar stereotaxic atlas of the human brain, Thieme, Stuttgart, 1988. [19] D.Tucker, Spatial sampling of head electrical fields: the geodesic sensor net, Electroencephalography and Clinical Neurophysiology, 87(3):154–163, 1993. [20] S. Turovets, et al., Bounded electrical impedance tomography for non-invasive conductivity estimation of the human head tissues, Electrical Impedance Tomography Conference, June 2009. [21] S. Turovets, et al., Conductivity analysis for high-resolution EEG, International Conference on BioMedical Engineering and Informatics, 2:386–393, 2008. [22] V. Volkov, A. Zherdetsky, S. Turovets, A. Malony, A fast BICG solver for the isotropic poisson equation in the forward EIT problem in cylinder phantoms, International Conference on Electrical Bioimpedance, Gainesville , FL, April 2010. [23] Y. Zhang, W. van Drongelen, B. He, Estimation of in vivo brain-to-skull conductivity ratio in humans, Applied Physics Letters, 89:2239031–3, 2006.

336

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-336

Simulation and Modeling of Metamorphopsia with a Deformable Amsler Grid Anabel MARTIN-GONZALEZa,1, Ines Lanzl b, Ramin Khoramnia b and Nassir NAVAB a a Chair of Computer Aided Medical Procedures (CAMP), TUM, Germany b Ophthalmology Department, Klinikum rechts der Isar, Germany

Abstract. A method to simulate and model metamorphopsia by means of a deformable Amsler grid is proposed. The interactively deformable grid is based on cubic B-splines to obtain a locally controlled deformation. By simulating metamorphopsia on normal sight volunteers, acquisition of a correction percentage is possible as a result of analyzing the magnitude of the simulated distortion and the applied correction model. The correction percentage obtained is 75.78% (7.36% standard deviation). This can express the feasible correction rate with the guidance of the patient qualitative feedback. The present work is motivated by the idea of obtaining a correction model of a patient with metamorphopsia and to implement this model into a head-mounted display to compensate the patient’s deformation in the near future. Keywords. Visual impairment, augmented reality, metamorphopsia

Introduction In ophthalmology, augmented reality (AR) is playing an important role as a result of enhancing the view of visually impaired people with different methodologies in order to improve their degenerated vision. The head-mounted display (HMD), a relevant interface in AR, seems to be a suitable medium to assist people with diverse eye diseases [1]. Macular disorders such as age-related macular degeneration (AMD), idiopathic epiretinal membrane (ERM) and macular hole have been found to cause metamorphopsia, a symptom described as the perception of distortion of objects. Patients with metamorphopsia visualize a straight line as an irregular or curved line. It is known that one of the main causes of metamorphopsia in individuals with macular diseases is the displacement of photoreceptors in the sensory retina [2]. Nevertheless, this disorder is not completely well understood. The use of Amsler charts is a common clinical approach for detecting metamorphopsia [3]. An Amsler grid is a printed squared grid (10×10 cm) containing equally spaced parallel horizontal and vertical lines. Variants of the Amsler grid have been elaborated to examine central vision [4], but the original chart seems to perform 1

Corresponding Author: Anabel Martin-Gonzalez, Technische Universität München (TUM), Boltzmannstr. 3, 85748, Garching, Germany; E-mail: [email protected].

A. Martin-Gonzalez et al. / Simulation and Modeling of Metamorphopsia

337

better. Although it is a standard mean for diagnosing and measuring metamorphopsia, an Amsler grid cannot evaluate quantitatively the degree of metamorphopsia. Some methods have been proposed for measuring metamorphopsia. Matsumoto et al., have developed a method for quantifying metamorphopsia in patients with ERM by the use of M-CHARTS [5]. The Preferential Hyperacuity Perimetry [6] (PreView PHP, Carl Zeiss Meditec, Dublin, CA) is suitable for mapping the area in the visual field affected by metamorphopsia and to follow the progression of deterioration. The PHP is based on hyperacuity (Vernier acuity), that is the ability to recognize the relative spatial localization of two or more stimuli [7]. Trevino [8] and Crossland [9] show different methods for macular assessment, but the main problem in these methods is the difficulty to reliably assess due to the lack of knowledge of the ground truth of the patient’s vision, some defects may not be covered by the grid lines. In [10], a dynamic Amsler grid created in computer-graphics was developed to overcome these deficiencies. Nevertheless, these methods do not provide the way patients perceive their distorted world in detailed, this is, their visual deformation model. As a first step on this research, we propose a deformable Amsler grid based system to simulate distorted vision in healthy eyes in order to analyze reliability of the system for obtaining an inverted deformation model, termed correction model of metamorphopsia. By having a deformation model, it could be possible to localize the macular areas causing deformations in optical coherence tomography (OCT) images of the patient retina and analyze them to see if it is viable to find any macular pattern related to the deformation’s shape. A correction model could be applied to images of AR display devices (i.e., head-mounted display) and therefore, a correcting system for patients with distorted vision could be achieved.

1. Methods & Materials The method consists of simulation of distorted vision for the human eye; acquisition of the correction model of the simulated visual distortion; and finally the analysis of the obtained results. The hardware used includes a 17” monitor, an eye tracker device, and a workstation with Intel Core Duo CPU at 2.40 GHz, 2GB of RAM (see Figure 1). 1.1. Deformable Amsler Grid In order to create a deformable Amsler grid, we have chosen a free form deformation (FFD) model [11], based on B-splines, which is a feasible tool for modeling deformable objects. Basically, the idea of FFD’s is to deform an object (i.e., Amsler

Figure 1. Simulation and modeling system (left); volunteer and examiner performing an experiment (right).

338

A. Martin-Gonzalez et al. / Simulation and Modeling of Metamorphopsia

grid) by manipulating an underlying mesh of control points. The resulting deformation gives us the metamorphopsia correction model of the patient and produces a smooth and C2 continuous transformation. To define a spline-based FFD, we denote the domain of the image as = {(x, y) | 0 x < X, 0 y < Y}. Let denote a nx × ny mesh of control points i,j with uniform spacing . Then, the FFD is defined as the 2D tensor product of the 1D cubic B-spline functions and the displacements of the control points: 3

T ( x, y )

3

¦¦ B

m

(u ) Bn (v)4i m , j n

(1)

m 0n 0

where i ¬x / n x ¼ 1 , j ¬y / n y ¼ 1 , u x / nx ¬x / nx ¼ , v y / n y ¬y / n y ¼ , and where Bm presents the m-th basis function of the B-spline: B0(u) = (1 – u)3/6, B1(u) = (3u3 – 6u2 + 4)/6, B2(u) = (–3u3 + 3u2 + 3u + 1)/6, B3(u) = u3/6. In contrast to thin-plate splines [12], B-splines are locally controlled. The resulting transformation from changing the control point i,j will affect only the local neighborhood of that control point. The degree of non-rigid deformation which can be modeled depends on the resolution of the mesh of control points. A deformable Amsler grid based on cubic B-splines will provide smooth transformations. 1.2. Simulation of Distorted Vision In order to simulate metamorphopsia on a normal sighted person, an image deformation is generated and placed on a specific location of the grid (Figure 2). Considering the grid center as the visual center (visual angle 0º), the eye tracker follows the gaze so that the deformation is displayed all the time in the same selected location of the person’s visual field. As a result, it is possible to recreate the visual imperfection caused by the displacement of photoreceptors in the human eye. In the same manner as with real metamorphopsia patients, to digitally correct distorted vision it is necessary to deform the grid section, located in the affected retinal field, in the opposite direction of the real perceived deformity. Therefore, to correct the simulated distortion an interaction with the mouse on the monitor moves the grid lines in the opposite trend of the perceived distortion until the person sees no deformation in the lines. This procedure will provide the metamorphopsia correction model. 1.3. Experiments In order to evaluate whether the grid resolution can affect the discernment of distortions or not, a squared and a non-squared grid are tested. The dimension for every grid is 46.9×26.2 cm. Eight points on both grids are selected for locating a pre-

Figure 2. Deformation example on the non-squared grid.

A. Martin-Gonzalez et al. / Simulation and Modeling of Metamorphopsia

339

generated distortion (Figure 3). Every location corresponds to a specific retinal area of the visual field (the grid center corresponds to visual angle 0º). The magnitude of deformation on every point of the grid is randomly defined covering a visual angle range of 0.18º to 1º. A group of volunteers is selected to participate in the experiments. The criteria for inclusion is best corrected visual acuity equal to 20/20. The experiment is as follows (Figure 1): the volunteer is located in front of the external monitor at a distance of 80 cm. One eye is covered and the other one is fixating the center of the displayed grid during the whole experiment. One of the eight predefined distortions is simulated on the screen. The examiner performs a blind correction; this is, without seeing the projected distortion to the volunteer (to recreate the real situation with a patient where the examiner cannot see what the patient perceives). During this simulated environment, the examiner obtains from the volunteer a description of the perceived distortion (location and shape orientation). Once the description is provided, the examiner stops the simulated distortion. The grid will become regular again (i.e., with straight lines) and the examiner will now see the screen to interact with the grid in order to acquire the correction model for that specific distortion; this is, to deform the grid, in the provided location and in the opposite given orientation. After this step, the examiner does not look the screen anymore and turns on the simulated distortion on the grid that includes the examiner’s modification. At this time, the simulated distortion may be reduced (or disappear completely) and the deformed grid line affected by the simulated distortion may seem straighter (or completely straight). The blind correction procedure will be performed until the person cannot perceive any distortion. The procedure will be done with the squared and nonsquared grid (the first one being tested is randomly chosen for each volunteer). Once the experiment is finished, the correction percentage related to the original simulated deformation is analyzed. The correction percentage will be measured by calculating which percentage of the magnitude of the displacement vector of the control point for simulating the distortion was reduced after the correction procedure. If the magnitude of the displacement vector after correction is 0, a 100% of correction will be measured. The eye tracker avoids the natural instinct of trying to fixate not the grid center, but the simulated distortion projected out of the central vision. Therefore, the eye tracker moves the simulated distortion to its corresponding location in the visual field according to the gaze movement, thus the distortion shifts its location and the person cannot focus it with the central vision. The visual angle V corresponding to each one of the eight points in the grid can be obtained by the equation V = 2 arctan (S/2D), where S is the object length in the real world and D is the distance from the eye to the object (i.e., monitor).

Figure 3. Squared grid (left) and non-squared grid (right) with evaluated locations.

340

A. Martin-Gonzalez et al. / Simulation and Modeling of Metamorphopsia

Table 1. Total correction rate (percent) on squared and non-squared grids. Squared Grid 75.75 10.12

AVG SD

Non-Squared Grid 75.78 7.36

Table 2. Correction rate (percent) and minimum recognizable visual angle (degrees) for each evaluated location. Location

Visual Angle

p0 p1 p2 p3 p4 p5 p6 p7

0.00 2.08 4.15 5.94 6.22 8.29 8.60 11.85

Correction Rate Squared Grid AVG SD 90.63 7.58 81.91 14.26 57.23 31.57 74.07 22.64 70.36 17.44 79.16 12.89 79.71 19.07 72.90 25.15

Non-Squared Grid AVG SD 80.23 12.97 81.13 8.69 72.85 23.47 73.49 23.50 76.05 16.29 81.49 11.31 66.61 26.47 74.38 12.02

Minimum Recognizable Visual Angle Squared Grid Non-Squared Grid AVG SD AVG SD .0364 .0303 .1290 .1147 .0871 .0621 .0986 .0722 .1259 .0804 .1163 .1120 .1518 .1224 .1192 .0993 .1662 .1032 .1328 .0922 .1643 .1170 .0918 .0852 .1204 .1346 .1336 .0860 .1593 .1182 .1705 .0908

2. Results The system provides a model of distorted vision in a range of 32.67º horizontal and 18.60º vertical visual angles. The non-squared grid has a horizontal line spacing of 2.97º and a vertical line spacing of 2.08º in visual angles. In the case of the squared grid, the lines have an approximately equal spacing of 1.98º horizontal and 2.08º vertical. The resolution of the mesh of control points on the grid for modeling transformations is 35×19 points. In total, 17 normal subjects (17 eyes) could reliably fulfill the task. The average (AVG) and standard deviation (SD) of the correction rate obtained are presented in Table 1. The Table 2 shows the visual angle (in degrees) corresponding to each selected location for evaluation on the grid with its correction rate. These results are plotted in a graph for easy visualization and analysis (see Figure 4). According to the results, the correction rate averages do not show any significant difference between using a squared or a non-squared grid. To go deeper in the studies, it is relevant to analyze the minimum recognizable visual angles for every selected location; this is obtained with the final magnitude of the displacement vector of the control point for simulating the distortion after the correction procedure. Table 2 presents the results. In Figure 4, it is shown that with a non-squared grid it is possible to distinguish smaller deformations in the middle

Figure 4. Correction rate (left); minimum recognizable visual angles (right).

A. Martin-Gonzalez et al. / Simulation and Modeling of Metamorphopsia

341

peripheral vision (p2 to p5); for locations near the visual center (p0 and p1) and far away from it (p6 and p7) a squared grid has slightly better results. As it could be expected, there is an incremental trend relating the minimum recognizable deformation and its location in the visual field, this means that the farther away the deformation is from the visual center the less a subject can recognize it. However, for these locations, a subject can still guide the examiner to perform a reasonable correction as seen in the results of Figure 4. An eye tracker device plays a very important role for simulation of metamorphopsia for a healthy eye, not only because it reproduces the feeling of having a real distortion moving with the gaze, but also for preventing the person from looking directly at the distortion during the experiment to describe it, instead of fixing the center of the grid. Moreover, an eye tracker can provide us with the feedback of whether the real patient is fixating the grid center or not, increasing the accuracy of correction and localization of the affected areas in the macula for further analysis. Therefore, tracking’s accuracy of a single eye is an essential parameter for success in modeling and correcting distorted vision. In addition, the system can provide a measurement of the deformation seen in the grid in order to have an estimation of the size of the affected area in the macula (in mm) according to the patients’ visual perception; which helps to evaluate the progression of the disease. The resolution of the mesh of control points in the grid can be increased to model more difficult metamorphopsia cases. The final correction model acquired can be implemented in a head-mounted display based augmented reality system. Thus, it will be possible to compensate distorted vision in real patients. Furthermore, the deformation model (inverted correction model) can be used as a model to describe metamorphopsia for medical diagnosis by locating the affected areas in macular OCT and fundus images with the help of a system we have already developed (see Figure 5). The OCT imaging technology defines a 3D volume of the retina by means of a set of cross-sectional images orthogonal to the retina’s surface plane, which is visible in a fundus image. An integrated eye tracker guides the OCT scan to the selected location (normally the fovea or vision center), and in relation to that position it acquires the cross-sectional images of the retinal layers including the photoreceptors layer. For registering the OCT images and the distortion information obtained using our Amsler grid experiments, we generate the projected image (in mm) on the retina produced by the grid at the distance of 80 cm. This is based on the fact that the visual plane of the subject is parallel to the grid and fixating its center (as done in the experiments). Thus, it is possible to approximately align the grid on top of the retina (fundus image) and therefore place the OCT images in the position where they were

Figure 5. Alignment of macular OCT image and deformable Amsler grid.

342

A. Martin-Gonzalez et al. / Simulation and Modeling of Metamorphopsia

scanned, see Figure 5. This alignment of information could help for a better understanding of metamorphopsia and could create a set of valuable multi-modal data for evaluating the possibility of development of novel quantitative and/or qualitative analysis of visual distortion based on OCT images.

3. Conclusions A system based on a deformable Amsler grid for simulating and modeling metamorphopsia was developed. The grid based on cubic B-splines provides smooth transformations suitable to correct visual deformations. The feasible correction rate for distorted vision by using a deformable non-squared grid and the guidance of the patient qualitative feedback is 75.78%. The use of an eye tracker increases the reliability of the results. A deformable Amsler grid based system could provide a simple and useful method for modeling distorted vision in patients with metamorphopsia. By implementing the correction model on an AR display device, it would be possible to improve the vision of patients with metamorphopsia.

Acknowledgment This project is supported by the Bayerische Forschungsstiftung and partly by the Secretaría de Educación Pública de México.

References [1] E. Peli, G. Luo, A. Bowers, and N. Rensing, Development and evaluation of vision multiplexing devices for vision impairments, Int J Artif Intell Tools 18 (2009), 365–378. [2] E. Arimura, C. Matsumoto, S. Okuyama, S. Takada, S. Hashimoto, and Y. Shimomura, Retinal contraction and metamorphopsia scores in eyes with idiopatic epiretinal membranes, Invest Ophthalmol Vis Sci 46 (2005), 2961–2966. [3] M. Amsler, Earliest symptoms of diseases of the macula, Br J of Ophthalmol 37 (1953), 521–537. [4] M.F. Marmor, A brief history of macular grids: from Thomas Reid to Edvard Munch and Marc Amsler, Surv Ophthalmol 44 (2000), 343–353. [5] C. Matsumoto, E. Arimura, S. Okuyama, S. Takada, S. Hashimoto, and Y. Shimomura, Quantification of metamorphopsia in patients with epiretinal membranes, Invest Ophthalmol Vis Sci 44 (2003), 4012– 16. [6] A. Loewenstein, R. Malach, M. Goldstein, I. Leibovitch, A. Barak, E. Baruch, Y. Alster, O. Rafaeli, I. Avni, and Y. Yassur, Replacing the Amsler grid: a new method for monitoring patients with age-related macular degeneration, Ophthalmology 110 (2003), 966–970. [7] M. Goldstein, A. Loewenstein, A. Barak, A. Pollack, A. Bukelman, H. Katz, A. Springer, A.P. Schachat, N.M. Bressler, S.B. Bressler, M.J. Cooney, Y. Alster, O. Rafaeli, and R. Malach, Results of a multicenter clinical trial to evaluate the preferential hyperacuity perimeter for detection of age-related macular degeneration, Retina 25 (2005), 296–303. [8] R. Trevino, Recent progress in macular function self-assessment, Ophthalmic Physiol Opt 28 (2008), 183–92. [9] M. Crossland and G. Rubin, The Amsler chart: absence of evidence is not evidence of absence, Br J of Ophthalmol 91 (2007), 391–393. [10] L. Frisén, The Amsler grid in modern clothes, Br J of Ophthalmol 93 (2009), 714–716. [11] T.W. Sederberg and S.R. Parry, Free-form deformation of solid geometric models, in: Proceedings of SIGGRAPH ’86, Computer Graphics 20 (1986), 151–160. [12] F.L. Bookstein, Principal wraps: Thin-plate splines and the decompositions of deformations, IEEE Trans Pattern Anal Machine Intell 11 (1989), 567–585.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-343

343

Development of a Customizable Software Application for Medical Imaging Analysis and Visualization Marisol MARTINEZ-ESCOBAR, Catherine PELOQUIN, Bethany JUHNKE, Joanna PEDDICORD, Sonia JOSE, Christian NOON, Jung Leng FOO 1, and Eliot WINER Virtual Reality Applications Center, Iowa State University, Ames, Iowa, USA

Abstract. Graphics technology has extended medical imaging tools to the hands of surgeons and doctors, beyond the radiology suite. However, a common issue in most medical imaging software is the added complexity for non-radiologists. This paper presents the development of a unique software toolset that is highly customizable and targeted at the general physicians as well as the medical specialists. The core functionality includes features such as viewing medical images in two- and three-dimensional representations, clipping, tissue windowing, and coloring. Additional features can be loaded in the form of ‘plug-ins’ such as tumor segmentation, tissue deformation, and surgical planning. This allows the software to be lightweight and easy to use while still giving the user the flexibility of adding the necessary features, thus catering to a wide range of user population. Keywords. Software Framework, User Interface Design, Visualization

1. Introduction Visualizing patient data has become an integral part of the diagnosis and clinical evaluation stages. As medical imaging and graphics technologies improve, visualization software applications are being expanded to include more advanced tools. Surgeons and doctors now have the ability to perform data visualization, analysis, and diagnosis outside of the traditional radiology setting. However, these advanced tools can sometimes be a hindrance when they are not developed with Human Factors considerations and the software tends to be unusable or require a steep learning curve. To address this problem, the Isis software framework was designed (Figure 1) by understanding the needs of the user and recognizing the physical and mental capabilities of the targeted user group. The framework was developed first by identifying several critical and common tasks performed by doctors and surgeons. The tools required to complete these tasks were then organized in an effective manner that would allow the user to understand and learn the software easily. Currently, the Isis software framework consists of three primary components: 1) Examiner, 2) Surgical planning, and 3) Data Analysis.

1 Corresponding Author: Jung Leng Foo, Virtual Reality Applications Center, 1620 Howe Hall, Iowa State University, Ames, Iowa, 50011, USA. E-mail: [email protected]

344

M. Martinez-Escobar et al. / Development of a Customizable Software Application

Figure 1. Breakdown of the various features and components in the Isis software framework. Isis Core has the features of a basic Examiner, while the other categories can be implemented as Isis Plug-ins.

Earlier work on this project was the development of Isis VR [1], an immersive environment to interact and manipulate three-dimensional representations of patient medical image data in real time. In an effort to foster collaboration between surgeons, work was also done to create a collaborative multi-modal environment for surgical planning [2]. A basic desktop viewer was developed to sync over the network with the virtual environment, where any user interactions on one would be updated on the other. The desktop viewer has since been renamed Isis, the desktop counterpart to Isis VR. The work presented in this paper discusses the redevelopment of Isis. A highly customizable software framework with a user interface tuned to the needs of primary care physicians and specialists (e.g., oncologists and surgeons). Isis provides a simple and effective environment for interaction and visualization of patient image data.

2. Methods 2.1. Display and Interaction Widgets The Isis interface can be divided into four separate sections: 1) The main display window, 2) Examiner features panel, 3) Mode switcher, and 4) Mode specific features panel. These features make up a simple, straightforward, and basic application to visualize and interact with digital medical image data. By organizing and grouping the examiner features panel into the right side, the user can access common features in all the modes. This arrangement allows the user to create effective mental mappings between the controls and the functions in the interface. The main display window can display two- and three-dimensional representations with which the user can interact (rotation, translation, and zoom). Additional controls are available in the examiner features panel, where the user can change the views, colors, and tissue types displayed. A screenshot showing a sample configuration of the main display with the features panel is shown in Figure 2.

M. Martinez-Escobar et al. / Development of a Customizable Software Application

345

Figure 2. The Isis software in ‘Core’ mode displayed in a tiled viewed.

The ‘View’ option in the figure allows a user to switch amongst predefined 2D and 3D views. A user can select all available views (as shown in the figure) or any combination. The ‘Color’ menu gives users quick access to predefined color schemes to highlight specific anatomical features or structures. ‘Tissue Type’ contains several predefined windowing settings as well as the ability to define custom values to isolate specific densities such as bone, muscle, cartilage, etc. 2.2. Software Libraries Isis was developed using several open source software libraries: • Qt (http://qt.nokia.com) • DICOM Toolkit (DCMTK) (http://dicom.offis.de/dcmtk.php.en) • Visualization Toolkit (VTK) (http://www.vtk.org). The user interface (UI) was designed using Qt, an open source cross platform UI framework. This facilitates a consistent look of the user interface across various operating systems for a consistent user experience. Currently, Isis is compatible with Windows 7, Windows XP, and Mac OSX. The software architecture of the Qt framework also decouples the UI and the backend code, allowing the UI to be designed independently of the source code development. Intermediate functions are used to connect the inputs from the UI to actual function calls in the source code. DCMTK was used to parse the input files and it has a large collection of tools to access the information stored in a DICOM/PACS file as well as perform the necessary functions to parse the image data. The main advantage of DCMTK is its compatibility with files created from the various scanner manufacturers.

346

M. Martinez-Escobar et al. / Development of a Customizable Software Application

VTK is the primary rendering engine for Isis. VTK contains extensive threedimensional rendering routines such as real time volume rendering, surface generation, and multi-planar reconstruction (MPR) based on the OpenGL standard. This allows Isis to take advantage of 3D graphics hardware acceleration as well as rendering on the computer processor. The rendering routines in VTK also includes features such as coloring and transparency of the 2D and 3D representations, as well as clipping planes to interactively ‘slice’ through a 3D volume. 2.3. Isis Core and Isis Plug-Ins During the software design phase, a decision was made to develop Isis as separate components: Isis Core and Isis Plug-ins. Isis Core contains the basic features such as two- and three-dimensional viewing, clipping, tissue windowing, and coloring; and additional features can be implemented into Isis Core as plug-ins. Plug-ins that are currently available are: 1. Tumor segmentation 2. Surgical planning The tumor segmentation plug-in will be based off segmentation algorithms developed by the authors. These include a fuzzy segmentation algorithm [3] and a probabilistic segmentation algorithm [4], and a colorization method to colorize grayscale images based on user inputs on properties of the tumor and region of interest. For diagnosis and treatment planning purposes, the segmented tumor can also be overlaid on the original patient data, providing the doctor with context information such as the shape and size of the approximated tumor relative to critical anatomical structures when combined with the surgical planning plug-in (Figure 3). This is a feature from Isis VR [5] and was redesigned for the desktop environment.

Additional features will include measurement tools (Figure 4) and landmark placements to provide additional information such as the size of the tumor, as well as the distance of critical structures, i.e. arteries, from the tumor. These tools will assist in

M. Martinez-Escobar et al. / Development of a Customizable Software Application

347

trocar placements for laparoscopic surgeries or incision locations in open surgeries. The advantage of having these advance tools as plug-ins is that each user can customize Isis based on their own needs and preferences. For example, a medical student in training might only require Isis Core, while a practicing surgeon will want Isis Core with the tumor segmentation and surgical planning plug-ins. As the development of Isis progresses, additional plug-ins can be implemented to cater to a variety of target user groups while still ensuring that the basic software application remains lightweight and robust.

3. Discussion and Conclusion As the research into medical imaging continues to produce more advance analysis and visualization tools, medical imaging software will continue to grow in complexity. However, the average user might not require or even understand half of the tools and features included in these software packages. The Isis software framework proposed in this paper creates a highly customizable and expandable platform. From the first year medical student to a practicing physician, the Isis Plug-ins can cater to a wide user group. Additional development is currently being planned, as well as a formal usability study for the user interface and the various features built in to Isis. The evaluation methods will measure the performance and the user experience as it relates to performance, i.e. time to localize anatomical features. User experience will be measured by conducting usability studies that will identify issues with the interface that may interfere with the performance. In addition, cognitive load testing will be assessed for students who want to learn anatomy using 3D visualization methods. Cognitive load can help to identify if the information on the interface overloads the processing capabilities of the user, preventing the student from making new associations in longterm memory and hindering learning [6].

References [1]

[2]

[3]

[4]

[5]

[6]

Foo JL, Lobe T, and Winer E. A Virtual Reality Environment for Patient Data Visualization and Endoscopic Surgical Planning, Journal of Laparoendoscopic & Advanced Surgical Techniques, 18 (5) 697-706, 2008. Foo JL, Martinez M, Peloquin C, Lobe T, and Winer E. A Collaborative Interaction and Visualization Multi-Modal Environment for Surgical Planning, Proceedings of 17th Medicine Meets Virtual Reality (MMVR) Conference, 142 (2009), 97-102. Published by IOS Press. Foo JL, Miyano G, Lobe T, and Winer E. Three-Dimensional Segmentation of Tumors from CT Image Data using an Adaptive Fuzzy System, Journal of Computers in Biology and Medicine, 62 (2009) 869878. Foo JL, Lobe T, and Winer E. Automated Probabilistic Segmentation of Tumors from CT Data using Spatial and Intensity Properties, Proceedings of SPIE Medical Imaging, Lake Buena Vista, FL, February 8-10, 2009. Foo JL, Miyano G, Lobe T, and Winer E. A Framework for Interactive Examination of Automatic Segmented Tumors in a Virtual Environment, Proceedings of 16th Medicine Meets Virtual Reality (MMVR) Conference, 132 (2008), 120-122. Published by IOS Press. Wickens, C, Lee J, Liu Y, and Gordon S. An Introduction to Human Factors Engineering, 2004. Published by Prentice Hall.

348

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-348

Pneumoperitoneum Technique Simulation in Laparoscopic Surgery on Lamb Liver Samples and 3D Reconstruction MARTÍNEZ-MARTÍNEZ F.a,1, RUPÉREZ M. J. a, LAGO M.A. a, LÓPEZ-MIR F. a, MONSERRAT C. a, and ALCAÑÍZ M.a a Instituto Interuniversitario de Investigación en Bioingeniería y Tecnología Orientada al Ser Humano, Universidad Politécnica de Valencia. Camino de Vera s/n 46022, Valencia, Spain. Phone/Phax: +34 96 387 75 18 (Ext. 67020) / +34 96 387 95 10

Abstract. In this paper, a procedure to experimentally simulate the behavior of the liver when the pneumoperitoneum technique is applied in laparoscopic surgery is presented, as well as methodology to make the comparison of each sample before and after insufflating the gas. This comparison is carried out using the 3D reconstruction of the volume from the CT images when either pneumoperitoneum is applied and when it is not. This methodology has showed that there are perceptible changes of volume when the pneumoperitoneum is applied. Keywords. Laparoscopy, reconstruction.

pneumoperitoneum,

liver,

simulation

and

3D

Introduction Laparoscopy is a minimally invasive technique based on the execution of little incisions on the abdomen aimed at introducing the instrumental needed to perform the intervention (Figure 1). At the beginning of this, CO2 is insufflated in the abdomen through a trocar in order to create enough space to make possible the intervention. This technique is referred to as Pneumoperitoneum [1]. In this paper, a procedure to experimentally simulate the behavior of the liver when the pneumoperitoneum technique is applied is presented. This procedure will show that there are perceptible changes of volume when the pneumoperitoneum is applied.

1

Corresponding Author. Email address: [email protected] (Martínez-Martínez F.)

F. Martínez-Martínez et al. / Pneumoperitoneum Technique Simulation in Laparoscopic Surgery

349

Figure 1. Laparoscopic surgery

1. Materials and Methods Two circular samples were obtained from four lamb livers, one of 100 mm of diameter and another smaller of 80mm of diameter. The samples were introduced in a glass receptacle which was hermetically sealed but connected to a tube through which the CO2 was insufflated (Figure 2, left). The device used to insufflate the CO2 was Wolf IP20 (Figure 2, middle). The pressure was kept constant during all the procedure. Two different ranges of pressure values were tested: 10-14 mmHg (the most common used in abdominal interventions) was applied to 4 samples and 16-19 mmHg was applied to the other 4 samples. The receptacle was introduced in the multi-detector spiral CT GE LightSpeed VCT–5124069 (Figure 2, right) and CT images were acquired from each sample before applying CO2 and when the CO2 was applied. The axial slices interval was of 0.625 mm. The experiments were carried out in Hospital Clínica Benidorm (HCB). The CT images were processed using the software ScanIP v4.0 from Simpleware in order to obtain the volume of each sample. The steps were: Segmentation and filtering, 3D reconstruction and volume calculation (Figure 3).

Figure 2. Material used for the experiment: Liver sample inside the hermetical glass receptacle (left), insufflating device Wolf IP20 (middle) and Multi-detector spiral CT GE LightSpeed VCT-5124069

2. Results The results are shown in Table 1. Table 1 shows the volume in cm3 of every sample when the CO2 is applied and when it is not applied for the two ranges of pressure. It also shows the volume difference between the same sample when CO2 is applied and when it is not. As volume of one voxel was 1.49x10-3 cm3, the precision was 1x10-3 cm3.

350

F. Martínez-Martínez et al. / Pneumoperitoneum Technique Simulation in Laparoscopic Surgery

Figure 3. Steps for the volume calculation

Table 1. Results Liver Samples Volume (cm3)

No CO2 With CO2 Δ Volume

LS1 178.552 176.923 1.639

10-14 mmHg LS2 LS3 211.785 71.639 211.313 71.256 0.472 0.383

LS4 203.438 203.355 0.083

LS5 94.796 94.290 0.506

16-19 mmHg LS6 LS7 200.268 83.776 200.247 83.660 0.021 0.116

LS8 195.307 194.841 0.466

3. Conclusions As a result of this work, it can be concluded that there is perceptible changes in the volume of the samples when CO2 is applied. This means that the liver is lightly compressed under these pressures when a laparoscopy surgery is carried out. This confirms that the Poisson’s coefficient largely responsible of the compression is high, which agrees with the value that can be found in the literature (0.45 according to [2]).

4. Discussion and Future Works The results of this paper are very useful for planning liver laparoscopic surgeries since it allows to simulate the compression of this organ when the pneumoperitoneum technique is applied. In future works, the methodology presented in this paper will be extended to the rest of abdominal organs present in an abdominal laparoscopic surgery.

Acknowledgements This project has been partially funded by MITYC (reference TSI-020100-2009-189).

References [1] H.J. Bonjer, Open versus closed establishment of pneumoperitoneum in laparoscopic surgery, Brithis Journal of Surgery 84 (1997), 599–602. [2] H. Shi, Validation of Finite Element Models of Liver Tissue Using Micro-CT, IEEE transactions on biomedical engineering, 55, 3 (2008), 978-984

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-351

351

Technology Transfer at the University of Nebraska Medical Center Kulia MATSUO, MD, MBAa,1 , Henry J. RUNGE, JDb, David J. MILLER, PhDc, Mary A. BARAK-BERNHAGEN, BSc, and Ben H. BOEDEKER, MD, PhDc a University of Nebraska Medical Center, Omaha NE b UNeMed Corporation, Omaha, NE c Dept. of Anesthesiology, University of Nebraska Medical Center, Omaha, NE Abstract: The course of developing a new product from an idea is a complicated process. This paper will discuss that process, in detail, from conception to product. We approach this by first discussing what the inventor must do begin the process of developing his or her idea, and then two pathways that occur simultaneously: the Technology Transfer process of patenting, marketing, and licensing the invention; and the engineering process of developing, modifying, and manufacturing the invention. Although the process is lengthy and most ideas never become a marketed product, there are those few ideas that do become realized into marketed products. Keywords. Technology transfer office, patent, prototype

Background The development process of an invention from idea to market is lengthy and complicated. There are multiple stages that occur when bringing a new idea to market and an idea can fail at each stage. Later stages of the technology transfer process are routinely referred to as “The Valley of Death.” The purpose of this work is to describe the general development of an invention from idea to product.

Methods The development of an invention begins with the conception of the idea arising from a perceived need. The inventor then submits a New Idea Notification (NIN) Application to the Technology Transfer Office (TTO). From there, the invention is evaluated and, based on an initial patentability and marketability analyses, the TTO may or may not decide to proceed to file a patent application and begin marketing the invention. If the TTO decides to proceed with the invention, a patent application, either provisional or non-provisional, is filed with the United States Patent and Trademark Office (USPTO). After processing and prosecuting the patent, the USPTO may decide to grant or not grant a patent. Throughout this process, a licensing associate at the TTO is assigned to

1 Kulia Matsuo, MD, MBA, 1633 North Capital Ave, MT, Suite 640, Indianapolis, IN 46202; E-mail: [email protected]

352

K. Matsuo et al. / Technology Transfer at the University of Nebraska Medical Center

manage the invention through its evaluation and marketing. The licensing associate develops non-confidential marketing materials about the invention, and contacts potential licensees. Typically at this point, the patent has not been granted yet, so the interested licensees must sign a Confidential Disclosure Agreement (CDA). If the licensee decides to pursue the invention, a license agreement with terms and a contract is developed regarding the manufacturing and selling of the product. While the TTO handles the protection and the marketing of the invention, continued engineering and development of the invention also occurs. Although the invention is beyond proof of concept, the invention needs additional testing to be marketable or a complete product. The inventor typically meets with an engineer to develop 3-D CAD models of the invention. Rapid prototypes are made to evaluate the geometry and shape of the invention. The inventor and engineer test and refine the prototype generating the next generation of prototypes. This process continues until the inventor and engineer feel the prototype is of high quality. The prototypes must undergo biocompatibility testing, and be FDA approved. During this time, a research protocol is generated and submitted to the university’s Institutional Review Board (IRB). After revisions to the prototype, a manufacturer is identified through the TTO or by the inventor. The manufacturing company makes working prototypes and sends it to the inventor and engineer for final approval. Once approved, the first production run occurs. The manufacturer may or may not be the same company that will market the device. Funding for the invention may come from different variables, such as established industrial partners or governmental grants.

Results Figure 1 shows a case study describing the development cycle of a video laryngoscope suction blade at the University of Nebraska. Other institutions’ procedures may vary. Idea: When intubating a trauma patient, the inventor noticed the airway was often obstructed with blood and secretions, making intubation difficult. Also when using a videolaryngoscope (VL) blade, fog created from the patient’s airway would obscure the camera’s view. The inventor thought that incorporation of suction into a VL blade would clear secretions. In addition, the suction could be dually used for either the insufflation of oxygen or drug delivery. A quick survey found no similar devices readily available on the market. Feasibility Study: Under FDA Exemption Sect. 21 CFR807.65D, the inventor tested the concept by taping a suction tube to the VL blade to clear secretions during intubation. This rough prototype was feasible and successful. Patenting & Marketing: New Invention Notification (NIN). Inventor initially disclosed the idea to his Technology Transfer Office (TTO) and suggested approaches to integrate both suction and insufflation capabilities into a variety of medical and industrial instruments. An initial search of the prior art uncovered a limited number of patents and patent publications in the same field as the inventor’s invention. Several of the approaches that the inventor took were substantially different than the other patents/patent applications. Protection & the USPTO. The inventor’s TTO filed a provisional patent application that included the inventor’s invention disclosure, data from the feasibility study and the specifications of prototypes that his laboratory had reduced to practice. With a filing date secure, the inventor was free to discuss the idea with commercial partners without creating any public disclosures. A year from the filing date of the provisional patent application, the inventor’s TTO would need to convert the provisional patent application to a utility application. During this year, market interest was assessed, and the TTO further solidified the patent position while also drafting the utility application. Marketing & Market Interest. After discussing the project with several companies, the inventor identified multiple partners that he could move forward in developing the invention. One partner, a mid-sized medical device firm, expressed interest in licensing the patent application filed by the inventor’s TTO. License Agreement & Commercialization. The mid-sized medical device firm negotiated a license agreement with the inventor’s TTO and became a licensee. The license contained standard terms for an academic institution. With the protection of the patent, the licensee could launch the

K. Matsuo et al. / Technology Transfer at the University of Nebraska Medical Center

353

product into a protected market. The inventor’s innovation would not only be sold as one of the licensee’s products, but it would also have the protection of the exclusive rights for over the life of the patent. Given the standard terms of the license, and as required under Bayh Dole, the University of Nebraska, the Department of Veterans Affairs, and the internal intellectual property policies of these institutions, the inventor’s lab and the inventor himself would all share in the upside of the invention. Engineering & Development: Using information generated from the rough prototype (which consisted of a disposable suction tube being taped to a video laryngoscope blade), a 3D model of the suction blade and disposable suction catheter was generated by computer-assisted design (CAD). Protocol. A protocol was written to develop and test the suction blade. Reduction to Practice. The inventor confirmed multiple embodiments of the device were effective in order to reduce the invention to practice and demonstrate that the concept was viable. Based on the 3D model and the inventor’s rough prototype, the suction blade was modeled by taking a stainless steel laryngoscope blade and cutting a channel from stainless steel tubing for containment of the suction catheter. The channel was glued onto the blade (an example of low cost prototype development). Visual Inspections. Based on visual inspection of these rapid prototypes, it was determined that this channel would be too sharp to be used on a patient; therefore, the next generation prototype was developed with the edge blunted. The cycle of visual inspections of the prototype and changes to the blade and catheter resulted in nearly a dozen generations of prototypes, after which a fully functional prototype blade was made by a large international endoscopy company. These prototypes were strong enough to be tried on mannequins to test the function of the suction blade and catheter during intubation. To accomplish further testing, a simulator was designed to adequately reproduce a severely traumatized upper airway. After the IRB accepted the protocol and signed non-disclosure agreements, anesthesia providers tested the prototyped blade on mannequins to test the effectiveness and user-friendliness of the device. Based on the tests, further alterations were made to the suction blade. Companies Produce Prototype. The inventor had previous relationships with a major endoscopy company that produces video laryngoscope systems, as well as a variety of local engineering and research firms. Utilizing these collaborators, the inventor developed a more refined prototype of the blade and a more refined rapid prototype of the disposable suction tube. After each company received drawings and the rough prototypes of the suction blade, they made rapid prototypes which were then sent to the inventor and engineer. In addition, the licensee manufactured the integrated catheter, which was developed in parallel to the laryngoscope blade, so the prototypes and drawings were also sent to this company. Final Prototype. The inventor tested the prototypes as they became available, made changes, and passed the information back to each respective company for further revisions. These revisions were used to develop the next generation prototype. This cycle continued until a final prototype of both the blade and the catheter was agreed upon. Biocompatibility Testing. The plastic used in the suction catheter was sent for biocompatibility testing by a third-party testing company once sufficient design iterations had been done such that the developers were relatively certain they had a working design. In this case, the VL blade was exempt because it was made from stainless steel, and the biocompatibility was already known and it was a modification to an existing, validated device.FDA Approval. The licensee compiled the design history file, including documents from the inventor and his collaborators, to submit to the FDA for review. Each alteration to the blade and catheter must be noted; the biocompatibility and bioburden testing of the licensee’s facilities must also be submitted. Based on the licensee’s analysis, the catheter was exempt from 510(k) premarket approval because it was a Class I device. The VL blade was Class II. Since none were Class III, the premarket approval application did not need to be filled out. The Medical Device Listing, Medical Device Labeling, and Good Manufacturing Practices applications also needed to be filled out by the manufacturers. Manufacturing. The international endoscopy company is currently preparing to manufacture the blade. The licensee is preparing tooling for manufacturing the suction catheter. Sales. Currently, the blade and the catheter will be sold separately. A potential partnership between the international endoscopy company and the licensee could allow the blade and the catheter together. Alternatively, the two companies may decide to develop symbiotic products that are not mutually exclusive, meaning the blade will work without the catheter and the catheter will work without the blade. However, the suction blade will be marketed in catalogues in the near future. Figure 1. Case Study: development cycle of a novel video suction blade at the University of Nebraska

Conclusions Many inventions begin as great ideas, but throughout the rigorous development process, very few inventions are actually commercialized. Although the process is long and complicated, all medical innovations must occur in a process similar to the one described. The “Valley of Death” is a good depiction of this developmental process.

354

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-354

CvhSlicer: An Interactive Cross-Sectional Anatomy Navigation System Based on High-Resolution Chinese Visible Human Data Q. MENG a , Y.P. CHUI a , J. QIN a,c , W.H. KWOK b , M. KARMAKAR b , P.A. HENG a,c a Dept. of Computer Science & Engineering, The Chinese University of Hong Kong b Dept. of Anaesthesia & Intensive Care, The Chinese University of Hong Kong c Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences Abstract. We introduce the design and implementation of an interactive system for the navigation of cross-sectional anatomy based on Chinese Visible Human (CVH) data, named CvhSlicer. This system is featured in real-time computation and rendering of high-resolution anatomical images on standard personal computers (PCs) equipped with commodity Graphics Processing Units (GPUs). In order to load the whole-body dataset into the memory of a common PC, several processing steps are ﬁrst applied to compress the huge CVH data. Thereafter, an adaptive CPU-GPU balancing scheme is performed to dynamically distribute rendering tasks among CPU and GPU based on parameters of computing resources. Experimental results demonstrate that our system can achieve real-time performance and has great potential to be used in anatomy education. Keywords. CvhSlicer, Anatomy navigation system, CPU-GPU balancing scheme

1. Introduction Compared with radiological images (e.g. CT/MR and ultrasound images), Visible Human (VH) data [1,2] is of unique value in research and education of human anatomy because of the ﬁne anatomical details presented in the high-resolution cross-sectional photos. Developing an interactive navigation system to visualize the cross-sectional anatomy in VH data, however, is a challenging task, especially on a standard personal computer (PC). The system, which should allow real-time navigation of the cross-sectional anatomy, is challenging due to the huge size of VH data and limited computation resources. Several navigation systems have been developed based on the data achieved from the Visible Human Project, such as the Real-time Slice Navigator [3] and the virtusMed [4]. The main limitation of these two systems is that the resolution of rendered sections is relatively low, which is not able to present small but important anatomical structures.In this paper, we aim to develop an interactive navigation system for high resolution Chinese Visible Human (CVH) data, which runs on common PCs equipped with commodity

Q. Meng et al. / CvhSlicer: An Interactive Cross-Sectional Anatomy Navigation System

Figure 1. An overview of the CvhSlicer system.

355

Figure 2. VH data compression. (a) Compression steps. (b) Cut Tree for bounding box.

Graphics Processing Units (GPUs). In order to load the whole-body data to a PC memory, compression techniques are applied on the huge high-resolution CVH data, without compromising anatomical details. Given time-critical interaction requirement and large size of the CVH data, an adaptive load-balancing scheme between GPU and CPU is implemented to maximize the performance of the system. Experiments demonstrate that users can interactively observe the cross-sectional anatomy from any given position and direction in a real-time manner.

2. System Overview The proposed system is designed with two main modules, namely VH data compression module and slice pixels computation module (see Figure 1). As frequent data exchange between the main memory and hard disk is too time-consuming for the system to achieve real-time performance, an alternative approach is to load all data to the main memory. However, this is not possible when the data size is very large. Generally, the size of VH data can be up to dozens, even hundreds, of GB. Thus, data compression is indispensable. In the proposed system, an almost lossless image compression procedure is ﬁrst adopted to compress the VH data, and then the compressed VH data is fully loaded to the main memory for subsequent slice pixels computation. In some cases, even after compression, the data for rendering some large images is still too large to be loaded into the memory of a commodity GPU. To the end, we design a computation balancing method to make use of the computational power of both GPU and CPU for resultant image rendering.

3. Visible Human Data Compression Visible human images, which are photos of transverse section of a human cadaver, are captured with ultra-ﬁne resolution (up to 100 microns). In order to retain most of the informative anatomical detail, it is necessary to compress the data and load it to main memory of a PC. In our framework, the compression is performed in three steps: (i) Simple scaling, (ii) Cut Tree with bounding boxes and (iii) DXT algorithm (as shown in Figure 2(a)). Visible human images are scaled ﬁrstly. Then, as shown in Figure 2, there are large areas of background in a typical VH image. Bounding boxes are used to detect useful blocks (i.e., foreground regions) and only the pixels in these blocks will be loaded to the main memory. In order to obtain bounding boxes that do not overlap with each other, we

356

Q. Meng et al. / CvhSlicer: An Interactive Cross-Sectional Anatomy Navigation System

deﬁned two cutting operations: Column Cut and Row Cut, which are applied on a VH image alternately and recursively to form a Cut Tree. If neither Column Cut nor Row Cut can cut (i.e., separate) an image block, the cutting operations are terminated. Finally, the leaves of the Cut Tree are the bounding boxes we need. Figure 2(b) is a demonstration of the Cut Tree. Finally, the DXT algorithm [5] is used to perform a further compression. The DXT algorithm is advantageous in its low quality loss and the capability of preserving image details. More importantly, decompression of DXT is very fast and has little effect on cutting slice pixels calculation. After the three steps compression, VH data is compressed to about 1GB, which can be easily loaded into a PC main memory.

4. Slice Pixels Calculation Based on the compressed VH data being loaded into the main memory, the color of each pixel on the slice can be calculated. For a single pixel, it is easy to ﬁnd out the corresponding “voxel” located in the “VH volume”. In order to get smooth images, trilinear interpolation is used for interpolating the 6 voxels around the target pixel. Since each pixel on the slice can be calculated independently, thus the whole slice can be calculated in parallel. In order to obtain a real-time performance, the computation task is distributed to both CPU and GPU based on parameters of computing resources such as the GPU’s memory size and the number of CPU cores. 4.1. CPU-GPU Cooperative Computation Framework In GPU-based computation, only memory on the display card can be accessed. The VH data being loaded to the main memory has to be copied to the display card memory for computation. However, the memory on the display card is usually not big enough to store the whole compressed VH data. In other words, for some large slices, GPU can not compute all the pixels at once. Since the CPU and GPU computation can be performed in parallel, the computation task can be accomplished cooperatively by CPU and GPU. Initially, the compressed VH images are loaded to a PC main memory from the hard disk. Afterwards, the VH images around the initial slice are loaded to the display card memory from PC main memory. The loaded data size depends on the size of the display card memory. Once the initial loading ﬁnishes, users can move and rotate the slice freely by using a Virtual Reality (VR) Input Device. The CPU computation starts as soon as the computation balancing ﬁnishes. In GPU, unused VH images are ﬁrst deleted from the display card memory. In order to enable the GPU to compute as many pixels as possible, an adaptively determined number of VH images are then copied to display card memory from PC main memory. Results from both CPU and GPU are rendered to the screen directly. 4.2. CPU-GPU Computation Balancing Method In CPU, pixels are calculated on multiple cores in parallel. According to Amdahl’s Law nC tC , where [6], ideally, the time for computing all the pixels in parallel is: TC (nC ) = P C TC is the computational time for one image on CPU, nC is the number of pixels calcu-

Q. Meng et al. / CvhSlicer: An Interactive Cross-Sectional Anatomy Navigation System

357

lated on CPU, PC is the number of CPU cores, and tC is the computation time for one pixel. However, in practice, as parallel overhead [6] has to be considered, it requires additional time to make the sequential program run in parallel. Hence, a more accurate representation of the total CPU computation time should be: 1 TC (nC ) = OC · PC + nC tC (1) PC The parallel overhead is denoted by OC · PC where OC is assumed to be a constant. In practice, OC and tC can be calculated ofﬂine. We run the program in serial and an average computing time for one pixel can be obtained, and it is tC . On the other hand, the parallel overhead constant OC can be obtained by running the program in parallel. To obtain the GPU calculation time, we also need to consider the time of deleting unused VH images and the time of copying useful images into the GPU memory, which can be calculated as: q s 1 nG tG S(i) + B S(i) + OG · PG + (2) TG (nG ) = A PG i=p i=r where S(i) is the size of the i-th VH image’s size. p, q, r, s are images indices, indicating images within [p, q] will be deleted from the display card memory, and images within [r, s] will be copied from the PC main memory to the display card memory. A and B are coefﬁcients. In other words, the ﬁrst term in Eq.(2) corresponds to the time of “deleting unused data from the GPU memory”; the second term corresponds to the time of “copying data to the GPU memory”; and the third term of Eq.(2) corresponds to the time of “GPU computation”. In Eq.(2), the values of tG , OG , A and B can be determined ofﬂine. In our GPU computation model, each pixel is computed on a single thread and the average thread running time can be obtained as tG . OG can be calculated in the same way as OC . We pre-compute the deleting and copying time for various numbers of slices in order to estimate A and B ofﬂine. p, q, r, s are calculated in runtime, i.e., during the calculation balancing stage. The objective of the computation balancing is to minimize: max {TC (nC ) , TG (nG )}

(3)

Since most parameters in Eq.(1) and Eq.(2) can be calculated ofﬂine, the computation balancing aims at optimizing p, q, r, s in order to minimize the total computational time. A binary search algorithm [7] is developed to search the optimal values of them.

5. System Implementation and Experimental Result The CvhSlicer System was developed according to the architecture illustrated in Figure 1. As shown in Figure 3, there are two screens in the system – one for displaying the 3D CVH model, and the other for displaying the cross-sectional images. Users can trans-

358

Q. Meng et al. / CvhSlicer: An Interactive Cross-Sectional Anatomy Navigation System

200

Computation Time (ms)

150

CPU−Serial CPU−Parallel CPU−GPU−Balance

100

50

0 100x100

Figure 3. A photo of the CvhSlicer system.

200x200

400x400 Slice Size

800x600

1200x800

Figure 4. Performance Comparison.

late, rotate and scale the 3D CVH model by mouse control. A SensAble Phantom Omni is used to manipulate a multi-planar reconstruction (i.e., translate or rotate the cutting plane) on CVH model. Tactile output can also be provided in form of a feedback force rendered on the Omni in order to enhance realism of our navigation system. We have tested CvhSlicer with the Chinese Visible Human Female (CVHF) dataset, which was obtained at 0.5mm intervals. There were 3640 slices in the CVHF dataset, each with 3872 × 2048 pixels, and the total size is about 71.6GB. We ﬁrst rescaled the data to 1/3 of the original size, which was further compressed to 1.13GB. Experimental results demonstrated that, for different sizes of slice, the system can still give real-time rendering. Figure 4 shows a comparison of the performance among CPU-Serial, CPU-Parallel, and the proposed CPU-GPU-Balance computation modes. All the experiments were performed on a PC with a Intel(R) Core(TM) 2 CPU, 2.66GHz, and a GPU of Nvidia Geforce 8800 GTX. From this comparison, it is observed that the CPU-GPU-Balance model achieved the best performance and the fastest rendering time for different sizes of slice. Acknowledgements The work described in this paper was fully supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region (Project No.CUHK412510). References [1] M. J. Ackerman, “The visible human project: a resource for education”, J. Acad. Med. Jun. 1999, vol. 74, pp. 667-670 [2] S. X. Zhang, P. A. Heng, “The Chinese Visible Human (CVH) datasets incorporate technical and imaging advances on earlier digital humans”, J. Anat, March 2004, vol. 204, pp. 165-173 [3] R.D. Hersch, B. Gennart, O. Figueiredo, M. Mazzariol, J. Tarraga, S. Vetsch, V. Messerli, R. Welz, L. Bidaut, “The Visible Human Slice Web Server: A ﬁrst Assessment”, Proceedings IS&T/SPIE Conference on Internet Imaging, San Jose, Ca, Jan. 2000, SPIE vol. 3964, pp. 253-258 [4] virtusMED http://www.umi.cs.tu-bs.de/virtusmed [5] L. Renambot, B. Jeong, J. Leigh, “Real-Time Compression For High-Resolution Content”, Proceedings of the Access Grid Retreat 2007, Chicago, IL, May, 2007 [6] C. Barbara, Using OpenMP : Portable shared memory parallel programming, The MIT Press, 2008, pp. 33-34 [7] S. G. Akl, H. Meijer, “Parallel Binary Search”, IEEE Transactions on Parallel and Distributed Systems, April 1990, vol. 1, no. 2, pp. 247-250 [8] NVIDIA Corporation, NVIDIA CUDA Programming Guide, 3rd ed. Santa Clara, CA. 2009

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-359

359

Generation of Connectivity-Preserving Surface Models of Multiple Sclerosis Lesions Oscar Meruvia-Pastora, Mei Xiaoa,1, Jung Soha, Christoph W. Sensena a SunCenter of Excellence for Visualgenomics Department of Biochemistry and Molecular Biology Faculty of Medicine, University of Calgary 3330 Hospital Drive NW Calgary, Alberta, T2N 4N1, Canada

Abstract. Progression of multiple sclerosis (MS) results in brain lesions caused by white matter inflammation. MS lesions have various shapes, sizes and locations, affecting cognitive abilities of patients to different extents. To facilitate the visualization of the brain lesion distribution, we have developed a software tool to build 3D surface models of MS lesions. This tool allows users to create 3D models of lesions quickly and to visualize the lesions and brain tissues using various visual attributes and configurations. The software package is based on breadth-first search based 3D connected component analysis and a 3D flood-fill based region growing algorithm to generate 3D models from binary or non-binary segmented medical image stacks. Keywords. Multiple Sclerosis, Lesions, Connected Components, Segmentation

1.

Introduction

Medical image slices are frequently reconstructed into a 3D surface model to facilitate realistic visualization. Such reconstruction usually results in a single 3D model built from all regions of interest in a stack. When the target object is an anatomically welldefined structure, the reconstructed 3D model can be explored as is. However, when the reconstructed model represents a randomly distributed collection of objects with similar characteristics, such as brain lesions formed by the progression of multiple sclerosis (MS) ([1, 2]), it is useful to select certain clinically meaningful portions of the model to investigate in more detail. Topological information contained in 3D image stacks is very useful. For example, instead of visualizing all the lesions together, users might want to distinguish each individually connected lesion from the whole model. It will be useful for the users to visualize the selected brain lesions and other brain tissues such as the cortex and subcortex with different visual attributes such as color and transparency. Connectivity can play a key role in these cases and we need well-connected models in order to

1

Corresponding Author: [email protected]

360

O. Meruvia-Pastor et al. / Generation of Connectivity-Preserving Surface Models of MS Lesions

separate them easily and quickly. Therefore, a well formatted surface mesh structure is needed. The goal of our tool is to quickly and accurately create 3D models from a segmented medical image stack, such that connected 3D regions are easily separable from the whole model. In this case, the users can easily select the lesions and related brain tissues and view them in any visual configurations they want. There are two main algorithms that can be used to create 3D models of MS lesions from MRI stacks: the marching cubes algorithm and its various extensions ([3-9]), and region-growing based surface creation algorithms ([10-13]). Most marching cubes algorithms work on binarylabeled medical image stacks. However, to study the dynamics of MS lesion changes, it is important to use non-binary segmented image stacks that can store both lesion and brain tissue information. Region growing algorithms can work on non-binary segmented stacks. We have developed a method for quickly generating surface models from a medical image stack, which can preserve connectivity of voxles such that connected components of a model can be easily selected and visualized. Our method uses breadthfirst search based 3D connected component analysis and 3D flood-fill based region growing.

2.

Materials and Methods

MS patients were recruited for MRI scanning in Halifax, Nova Scotia, Canada. Each patient has been scanned six times at an interval of once a month. T1, T2, and T2 FLAIR MRI images were obtained for use in our model creation tool. A sequence of image processing steps was performed on the head MRI data generated from each scan session, to convert them to input image data for our software. First, the brain area was separated from each scan by using the FMRIB (Functional MRI of the Brain) Software Library (FSL, http://www.fmrib.ox.ac.uk/fsl). This step was performed on the T1 image set. Second, after brain extraction from the images, for each patient, all the scans were registered together. This step was required in order to make meaningful comparisons when visualizing changes in MS lesions of a patient from one scan session to the next. Third, we performed segmentation of the MRI scans to retrieve these types of tissues: cerebrospinal fluid (CSF), cortex, and sub-cortex. Finally, MS lesions were manually segmented from the FLAIR images based on white matter hyperintense areas. The T2 images were consulted frequently, since lesions that were not as clearly bright on FLAIR could be quite apparent on T2. T2 images were also used for confirming lesions identified on FLAIR. The pipeline of the model creation program consists of two major processing steps (see Figure 1): •

A stack of 2D images that contains the MS lesions goes through 3D connected component analysis. The breadth-first search based 3D connected component analysis will separate the MS lesions into several connected components and a seed for each component is saved into a text file.

•

The stack of 2D images in the previous step and the text file that contains the seed information are used in the flood-fill based region growing algorithm. A 3D model that contains all the MS lesions is created and saved into an OBJ file.

O. Meruvia-Pastor et al. / Generation of Connectivity-Preserving Surface Models of MS Lesions

361

Figure 1. Procedure for building the lesion models.

The flood-fill based region growing algorithm starts by finding the boundary of the region beginning from the seed. Voxels of intensity within a certain range from the seed are searched for until the boundaries are met. For each boundary voxel with a coordinate (x, y, z), eight surrounding points (x ± 0.5, y ± 0.5, z ± 0.5) are used to created faces. Those points are used as the vertices for the mesh. Each surrounding point is uniquely indexed according to the voxel coordinate (x, y, z). In order to avoid using duplicated vertices, the index and positions of those surrounding points are saved in a hash table. For each boundary voxel, two triangles are created by traversing the surrounding points clockwise in order to guarantee a closed and consistent mesh. The final mesh vertices are saved in an indexed triangle array. At the end we have a mesh writer to save the contents of the indexed triangle array into an OBJ file, which can be display by most 3D mesh visualization and processing tools. The resulting surface might have obvious staircase effects, where a smoothing filter can be applied to each voxel to lessen the effects. The number of smoothing steps can be specified by users.

3.

Results

We have created a Java3D™-based 3D medical image processing and visualization software package for facilitating brain lesion studies such as those required in MS. Although there are many automatic image segmentation programs developed for handling medical images, MS lesions usually have fuzzy boundaries (see Figure 2), and hence they still need to be manually segmented by neuroscience experts. The segmented images usually contain information on multiple brain tissue types. Each tissue type may include pixels in certain intensity value ranges, resulting in non-binary image stacks. Our program can be used to detect the 3D connected components in those

362

O. Meruvia-Pastor et al. / Generation of Connectivity-Preserving Surface Models of MS Lesions

non-binary segmented images. Once the users set up the intensity range for the tissue type they are interested in detecting, a breadth-first search algorithm is applied to the image stack, recursively searching the 26 surrounding pixels in an image stack. After the search, the total number of connected components found and the starting pixel of each component will be saved in a text file. If the users are interested, it is also possible to record the volume of each component and even the positions of pixels comprising each component.

Figure 2. A slice from the MRI scan of an MS patient. Highlighted by the circles are inflammation areas.

The locations and volumes of MS lesions are of great significance for clinical studies to understand the damage to the brain tissues and cognitive abilities of the MS patients. By flood-fill based region growing from a pixel in an image stack, the boundaries of a separated MS lesion can be detected and triangular faces can be used to approximate the shape of that specific lesion. The mesh models that are created (see Figure 3) can be loaded into a 3D visualization program. By displaying and manipulating the visual attributes of 3D models for different types of brain tissues and lesions, MS lesion development patterns can be investigated in the context of surrounding anatomical structures. Polygonal mesh models also have an advantage for lesion pattern studies. Unlike volume rendered models, the inner structures of polygonal models allow us to start from one vertex and traverse all the connected vertices to extract the 3D connected component and visually distinguish it from the rest of the model, in real time. For example, in our software package, we have a simple interface that allows the user to double click on a polygonal model, such that the single lesion that is connected to the clicked point can be extracted from the whole lesion model and displayed in another visualization window (see Figure 4). The connectivity calculation required for this component separation feature is performed by using the vtkPolydataConnectivityFilter class from VTK ([14, 15]).

O. Meruvia-Pastor et al. / Generation of Connectivity-Preserving Surface Models of MS Lesions

363

Figure 3. Rendering of a cortex model generated by our surface model building method (left) and rendering of cortex and MS lesion models generated by our surface model building method (right).

Figure 4. A connected lesion can be selected and highlighted by double clicking on it.

4.

Discussion

3D models reconstructed from medical image stacks such as MRI and CT have been widely used in diagnosis and medical research. The marching cubes algorithm can be used to create high-resolution 3D mesh models. However, the marching cubes algorithm creates at least one triangle per voxel while it passes through the surfaces, which creates a huge amount of triangles ([13]). Depending on the implementation of the marching cubes algorithm, the mesh might not be consistent with the actual layout of the voxels. For example, Figure 5 shows two screen shots after applying vtkPolydataConnectivityFilter to two different mesh models. They both were generated from the same image stack (one MRI stack of MS lesion images). The left model was rendered using our approach. The right model was rendered using the vtkMarchingContourFilter. We can see that on the left image, once we double click on a lesion, the whole connected lesion is selected and its visual attributes can be changed as one single model. However, on the right image, once we double click on a lesion,

364

O. Meruvia-Pastor et al. / Generation of Connectivity-Preserving Surface Models of MS Lesions

only part of the lesion is selected as shown by the changed color, and the rest of the same lesion has not been selected as shown by the same color as the rest of the whole model. If the connectivity of mesh models cannot be fully detected, the developmental patterns of the MS lesions can not be accurately expressed.

Figure 5. Comparison of connectivity in surface models. The model on the left is created using our surface model building software. The selection can clearly separate a connected component from the whole model. On the right, the model is created using the VTK marching cubes algorithm. The selection retrieves only part of a connected component from the whole model.

Marching cubes algorithms are usually applied to a whole image stack and every voxel will be checked. Therefore, for MRI and CT image stacks, it usually takes a long time to create a high-resolution mesh. Some versions of the marching cube applications also try to find the connected components, but processing a high-resolution mesh to find the connected components is very time-consuming and needs a large amount of computing resources. We tested a marching cube based program Afront ([16]) on the same MS lesion image stack. It took longer time and did not create a similar mesh as shown in Figure 5. Lewiner et al (2003) ([5]) developed a software package to create topologically correct manifold mesh. We tested the C++ implementation of the program. For any implicit functions applied to a grid, the resulting surface is consistent and the quality of the mesh seems to be very good. However, they did not provide any mesh creation method from medical image stacks. Therefore, the rendering time for our MS lesion image stacks in their program is not known. Our software provides a valuable tool to study disease development by creating 3D models of the observed patterns in medical images. By building consistent mesh models quickly and efficiently directly from a medical image stack, researchers can create a large number of models that represent different individuals and different developmental stages of a disease. The ability to retrieve connected components from such a model is a key feature of our software which enables the user to effortlessly focus on a clinically important part from the whole image stack. In addition to double clicking a certain potion of the model to highlight the region of interest, other cutting-based model creation tools will add more flexibility to our method. For example, Xiao et al (2010) ([17]) provided a generic model building

O. Meruvia-Pastor et al. / Generation of Connectivity-Preserving Surface Models of MS Lesions

365

algorithm. By using a virtual dissection method, various models can be built quickly and efficiently. Together with our current model building tool, a highly flexible virtual dissection and selection based biological scene creating tool can be developed for studies and discoveries of disease patterns.

Acknowledgement This work has been supported by Genome Canada through Genome Alberta; Alberta Science and Research Authority; Western Economic Diversification; the Governments of Canada and of Alberta through the Western Economic Partnership Agreement; the iCORE/Sun Microsystems Industrial Research Chair program; the Alberta Network for Proteomics Innovation; and the Canada Foundation for Innovation. We thank Heather Angka, Carl Helmick, Jordan Fisk and John Fisk for MRI data acquisition and processing. We also thank Megan Smith for comments on the manuscript.

References [1] M. Inglese, R. I. Grossman, M. Filippi, Magnetic resonance imaging monitoring of multiple sclerosis lesion evolution, Journal of Neuroimaging 15(4 Suppl) (2005), 22S-29S. [2] F. Zipp, A new window in multiple sclerosis pathology: non-conventional quantitative magnetic resonance imaging outcomes, Journal of the Neurological Sciences 287(1 Suppl) (2009), S24-S29. [3] H. C. Hege, M. Seebass, D. Stalling, M. Zöckler, A generalized marching cubes algorithm based on non-binary classifications, Konrad-Zuse-Zentrum für Informationstechnik Berlin Technical Report SC97-05 (1997). [4] W. E. Lorensen, H. E. Cline, Marching cubes: a high resolution 3D surface construction algorithm, ACM SIGGRAPH Computer Graphics 21(4) (1987), 163-169. [5] T. Lweiner, H. Lopes, A. W. Vieira, G. Tavares, Efficient implementation of Marching Cubes’ cases with topological guarantees, Journal of Graphics Tools 8(2) (2003), 1-15. [6] G. M. Nielson, Dual marching cubes, In Proc. IEEE Conf. Visualization (2004), 489-496. [7] G. M. Nielson, On marching cubes. IEEE Transactions on Visualization and Computer Graphics 9(3) (2003), 283-297. [8] S. Raman, R. Wenger, Quality isosurface mesh generation using an extended marching cubes lookup table, Computer Graphics Forum 27(3) (2008), 791-798. [9] S. Schaefer, J. Warren, Dual marching cubes: primal contouring of dual grids, In Proc. 12th Pacific Conf. Computer Graphics and Applications (2004), 70-76. [10] M. del Fresno, M. Venere, A. Clausse, A combined region growing and deformable model method for extraction of closed surfaces in 3D CT and MRI scans. Computerized Medical Imaging and Graphics 33(5) (2009), 369-376. [11] I. Cohen, D. Gordon, VS: a surface-based system for topological analysis, quantization and visualization of voxel data, Medical Image Analysis 13 (2009), 245-256. [12] B. Reitinger, A. Bornik, R. Beichel, Consistent mesh generation for non-binary medical datasets, In: Bildverarbeitung für die Medizin (2005), 183–187. [13] Y. Xi, Y. Duan, A novel region-growing based iso-surface extraction algorithm, Computers & Graphics 32(6) (2008), 647-654. [14] S. Pieper, B. Lorensen, W. Schroeder, R. Kikinis, The NA-MIC Kit: ITK, VTK, Pipelines, Grids and 3D Slicer as an open platform for the medical image computing community, In: Proc. 3rd IEEE Int. Symp. on Biomedical Imaging (2006), 698-701. [15] W. Schroeder, K. Martin, B. Lorensen, The Visualization Toolkit, Prentice-Hall, 2006. [16] J. Schreiner, C. E. Scheidegger, C. T. Silva, High-quality extraction of isosurfaces from regular and irregular grid, IEEE Transactions on Visualization and Computer Graphics 12(5) (2006), 1205-1212. [17] M. Xiao, J. Soh, O. Meruvia-Pastor, E. J. Schmidt, B. Hallgrimsson, C. W. Sensen, Building generic anatomical models using virtual model cutting and iterative registration, BMC Medical Imaging 10(5) (2010).

366

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-366

A Comparison of Videolaryngoscopic Technologies David J. MILLER, PhDa,b, Nikola MILJKOVICc,d, Chad CHIESAc,d, Nathan SCHULTEc,d, John B. CALLAHAN, Jr. BSd, and Ben H. BOEDEKER, MD, PhDa,b,1 a Department of Anesthesiology, University of Nebraska Medical Center, Omaha, NE b Research Service, Omaha VA Medical Center, Omaha, NE c College of Information Science &Technology, University of Nebraska, Omaha, NE d The Peter Kiewit Institute and Complex, Omaha, NE

Abstract. Difficulty in managing the airway is a major contributor to the morbidity and mortality of the trauma patient. The video laryngoscope, with its camera at the distal tip of the intubation blade, allows the practitioner an improved view of the glottic opening during intubation. The image from this viewer is transmitted to a monitor, allowing the intubating practitioner to “see around the corner” of a patient’s airway. The purpose of the present study was to assess and compare the video quality of commercially available video laryngoscopy systems. It was found that between the STORZ C-MACTM and the Verathon GlideScope®, there was little difference between the video quality; the difference came down to user preference. Keywords. Video laryngoscope, intubation

Background Inadequate airway management is a major contributor to patient injury, morbidity and mortality [1-2]. Indirect laryngoscopy provides a method to improve the view of the glottic opening during intubation. The video laryngoscope has a camera or lens at the distal tip of the intubating blade. The image from this viewer at the distal tip is transmitted to a monitor. This permits the intubating practitioner to “see around the corner” during intubation [3]. The purpose of the study was to analyze commercially available laryngoscope products to determine functionality, ease of use and feature sets.

Methods & Materials A team of four investigators at the University of Nebraska (Omaha) and the Peter Kiewit Institute (Omaha, NE) performed simulated intubations using a number of video laryngoscopy systems. The analyzed systems included the GlideScope Portable GUL (Verathon Inc., Bothell, WA) (Figure 1), a prototype system developed by STORZ as a predecessor to their C-MAC™ (a standard STORZ Macintosh blade with USB

1

Corresponding Author: Ben H. Boedeker, MD, PhD, Professor, Department of Anesthesiology, Director, Center for Advanced Technology & Telemedicine, University of Nebraska Medical Center, 984455 Nebraska Medical Center, Omaha, NE 68198-4455, USA; E-mail: [email protected]

D.J. Miller et al. / A Comparison of Videolaryngoscopic Technologies

367

connectivity to an ultra mobile PC; “UMPC”) (Figure 2), and the STORZ C-MAC™ (KARL STORZ Endoscopy, Culver City, CA) (Figure 3).

Figure 1. The Verathon GlideScope®

Figure 2. Storz prototype to the CMAC™

.

Figure 3. The Storz CMAC™ (photo courtesy of KARL STORZ Endoscopy-America, El Segundo, CA).

Testing was performed with a Laerdal Difficult Airway Trainer™ (Laerdal Medical Corporation, Wappingers Falls, NY) in a setting that simulated difficult airways, adverse lighting conditions and various system configurations (e.g. maximizing screen contrast, minimizing screen brightness, maximizing screen color hue, etc.). The equipment was assessed based on the investigator’s perceptions of color, clarity and brightness of the onscreen image for each of the systems. Perceptions were

368

D.J. Miller et al. / A Comparison of Videolaryngoscopic Technologies

given one of three possible ratings: 1=High, 2=Moderate or 3=Low. The statistics were performed using a two-tailed Wilcoxon Rank Sum test for independent samples.

Results A summary of the test results are shown in Table 1. Statistical analysis showed that there was no statistical differences between image, clarity, color, brightness or overall score of any of the systems tested (=0.05).

Table 1. Summarized Results of Tested Video Laryngoscopes System

CMAC

GlideScope

UMPC

# of scenerios Clarity Color Brightness Ave of Total Score

8 2.13 1.75 2.50 6.38

8 2.38 1.38 2.38 6.13

8 1.88 1.75 1.88 5.50

SD

2.50

1.96

2.20

Conclusions Results showed that there were no significant differences in video quality between the three systems; thus, the choice of systems falls to user preference (which can vary from person to person) and qualitative analysis of features that are outside the scope of this study. Future investigations are planned to evaluate additional videolaryngoscopy solutions in an effort to create a platform-agnostic videolaryngoscopy suite.

References [1] [2] [3]

L. Hussain, A. Redmond. Are pre hospital deaths from accidental injury preventable? Br Med J 308 (1994), 1077-1080. C.G. Miller. Management of the difficult intubation in closed malpractice claims. ASA Newsletter 64 (2000), 13-16 & 19. B.H. Boedeker, S. Hoffman, W.B. Murray. Endotracheal intubation using virtual images: learning with the mobile telementoring intubating video laryngoscope. Stud Health Technol Inform 125 (2007): 4954. Published by IOS Press.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-369

369

Telemedicine Using Free Voice over Internet Protocol (VoIP) Technology David J. MILLER, PhDa,b, Nikola MILJKOVICc,d, Chad CHIESAc,d, John B. CALLAHAN, Jr., BSd, Brad WEBB, MPASa,b, and Ben H. BOEDEKER, MD, PhDa,b,1 a Department of Anesthesiology, University of Nebraska Medical Center, Omaha, NE b Research Service, Omaha VA Medical Center, Omaha, NE c College of Information Science & Technology, University of Nebraska, Omaha, NE d The Peter Kiewit Institute and Complex, Omaha, NE

Abstract. Though dedicated videoteleconference (VTC) systems deliver high quality, low-latency audio and video for telemedical applications, they require expensive hardware and extensive infrastructure. The purpose of this study was to investigate free commercially available Voice over Internet Protocol (VoIP) software as a low cost alternative for telemedicine. Keywords. Telemedicine, Voice over laryngoscope, VTC, C-MAC, Skype™

Internet

Protocol

(VoIP),

video

Background There has been a recent increase in telemedicine projects, ranging from instruction and distance mentoring to performance of medical procedures [1-3]. These require a large amount of Internet bandwidth and expensive hardware to perform. Voice over Internet Protocol (VoIP) uses existing Internet infrastructure to transmit voice/video images over a distance. Though the technology has been used previously to provide medical care to patients far separated from definitive care centers, the systems developed were designed for the specific user requirements and took time to develop and perfect. The purpose of this study was to analyze the feasibility of using a commercially available VoIP technology to provide telemedical care.

Material & Methods Researchers at the University of Nebraska Medical Center (UNMC) and the Peter Kiewit Institute (PKI) downloaded the free version of SkypeTM, a VoIP software package that allows full duplex video and audio communication between users (www.skype.com). PKI used a typical PC on a wired LAN with 2GB of RAM, a 2.66 GHz Intel® Core™ 2 Duo processor running Windows XP Professional and a

1 Corresponding Author: Ben H. Boedeker, MD, PhD, Professor, Department of Anesthesiology, Director, Center for Advanced Technology & Telemedicine, University of Nebraska Medical Center, 984455 Nebraska Medical Center, Omaha, NE 68198-4455, USA; E-mail: [email protected]

370

D.J. Miller et al. / Telemedicine Using Free Voice over Internet Protocol (VoIP) Technology

microphone with external speakers, while the UNMC researcher (located in Long Beach, CA) used a notebook with an Intel® CoreTM 2 Duo CPU (2.20GHz) with 1.99 GB of RAM, integrated stereo speakers, an integrated 1.3 mega pixel web camera with digital microphone, and an AT&TTM 3G cellular broadband access card, connected at 3.6 MBps with a signal strength of -92dBm. Instead of a web camera, PKI used a CMAC™ video laryngoscope blade (KARL STORZ, Tuttlingen, Germany) with an NTSC/USB conversion module. The second setup was between two offices at UNMC, both on the wired LAN. The first computer was the notebook listed above (UNMC 1), while the other was a notebook with an Intel® CoreTM 2 Duo CPU (2.40GHz) 4.00 GB of RAM, and integrated stereo speakers (UNMC 2). Each user was using a MicrosoftTM LifeCam VX-5000 1.3 megapixel web camera with integrated digital microphone. The web camera was pointed at the C-MAC screen and video was transmitted without direct connection to the source computer. In both trials, one participant (PKI and UNMC 1, respectively) was the “student” being taught intubation, while the other participant (UNMC 1 and UNMC 2, respectively) was the “instructor” who was directing the student’s actions via VoIP to properly intubate a Laerdal Difficult Airway Manikin (Laerdal Medical Corporation, Wappingers Falls, NY).

Results It is possible to instruct the basics of airway management, both by video demonstration and audio instruction, using a 3G cellular or local area network and using a nonintegrated webcam as an “analog” transmission method. The various setups and the resultant video quality are illustrated by Figures 1-3.

Figure 1. SkypeTM interface showing view from USB-enabled C-MAC blade over 3G cellular broadband network.

Figure 2. (a) View of glottic opening visible on C-MAC screen during instruction; (b) View of glottic opening as seen during SkypeTM intubation session.

D.J. Miller et al. / Telemedicine Using Free Voice over Internet Protocol (VoIP) Technology

371

Figure 3. “Non-connected” SkypeTM setup including: (1) C-MAC monitor, (2) webcam pointed at C-MAC screen, (3) notebook PC and (4) intubation mannequin.

Image quality and video lag times were evaluated by an anesthesiologist from UNMC and were deemed acceptable for use in training and could be used for clinical practice, given the appropriate clearances by the FDA and local IRB.

Conclusions Providing instruction in airway management does not require high-dollar video teleconferencing equipment. Although image quality is less than that of expensive VTC equipment, it is more cost effective and can be used over long distances and using commercially available technology to provide video and audio, adequate to teach and perform intubations.

References [1] [2] [3]

B.H. Boedeker, S Hoffman, W.B. Murray. Endotracheal intubation training using virtual images: Learning with the mobile telementoring intubating video laryngoscope. Stud Health Technol Inform 125 (2007), 49-54. Published by IOS Press. R.E. Link, P.G. Schulam, L.R. Kavoussi. Telesurgery remote monitoring and assistance during laparoscopy. The Urologic Clinics of North America 28 (2001), 177-188. R.H. Taylor, J. Funda, B. Eldridge, S. Gomory, K. Gruben, D. LaRose, et al. A telerobotic assistant for laparoscopic surgery. Engineering in Medicine and Biology Magazine, IEEE 14 (1995), 279-288.

372

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-372

iMedic: A Two-Handed Immersive Medical Environment for Distributed Interactive Consultation Paul MLYNIECa,1, Jason JERALDa, Arun YOGANANDANa, F. Jacob SEAGULLb, Fernando TOLEDOc, Udo SCHULTHEISc a Digital ArtForms b University of Maryland c Wichita State University

Abstract. We describe a two-handed immersive and distributed 3D medical system that enables intuitive interaction with multimedia objects and space. The system is applicable to a number of virtual reality and tele-consulting scenarios. Various features were implemented, including measurement tools, interactive segmentation, non-orthogonal planar views, and 3D markup. User studies demonstrated the system’s effectiveness in fundamental 3D tasks, showing that iMedic’s two-handed interface enables placement and construction of 3D objects 4.5-4.7 times as fast as a mouse interface and 1.3-1.7 times as fast as a one-handed wand interface. In addition, avatar-to-avatar collaboration (two iMedic users in a shared space—one subject and one mentor) was shown to be more effective than face-to-face collaboration (one iMedic user/subject and one live mentor) for three tasks. Keywords. Two-handed interface, 3D interaction, 3D visualization, volume rendering, avatar collaboration, user studies

Introduction Under Congressional funding administered by the U.S. Army Telemedicine and Advanced Technology Research Center (TATRC), Digital ArtForms has developed an immersive 3D tele-consultation system called iMedic (Immersive Medical Environment for Distributed Interactive Consultation). iMedic enables remote experts to share 3D DICOM imagery and other 2D/3D assets via the Internet, immersing themselves in common data sets for purposes of diagnosis, planning, and education. To make it appropriate for use in medicine, the project required the development of certain common features such as volumetric rendering, clipping, and measurement capabilities. It also required the development of less common features such as 3D markup and intuitive two-handed DICOM data examination. The goal was to build a broadly capable system, applicable to a wide range of subspecialties such as radiology and surgical planning. Three user studies investigated the effectiveness of the system.

1

[email protected].

P. Mlyniec et al. / iMedic: A Two-Handed Immersive Medical Environment

373

1. The iMedic System An iMedic system consists of a PC, a set of Spacegrips™ (two 6-DOF handheld tracked controllers with 4 buttons each, see lower left corner of Figure 4), a visual display (optionally stereo), and the iMedic software. iMedic is built upon Digital ArtForms’ Two-Handed Interface (THI) software libraries that had previously enabled immersive CAD, command & control, and other applications. THI was believed to be a good fit for the spatially intensive visualization and interaction required for 3D medical problems. THI enables intuitive manipulation of objects and space through intuitive hand gestures, similar to that described by Mapes and Moshell [4]. Users intuitively and directly manipulate objects by simply reaching out and grabbing the objects. For navigation, Figure 1 shows a schematic for manipulating the viewpoint. Users translate the viewpoint by grabbing space with one or two hands. Users rotate the viewpoint by grabbing space with both hands and rotating about the midpoint between the hands. Likewise, users scale the world by grabbing space with both hands and moving the hands apart or bringing them closer together. Translation, rotation, and scale can all be performed simultaneously in a single gesture, enabling users to quickly set their viewpoint to a position, orientation, and scale of interest.

Figure 1. Translation, scale, and rotation of the scene.

The iMedic software utilizes a well-established set of libraries. In addition to object/viewpoint manipulation, the pre-existing libraries support the following features:

A 2D control panel (Figure 5) and widgets that float over the non-dominant hand for manipulation with the dominant hand

Tele-collaboration, including collaboration of objects and avatars, enabling two or more physically-separated users to interact in the same virtual scene (Figure 2) with independent viewpoints

A callback architecture that separates interaction, rendering, and application

Advanced real-time rendering through the use of OpenSceneGraph [5]

Various shaders

3D selection capability through the use of the RAPID collision library [3]

Support for various display types (CAVE™s, HMDs, Walls), with an option for stereo

Software/hardware options, scene/object definitions, and component customizations, all easily configured via XML configuration files

374

P. Mlyniec et al. / iMedic: A Two-Handed Immersive Medical Environment

Figure 2. Avatar collaboration. The user follows another user into musculature.

Figure 3. The Volumetric isosurface and maximum intensity rendering mixed with polygon rendering.

Some of the features implemented over the period of the project are listed below. 1.1. Volume Rendering We built upon basic volume rendering capability provided by OpenSceneGraph (direct volume, maximum intensity, and iso-surface renderings) to provide a powerful volume visualization tool, adding functionality that enables natural interaction with volumes. In most volume rendering applications [2], viewpoint manipulation (a 6-degree of freedom operation), happens via a 2D mouse and/or keyboard. Interactive techniques are limited with a mouse because there are no surfaces to project the mouse cursor onto. These limitations often result in unintuitive and awkward forms of interaction, resulting in loss of control for the user. iMedic solves this problem by enabling the user to move and rotate about arbitrary points in space and to walk through the volume and explore it in an intuitive, semi-physical manner. 1.2. Merging Volumetric Rendering with Polygonal Rendering Integrating polygonal and volumetric data requires a way of merging two independent rendering methods together into a coherent visualization. In particular, occlusion and transparency cues are important for providing a sense of depth when the hands (represented by 3D cursors) are placed in the volume. Our blending technique also

P. Mlyniec et al. / iMedic: A Two-Handed Immersive Medical Environment

375

enables drawing of transparent volumes and/or maximum intensity projections over iso-surfaces (Figure 3). 1.3. The Slicebox The Slicebox provides a means of rapidly exploring 3D DICOM datasets by passing a hand-held plane through the volume data and displaying the corresponding oblique cross-sectional imagery on that plane (Figure 4). In this way, the Slicebox is a quick and easy means of discovering the shape of anatomic structures that are not aligned to a principle axis of the data. 1.4. Segmentation and Reconstruction The Slicebox is the front-end for seed planting in our segmentation / reconstruction tool. Users drop seeds inside the Slicebox on the cross section and then build a 3D surface construction from the volumetric dataset (Figure 4). THI enables the user to quickly input seeds (inside and outside the structure of interest) and to control the seed size. As the user drags the seed point across the cross-sectional plane, its statistical reliability is reflected by its brightness—the brighter the seed the higher its reliability. 1.5. The Spindle We found that the center of viewpoint rotation/scale, located between the hands, caused confusion for novice users. We added a visual representation that we call the “Spindle”. The spindle consists of geometry connecting the two hands, with a visual indication of the center point—the center of rotation/scale. In addition to visualizing the point between the two hands, the connecting geometry provides depth-occlusion cues that are directly mapped to the hands. These occlusion depth cues are especially important for monoscopic displays where stereo depth cues are absent. 1.6. Measurement and Marking Tools Interactive analysis in virtual environments often requires quantification of dataset features. For example, a user may want to count the number of vessels exiting a mass (e.g., a tumor), or to indicate areas of interest for further investigation. The system provides an array of tools to interactively mark/count (via dropped fiducial count markers) and measure linear segments, surface area, and angles in the data. 1.7. The Viewbox The Viewbox, similar to a world-in-miniature [9], is a way to “capture” a portion of the 3D world and view it from a second viewpoint. The user surrounds an area of interest with a stretchable box and replicates that space via the control panel. The Viewbox is manipulated just like any other object, affording the user an independent and simultaneous view of the scene. The user can also reach into the Viewbox and manipulate the space contained within, in the same way that he can with the standard view. The Viewbox enables users to maintain a micro and macro view of the scene simultaneously. Since the Viewbox space is a reference to the real scene, anything that

376

P. Mlyniec et al. / iMedic: A Two-Handed Immersive Medical Environment

happens in the world is reflected in the Viewbox and vice versa. The Viewbox can be attached to the panel so that it is carried with the user, or it can be attached to the viewpoint, resulting in a heads-up display of a secondary view of the world. 1.8. 3D Whiteboarding In typical mouse-based 3D painting, the projection of the cursor's 2D position onto 3D surface geometry determines the location to be painted. Such a solution can result in frustration for various reasons—for example, a paint stroke can result in a longer stroke than intended when the surface is tangential to the view direction. iMedic 3D Whiteboarding enables users to naturally mark up, or paint on, surface geometry (the canvas). Users can grab the canvas in one hand and simultaneously paint using the other hand, an action very similar to painting a solid object in real life. In addition to using the hand’s 3D cursor as a paintbrush, users can select any polygonal object in the environment for use as a stamp. This provides infinite options for brush styles since users can create new objects and then use them as custom-created brushes. The user can modify the size of the brush relative to the canvas by scaling the canvas and/or the object used as the brush. 1.9. 2D Multimedia (Images/Video)

Figure 4. The slicebox and vascular segmentation of a brain.

Figure 5. The user inside a colon, the control panel, and a 2D video.

Still images and video (Figure 5) can be manipulated just as any other object. In addition to displaying the image/video on an object in the world, the video is displayed on the video control panel, enabling the user to preview the video even when the video object is not visible. The user controls movie playback via a control panel that includes standard functionality such as play, reverse play, pause, scroll, and speed change. 1.10. Clipping Clipping is an important aspect of 3D medical visualization packages—it helps users survey a particular portion of a dataset in a selective manner. In traditional interfaces, users either analytically define the clipping object's position and orientation or use a mouse interface, often resulting in user frustration. In contrast, iMedic lets the user grab the clipping plane with either or both hands and move it within the volume to control its position and orientation in a manner similar to holding a plate. In this way, users can focus on the information in the dataset, rather than on the adjustment of

P. Mlyniec et al. / iMedic: A Two-Handed Immersive Medical Environment

377

control parameters. The system supports both surface geometry and volumetric data. In addition to clip planes, the system supports clip spheres and clip cylinders.

2. User Studies Three user studies were conducted to evaluate the system. We chose to focus the studies on the most fundamental aspects of interaction—object and viewpoint manipulation. Based on anecdotal evidence, we believed that THI would significantly outperform both a mouse interface and a one-handed wand interface, but no hard numbers existed. Studies 1-2 were designed to compare the effectiveness of various interface devices and paradigms for visualizing and interacting with 3D datasets. Study 3 was designed to evaluate avatar collaboration. 2.1. Study 1 (Novice Users) Study 1 [7], a preliminary study which used medical imaging of human anatomy from computer tomography (CT) scans, compared two interfaces: THI and Amira (a mature mouse interface commonly used in 3D medical imaging [1]). Novice users with medical background used each system to carry out navigation, visual search, and measurement tasks with abstract synthetic and anatomical objects. Results indicate that for novice users (n=25) with 15 minutes of training and practice, there was no clear advantage to either interface, due to large variability, in subjective or objective measures. However, a case study of an experienced user showed clear advantages in all tasks using THI. This implied that THI users require longer training and practice time to take advantage of the interface. 2.2. Study 2 (Trained Users for Docking and Construction Tasks)

Figure 6. Study 2 docking task (left) and construction task (right).

Study 2 [6] consisted of a docking (Figure 6, left) and a construction task (Figure 6, right) in which users placed geometric objects. Properly trained THI users (n=20, average training and practice time of 48 minutes) completed the tasks with THI, a mouse-driven interface, and a wand-based interface. For the docking task, THI was 4.7 times as fast as the mouse interface (p < 0.001) and 1.8 times as fast as the wand interface (p < 0.001). For the construction task, THI was 4.5 times as fast as the mouse interface (p < 0.001) and 1.3 times as fast as the one-handed wand interface (p < 0.005). We believe the advantage of THI over the wand was greater with the docking task than

378

P. Mlyniec et al. / iMedic: A Two-Handed Immersive Medical Environment

for the construction task because the docking task required more viewpoint navigation, deriving the full benefit of two-handed interaction with THI. For simple one-handed object manipulation, wand interfaces have functionality similar to THI. 2.3. Study 3 (Collaboration) Study 3 [8] compared performance of three collaborative tasks using iMedic under two collaboration methods: physical face-to-face and virtual avatar-to-avatar collaboration. In face-to-face collaboration, a subject used iMedic while guided by a mentor who was physically co-located with the subject. The mentor could interact with the subject using speech, gesture, facial expressions, and other means without restriction. In the avatar-to-avatar collaboration, the mentor was physically separated from the subject, and they could only interact via iMedic avatars within the same virtual space. Three mentored tasks were investigated: maze navigation, viewpoint replication, and stent placement. Preliminary results indicate that virtual collaboration led to significantly faster completion times for each of the three tasks. Furthermore, preliminary results suggest a higher success rate in virtual collaboration for the stent placement task, as compared to face-to-face collaboration. Full results will be reported elsewhere.

3. Conclusions We built a two-handed 3D medical collaborative multimedia system. This system can be used for surgical planning, radiological review, and tele-consultation. For fundamental 3D tasks, our interface is 4.5-4.7 times as fast as a mouse interface and 1.3-1.8 times as fast as a wand interface. Additionally, preliminary evidence suggests that iMedic collaboration is more effective than face-to-face collaboration for maze navigation, viewpoint replication, and stent placement.

References [1] Amira, 2010, Visual Imaging Software, http://www.amira.com, accessed July 2010. [2] K. Engel, M. Hadwiger, J. Kniss, C. Rezk-Salama, D. Weiskopf. Real-Time Volume Graphics, A K Peters, Ltd, 2006. [3] S. Gottschalk, M. Lin, D. Manocha. OBB-Tree: A Hierarchical Structure for Rapid Interface Detection, Proceedings of SIGGRAPH ‘96 (1996), 171-180. [4] Mapes, D., Moshell, J.M., 1995, A Two Handed Interface for Object Manipulation in Virtual Environments, Presence: Teleoperators and Virtual Environments 4:4 (1995), 403-416. [5] P. Martz, OpenSceneGraph Quick Start Guide: A Quick Introduction to the Cross-Platform open Source Scene Graph API, Skew Matrix Software, 2010. [6] Schultheis, U., Toledo, F., Mlyniec P., Jerald, J., Yoganandan, A., 2011. Comparison of a Two-Handed Interface to a Wand Interface and a Mouse Interface for Fundamental 3D Tasks, Submitted to ACM SIGCHI Conference on Human Factors in Computing Systems (2011). [7] F.J. Seagull, P. Miller, I. George, P. Mlyniec, A. Park, Interacting in 3D Space: Comparison of a 3D Two-handed Interface to a Keyboard-and-mouse Interface for Medical 3D Image Manipulation, Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 53:27 (2009), 2004-2008. [8] F.J. Seagull , Mlyniec, P., Jerald, J., Yoganandan, A., 2010. .“Comparison of virtual and face-to-face collaboration using the iMedic system.” Technical report #2010-02-TATRC. Los Gatos, CA: Digital Artforms. [9] R. Stoakley, M. Conway, R. Pausch, Virtual Reality on a WIM: Interactive Worlds in Miniature, Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (1995), 265272.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-379

379

Patient Specific Surgical Simulator for the Evaluation of the Movability of Bimanual Robotic Arms Andrea MOGLIA a,1, Giuseppe TURINI a, Vincenzo FERRARI a, Mauro FERRARI a, Franco MOSCA a a EndoCAS, Center for Computer Assisted Surgery, via Paradisa 2, 56124 Pisa (Italy)

Abstract. This work presents a simulator based on patient specific data for bimanual surgical robots. Given a bimanual robot with a particular geometry and kinematics, and a patient specific virtual anatomy, the aim of this simulator was to evaluate if a dexterous movability was obtainable to avoid collisions with the surrounding virtual anatomy in order to prevent potential damages to the tissues during the real surgical procedure. In addition, it could help surgeons to find the optimal positioning of the robot before entering the operative room. This application was tested using a haptic device to reproduce the interactions of the robot with deformable organs. The results showed good performances in terms of frame rate for the graphic, haptic, and dynamic processes. Keywords. Robotic surgical simulation; patient specific surgical simulation; surgical robots setup.

Introduction Pioneered by Computer Motion (Sunnyvale, California, Unites States) with the ZEUS system and become increasingly popular among surgeons over the past few years thanks to the da Vinci system by Intuitive Surgical (Sunnyvale, California, United States), surgical robotics represents a viable solution to complex minimally invasive surgery (MIS) [1]. This novel approach has several advantages over laparoscopy, including: improved manoeuvrability of instruments by allowing wristed and finger movements (da Vinci EndoWrist®), removal of trocar fulcrum effect (inversion of movements), tremor minimization, motion scaling, 3D visualization, and a more comfortable ergonomic position of the surgeon [2]. From the clinical viepoint the benefits of surgical robotics translate into safe and fine scale operations, reduced trauma, shortened recovery time, low level of fatigue for surgeons even after using the robot for prolonged time [3], allowing also to perform new kinds of MIS interventions. As in laparoscopy, in the setup stage before the operation a proper placement of trocars is helpful to prevent possible collisions between the robot ams, as documented in previous works concerning ZEUS and da Vinci robots [3] [4] [5]. The trend of traditional MIS, followed by robotic surgery, is to reduce the number of access ports as pursued by a novel approach called single incision laparoscopic 1

Corresponding Author: Andrea Moglia, EndoCAS, Center for Computer Assisted Surgery, via Paradisa 2, 56124 Pisa, Italy; E-mail: [email protected].

380

A. Moglia et al. / Patient Speciﬁc Surgical Simulator

surgery (SILS) [6]. The correct positioning of the robot is of paramount importance for SILS procedures due to the limited workspace of the robot arms, as exemplified by the bimanual robotic arm unveiled by Intuitive Surgical at International Conference on Robotics and Automation 2010 (Anchorage, Alaska, United States) and by the ongoing research activities within ARAKNES (Array of Robots Augmenting the KiNematics of Endoluminal Surgery) Project. In this paper we discuss the development of a simulator for a single port robot with bimanual abilities to validate its potential application, in particular to bariatric surgery and cholecystectomy. In particular, given a bimanual robot with its own geometry and kinematics, and a patient specific anatomy, the proposed simulator allows to evaluate in a virtual environment if a dexterous movability of the robot is achievable, avoiding collisions with the anatomy to prevent potential damages in the real surgical procedure. In addition it can help surgeons before entering the operative room to choose the optimal positioning of the robot and the access port in the abdominal wall. This simulator can be customized for any present or future bimanual surgical robots. The first prototype includes the following features: robot motion via inverse kinematics, robot motion tracking, patient specific virtual anatomy, deformable organs, haptic feedback, and customizable robot configuration. This work is carrying on within the aforementioned ARAKNES initiative and aims at realizing a realistic simulator of the final surgical robot for training and planning, based on patient specific biomechanical modeling.

Methods & Materials Modeling of the Virtual Scene A 55-year old man underwent a total body computed tomography (CT) with contrast agent (stomach insufflated with carbon dioxide (CO2)) at the Radiology Department of Cisanello Hospital in Pisa (Italy). The medical dataset was processed using our segmentation pipeline, developed customizing ITK-SNAP, a software for the generation of 3D virtual models (e.g. in STL format) from the stack of CT images (in DICOM format) [7]. Being the obtained 3D models quite raw, an optimization stage occurred before processing them by the algorithms generating the dynamic models. This task was performed through MeshLab (Visual Computing Lab, ISTI-CNR, Pisa, Italy) and Autodesk® Maya (Toronto, Ontario, Canada), and consists in mesh simplification, artefacts removal, and holes filling [8]. During this stage the complexity of the mesh was heavily reduced, enhancing the simulation performance without losing a good visual appearance. The virtual organs, with the same shapes and dimensions as those of a real patient, were placed in a virtual mannequin, generated after segmenting a CT acquisition of a commercial Phantom OGI by Coburger Lehrmittelanstalt (Coburg, Germany). The simulator enables the loading and rehearsal of different configurations of single port bimanual robots, inserted through the laparoscopic port by a rigid over-tube. Both robotic arms were modelled as a sequence of joints and links and terminate with an end effector. This was designed to host different surgical instruments for a wide range of specialized tasks, including gripper, forceps, and scalpels. A screenshot of the complete virtual scene is illustrated in Figure 1.

A. Moglia et al. / Patient Speciﬁc Surgical Simulator

381

Figure 1. Screenshot of the simulator.

The configuration properties of the robot arms are described in a script file (RBT, a customized format), loaded at simulation launching or whenever the user selects a different robot component, choosing the links and joints characteristics. Overall, the virtual scene is represented by a script file (PRJ, a customized format) describing the position, orientation and properties of each virtual organs, surgical robot configuration, laparoscopic camera preferences, and application settings. Algorithms and Data Structures For the purposes of this work we chose CHAI 3D, an open-source framework of C++ libraries for computer haptics, visualization and interactive real-time simulation.

Figure 2. Dynamic skeleton of the gallbladder, pointing out fixed spheres in red and those free to move in green (left). Spheres tree for collision detection between robot arms and gallbladder (right).

The dynamic model of the deformable objects is composed of a volumetric model of the object consisting of a skeleton of spheres (nodes) and links and generated automatically, using the filling sphere approach [9]. In our case, the volumetric model approximates the volume of the organ and its nodes and links are connected through

382

A. Moglia et al. / Patient Speciﬁc Surgical Simulator

elastic links to the vertices of the surface model, generated from the optimized organ surface with point masses on the mesh vertices and damped springs on the edges. This method decouples local deformations, modelled by the surface model and affecting a small surface portion, and global ones, modelled by the volumetric model and influencing a large portion of the surface [9]. In our simulation a dynamic skeleton was associated to stomach, liver, and gallbladder. In Figure 2 (left) the skeleton of gallbladder is represented, with red and green spheres indicating respectively fixed and movable nodes. For the purposes of our work, collision detection concerns only those organs within the workspace of the robot arms, in particular the deformable ones. The remaining organs are static and they do not participate in the collision detection. In particular collisions with the deformable objects are simplified considering only interactions with volume spheres of the volumetric model, as depicted in Figure 2 (right) [9]. In addition not all spheres of each organ participate in the collision detection phase, but only those which can be reached by the robot arms. In this way the number of frame rate, and consequently the performance, can be increased without losing realism during the interaction. The data structures for the collision detection of the robot arms were designed to model different levels of detail for each part of the arm, represented by a joint and the following link in the kinematic chain. This choice is due to the different probability of interaction of each part of the robot arm with both the deformable organs and the other arm. For example, by considering the interaction with the deformable organs, the tip has the highest probability of interaction while the over-tube the lowest. The data structure consists in a spheres tree with an appropriate level of detail, as depicted in Figure 2 (right). Additionally, self-collisions of an arm and itself and between the two robot arms were considered. The former were avoided implicitly by the robot kinematics, while the latter handled as standard collisions thanks to the independent collision detection data structure of each robot arm. Since the data structures for collision detection of both deformable organs and robot arms are based on spheres, the single tests are spheres vs. spheres intersections, resulting in the fastest computation. Each of these tests generates two force components, equal and opposite, resulting in the force feedback and the dynamic simulation of the deformable organs.

Simulation The presented simulation is composed of a graphic, a dynamic, and a haptic processes (Figure 3). The first deals with the user interface and the visualization of the virtual scene, using the scene graph provided by CHAI 3D, based on OpenGL. The second concerns collision detection, dynamics of deformable objects, inverse kinematics of robot arms, and force feedback. The haptic process ensures that the force feedback is updated with a proper frequency for a realistic interaction.

A. Moglia et al. / Patient Speciﬁc Surgical Simulator

383

Figure 3. Flowchart of the simulation processes.

The set up of the virtual environment includes an abdominal access for the single port bimanual robot with liver, stomach, and gallbladder as targets of the surgical procedures (bariatric surgery and cholecystectomy). In this regard, these organs were dynamically modelled to simulate the deformations caused by the interaction with the robot arms. All the physical models were generated offline and loaded during the simulation startup. The virtual scene comprises also the other organs of the upper abdomen, the backbone, the surgical robot, and a mannequin. The surgical robot used in this simulation has two arms, both with 6 d.o.f (degrees of freedom) and a generic gripper as end effector. The joints have the following configuration: shoulder (one roll and one pitch), elbow (one pitch), and wrist (one roll, one pitch, and one roll). The motion of the robot arms links is computed by inverse kinematics when controlling the end effector with the haptic interface.

Results The current targets of our surgical simulator within ARAKNES project are bariatric surgery and cholecystectomy. The former is a procedure performed in patients suffering from obesity. It can be divided into restrictive (gastric banding, vertical gastroplasty, sleeve gastrectomy), malabsorptive (biliopancreatic diversion), or a combination of both (gastric bypass) [10]. Bariatric surgery was selected since surgical robotics offers improved ergonomics over laparoscopy against large thick abdominal walls [2]. On the other hand, since robotic cholecystectomy has a proven safety and is a sort of benchmark for surgical devices, it was selected as the second target procedure [11]. The virtual scene includes patient specific organs. Since stomach, liver, and gallbladder are the target organs of the selected procedures, they are simulated as deformable objects in order to provide realistic interaction when the robot arms collide with them. The other organs are the pancreas and the kidneys. Our simulation was tested on a workstation running Microsoft Windows 7 (Intel Core i7 – 3 GHz, 12 GByte RAM, 2 GPU nVidia GTX 285) in a virtual scene composed of 52k vertices and 104k triangles. The dynamic skeleton of the deformable organs (stomach, liver, and gallbladder) was made up of 257 nodes and 1.217 links.

384

A. Moglia et al. / Patient Speciﬁc Surgical Simulator

Figure 4. Real-time visualization of the tracking of the robot arm motion (left). Workstation running the simulation and haptic device (right).

The graphic process provided a frame rate ranging from 30 to 50 fps, the dynamic process from 1.0 to 1.6 kHz, and the haptic process exceeded 1.0 kHz. The required memory to run in real-time the simulation was 94 MByte. During the simulation the user can activate the robot arms motion tracking. These data can be visualized in real-time (Figure 4 on the left) or saved in a log file for a postprocessing analysis. The arms of the bimanual robot are guided by a Falcon, a haptic interface by Novint (Albuquerque, New Mexico, United States), as shown in Figure 4 (right). The user has the possibility to select which arm to control via keyboard input.

Conclusions In this paper we described a simulation for the evaluation of the movability of bimanual surgical robots in a virtual anatomical district of a real patient and reconstructed after segmentation. The motion of the robot is computed by inverse kinematics and can be preoperatively simulated and rehearsed. Thanks to the integration with one haptic interface, surgeons can drag the robot arms in a consistent way with the kinematics of the real bimanual robot. The present version of our simulation enables surgeons to drag the arms of a bimanual robot in order to evaluate its movability to avoid collisions with the surrounding virtual anatomy, which might turn into damages during the real surgical procedure. This simulator provides also force feedback to users when they touch patient specific organs using the haptic device. Moreover, it can help surgeons to plan the optimal positioning of the robot support. Besides experiencing a real-time interaction of the robot arms with the anatomy with visual and haptic feedback, it is possible to perform a detailed analysis on the end effector trajectory and on the distance of each link from the surrounding tissues thanks to the data stored in the log file. The discussed work is in progress. Currently we are extending the dynamic model provided by CHAI 3D for the management of dynamic objects in order to represent deformations in a more realistic way. In particular we are optimizing the skeleton of deformable meshes, tweaking nodes and links properties, and the structure of the

A. Moglia et al. / Patient Speciﬁc Surgical Simulator

385

skeleton (number of nodes and links and their arrangement). On the other hand we are working on the tuning of the dynamic properties of the virtual organs, based on patient specific data, to reproduce the pathological states of the target surgical procedures of our simulator. Furthermore, since the modeling technique we are using does not lend itself to the implementation of complex tasks like mesh cutting, we are working on alternative approaches to pursue this objective. We are also integrating into the simulation other haptic devices available at our center, as the Freedom 6S by MPB Technologies (Montreal, Quebec, Canada).

Acknowledgment The research leading to these results has received funding from the European Community's Seventh Framework Programme (FP7/2007-2013) under grant agreement num. 224565 (ARAKNES Project). The authors would like to express a sincere thank to Dr. Lorenzo Faggioni for acquiring the CT dataset of DICOM images, and Ms. Marina Carbone for performing the segmentation of this dataset.

References [1]

C. Nguan, A. Girvan, P.P. Luke, Robotic surgery versus laparoscopy; a comparison between two robotic systems and laparoscopy, Journal of robotic surgery 1 (2008), 263-268. [2] E.B. Wilson, The evolution of robotic general surgery, Scand J Surg 98 (2009), 125-129. [3] M. Hayashibe, N. Suzuki, M. Hashizume, K. Konishi, A. Hattori, Robotic surgery setup simulation with the integration of inverse-kinematics computation and medical imaging, Comput Methods Programs Biomed 83 (2006), 63-72. [4] M. Hayashibe, N. Suzuki, M. Hashizume, Y. Kakeji, K. Konishi, S. Suzuki, A. Hattori, Preoperative planning system for surgical robotics setup with kinematics and haptics, Int J Med Robot 1 (2005), 7685. [5] A. Pietrabissa, L. Morelli, M. Ferrari, A. Peri, V. Ferrari, A. Moglia, L. Pugliese, F. Guarracino, F. Mosca, Mixed reality for robotic treatment of a splenic artery aneurysm, Surg Endosc 24 (2010), 1204. [6] M.B. Ostrowitz, D. Eschete, H. Zemon, G. DeNoto, Robotic-assisted single-incision right colectomy: early experience, Int J Med Robot 5 (2009), 465-470. [7] G. Megali, V. Ferrari, C. Freschi, B. Morabito, F. Cavallo, G. Turini, E. Troia, C. Cappelli, A. Pietrabissa, O. Tonet, A. Cuschieri, P. Dario, F. Mosca, EndoCAS navigator platform: a common platform for computer and robotic assistance in minimally invasive surgery, Int J Med Robot 4 (2008), 242-251. [8] P. Cignoni, M. Callieri, M. Corsini, MeshLab: an Open-Source Mesh Processing Tool, Sixth Eurographics Italian Chapter Conference (2008), 129-136. [9] F. Conti, O. Khatib, C. Baur, Interactive Rendering Of Deformable Objects Based On A Filling Sphere Modeling Approach, Proceedings of IEEE International Conference on Robotics and Automation (2003), 3716-3721. [10] M. Ibrahim, D. Blero, J. Deviere J, Endoscopic options for the treatment of obesity, Gastroenterology 138 (2010), 2228-2232. [11] S. Breitenstein, A. Nocito, M. Puhan, U. Held, M. Weber, P.A. Clavien. Robotic-assisted versus laparoscopic cholecystectomy: outcome and cost analyses of a case-matched control study, Ann Surg 247 (2008), 987-993.

386

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-386

CyberMedVPS: Visual Programming for Development of Simulators Aline M. MORAISa,1 and Liliane S. MACHADO b a, b Federal University of Paraíba, UFPB - Brazil

Abstract. Computer applications based on Virtual Reality (VR) has been outstanding in training and teaching in the medical filed due to their ability to simulate realistic in which users can practice skills and decision making in different situations. But was realized in these frameworks a hard interaction of non-programmers users. Based on this problematic will be shown the CyberMedVPS, a graphical module which implement Visual Programming concepts to solve an interaction trouble. Frameworks to develop such simulators are available but their use demands knowledge of programming. Based on this problematic will be shown the CyberMedVPS, a graphical module for the CyberMed framework, which implements Visual Programming concepts to allow the development of simulators by non-programmers professionals of the medical field. Keywords. Virtual Reality, Visual Programming, CyberMed.

Introduction A framework is defined as an abstract implementation for application development on a particular problem domain, with the advantage of reuse of components [6]. An important area of its application is the health sciences, particularly to the development of training simulators or tools to assist specific procedures. Thus, characteristics related to Virtual Reality (VR) as immersion, touch sense, 3D viewing, among others, are increasingly supported by these frameworks in order to assist training [1] [2], planning [3] [4] and assistance [5] tasks. It was observed that frameworks specifically conceived to development of medical simulators based on VR offer few or none resources of usability for non-programmer users. VP is divided in four components: Visual Expressions (VE), Visual Programming Language (VPL), Visual Programming Environment (VPE) and Visual Programming System (VPS). VE are sets of graphical elements called symbols; VPL are languages which have visual expressions; VPE are environments composed by visual expressions aggregated to textual language inputs; and a VPS is defined as a composition of VE in an absolutely graphical programming environment. The definition of which VP components will be adopted is essential for the development process of a VP module integrated to medical framework. A related works revision was made in scientific journals and papers between the years 2005 and 2010. The study was based in parameters like as presence and type of VP, VR features and functionalities of each one. This research found several frameworks utilized in health area which had a few VP technique or some type of 1

Corresponding Author: Aline M. Morais, UFPB/CCEN/LabTEVE, Cidade Universitária s/n, João Pessoa/PB – Brazil, 58051-900. Email: [email protected]

387

A.M. Morais and L.S. Machado / Visual Programming for Development of Simulators

visual resource. A comparison of the characteristics of these frameworks allowed identifying if they had approaches for medical profile and which VP technique was used, for example. Among the medical frameworks with visual approaches for programming found in scientific literature, the MeVisLab [3], SCIRun [4] and NeatVision [5] are characterized as frameworks with VPE due to the presence of visual expressions and code line input. Besides, the ViMeTWizard [2] is other medical framework which applies visual resources, but without a VP approach. Then, none of the tools analyzed are VPS and demands from users some knowledge of programming techniques for the utilization of their full capabilities. In parallel, interviews were conducted in order to identify needs related to tools for these professionals. The interviews showed that 58,06% of health professionals could use a tool with VR resources since this tool is easy to use. The bibliographic review and the interviews result allowed observing a lack of tools for health professionals that allow this public to develop by themselves applications based on VR for education and training. 1. The CyberMedVPS CyberMedVPS was designed to be used by health professionals to allow them to develop their own VR applications. This VPS was projected to be composed by the CyberMed, a framework to the development of VR systems based on PCs whose goal is supporting the creation of medical training applications, particularly those related to the simulation of medical procedures in immersive virtual environments [1]. CyberMed was chosen due to its support to stereoscopic visualization, haptics interactions, interactive collision detection and deformation, tracking devices, collaboration and, specially, user performance assessment. Additionally, this framework is stable, free and has been continuously developed and expanded. The conception of CyberMedVPS as a VPS extinguishes any necessity of textual code input to generate results and interaction happens only in graphical mode based on flowcharts. Thus, textual programming is not necessary for generation or execution of VR applications. Through several graphical components, users can connect boxes to make a flowchart representing steps of instructions required to execute a VR application. A text window (Fig. 1X-D) provides feedback with messages related to the flowchart execution: if there is some kind of error in serialization of instructions, the VPS will point it out in the text window. CyberMedVPS design also include the offer of an option to export the flowchart to textual code for programmer users who wish to modify the VR applications using the native language of CyberMed (C++). The validation and correction of the flowchart (Fig. 1Y) is done automatically when required by user. X

Y

Figure 1. Interface for CyberMedVPS (X) and example of boxes in a flowchart in CyberMedVPS (Y)

388

A.M. Morais and L.S. Machado / Visual Programming for Development of Simulators

The CyberMed components are offered to user by CyberMedVPS. These CyberMed components are able to generate the main configuration file, called main.cpp, and are based in design pattern MVC (Model, View and Core). Together these elements are useful to provide several VR functionalities to user. A changing of parameters is facilitated because the details of subcomponents are graphically elucidated with the usage of VP. 1.1. CyberMedVPS Architecture To the creation of CyberMedVPS was adopted a step sequence following: 1) Applying of a interview with health professionals in order to understand the real needs in RV tools; 2) Test with prototypes created from interview results; 3) CyberMedVPS implementation and 4) CyberMedVPS final test with the target public. As shown in Figure 2 the CyberMedVPS is composed by two layers: the graphical interface layer has all visual elements that can be manipulated by users to produce the medical applications and the integration of this layer to CyberMed depends on the communication layer that will relate the graphical elements of the flowchart to the framework commands. Thus, this communication layer allows bidirectional flow between the graphical interface and the framework.

Figure 2. CyberMedVPS functional architecture

2. Conclusions This project is under development and included in the National Institutes of Science and Technologies - MACC supported by the Brazilian National Council of Scientific and Technological Development. The results of the implementation should be evaluated by professionals to validate the project goals. This project was approved by the Ethics Committee of Lauro Wanderlei Universitary Hospital of Federal University of Paraíba, Brazil. References [1] D. F. L. Souza, I. L. L. Cunha, L. C. Souza, R. M. Moraes and L. S. Machado, A Framework for Development of Virtual Reality-Based Training Simulators, MMVR 17, pp. 174-176, 2009. [2] A. C. M. T. G. Oliveira and F. L.S. Nunes, Building a Open Source Framework for Virtual Medical Training, Journal of Digital Imaging, DOI = 10.1007/s10278-009-9243-3, 2009. [3] J. Rexilius, W. Spindler, J. Jomier, M. Konig, H. K. Hahn, F. Link and H. Peitgen, A Framework for Algorithm Evaluation and Clinical Application Prototyping using ITK, MICCAI Workshop, 2005. [4] C.R. Jonhnson and S.G. Parker, Applications In Computational Medicine Using Scirun: A Computational Steering Programming Environment, Proc. H.W. Meuer, editor, Supercomputer ‘95, p. 2–19. Saur-Verlag, 1995. [5] P. F. Whelan, R. J. T. Sadleir and O. Ghita, NeatVision: Visual Programming for Computer-aided Diagnostic Applications, Informatics in Radiology (infoRAD), Published online 10.1148/rg.246045021, 2004. [6] M. Mattsson, J. Bosch, Framework Composition: Problems, Causes and Solutions, Course Documentation. Technology of Object-Oriented Languages and Systems, 1997.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-389

389

A Bloodstream Simulation Based on Particle Method Masashi NAKAGAWA†, Nobuhiko MUKAI†, Kiyomi NIKI† and Shuichiro TAKANASHI‡ †Graduate School of Engineering, Tokyo City University, Japan ‡Department of Cardiovascular Surgery, Sakakibara Heart Institute, Japan E-mail: [email protected]

Abstract. Many surgical simulators use mesh method to deform CG models such as organs and blood vessels because the method can easily calculate the deformation of models; however, it has to split and reconstruct the mesh of the models when the model is broken such as bleeding. On the other hand, particle methods consider a continuous body such as solid and liquid as a set of particles and do not have to construct the mesh. Therefore, in this paper, we describe how to simulate bloodstream by using MPS (Moving Particle Semi-implicit) method that is one of particle ones. In the simulation, we use the aorta model as the blood vessel model, and the model is constructed with particles. As the result of the simulation, it took 20ms to deform the blood vessel and to simulate bleeding with the model that is constructed with 15,880 particles for the blood vessel and 6,688 particles for the blood. Keywords. computer graphics, virtual reality, surgical simulators, particle method

Introduction In these days, surgical simulators have been more useful for surgical training and preoperative planning because medical treatment is developed with engineering technology and surgeries are more complicated. Surgical simulators use CG (Computer Graphics) and VR (Virtual Reality) technologies, and give us chances for training or preoperative planning with patient’s data [1,2]. Most of the organ models used by these simulators are consisted of finite element model or mass spring model and we have performed faster blood vessel deformation by using mass spring model [3]; however, this model did not consider blood flow inside of the vessel and the inside of the vessel was constructed with an elastic body. Then, this model could not be used for more precise simulation with bleeding and blood flow. In some blood flow simulations, FEM (Finite Element Method) or LBM (Lattice Boltzmann Method) is used [4,5]; however, FEM requires to reconstruct the model when the topology changes and it takes much time especially for bleeding that combines some particles and/or breaks them into small ones. In addition, LBM takes too much time to calculate displacement of particles since the space needs to be divided into some voxies, which number depends on the size and the shape of the boundary. On the other hand, particle methods construct the models as a set of particles so that it does not have to reconstruct the mesh every time the topology changes. Particle methods are very useful to represent complex phenomenon such as bleeding. The only

390

M. Nakagawa et al. / A Bloodstream Simulation Based on Particle Method

thing we have to consider is that the calculation time by particle method depends on the number of particles. There are two major particle methods : SPH (Smoothed Particle Hydrodynamics) and MPS method, which has been developed to deal with incompressible fluid and also used to express breaking wave [6] and analyze red blood cells [7]. Therefore in this paper, we adapt MPS method as the particle method and perform the simulation of bloodstream and blood vessel deformation by using fluid dynamics and elastic dynamics theories.

1. Methods 1.1. Calculation Model The discretization using MPS method needs particle interaction calculation for which a weight function is defined with Eq.(1) under the condition that re is the effective radius within which particles interact each other, rij is the distance between particle i and particle j. Here, the gradient, divergence and Laplacian in the space region of a particle are defined [8].

w(rij ) =

re − 1 (rij ≤ re ), w(rij ) = 0 (rij > re ) rij

(1)

1.2. Blood Model ρ

1 ⎛ ∂vα ∂v β ⎞ Dv α ∂σ αβ α S αβ = ⎜ β + α ⎟ ...Eq.(1 − 3) = + ρ − K ... Eq .(1 1) 2 ⎝ ∂x ∂x ⎠ ∂x β Dt σ αβ = − pδ αβ + λ S γγ δ αβ + 2 μ S αβ ...Eq.(1 − 2)

ρ

Dv α ∂p ∂ 2v γ ∂ 2vα = − α + ( λ + μ ) α γ + μ β β + ρ K α ...Eq.(1 − 4) Dt ∂x ∂x ∂x ∂x ∂x ∂v γ Dρ + ρ γ = 0...Eq.(1 − 5) ∂x Dt Dv α ∂p ∂ 2v α ρ = − α + μ β β + ρ K α ...Eq.(1 − 6) v*, r * Dt ∂x ∂x ∂x k +1 * ⎧ ⎪ vi = vi + v′ ρ n − n0 ...Eq.(1 − 8) 2 ⎨ k +1 * ...Eq.(1 − 7) ∇ p=− 2 ⎪⎩ri = ri + Δtv ′ n0 Δt v′ Figure 1. Relationship among fluid model equations

Blood is generally non-Newtonian fluid; however, when the speed is fast, it can be approximated as Newtonian fluid so that in this paper, blood is treated as incompressible Newtonian fluid because the target vessel is the aorta, within which blood flows very fast. In addition, continuous equation and Cauchy’s equation of motion are used for governing equations of fluid. Cauchy’s equation is given with Eq.(1-1) in Figure 1. The superscripts (α, β and γ) are used for the index notation of generic terms. One of the superscripts is one component of x, y and z in Cartesian

391

M. Nakagawa et al. / A Bloodstream Simulation Based on Particle Method

coordinate system, where v is velocity, t is time, ρ is density, σ is stress tensor, K is force in unit mass, and D/Dt is Lagrangian difference. In Newtonian fluid, stress tensor is expressed with Eq.(1-2) in Figure 1, where p is pressure, δ is Kronecker delta, μ is viscosity, and S is strain rate tensor shown as Eq.(13) in Figure 1. By substituting the Eq.(1-2) into the Eq.(1-1), we obtain Eq.(1-4) in Figure 1. In addition, Eq.(1-5) in Figure 1 is satisfied by continuous equation under the condition of incompressibility. As a result, Eq.(1-4) becomes Eq.(1-6) in Figure 1. By discretizing Eq.(1-6) in Figure 1, we obtain Eq.(2).

ρi

Dv iα d = 0 Dt n

(

l = ∑ rij j ≠i

2

∑

p j − pi rij

j ≠i

)

2

rij w ( rij ) + μ

2d ln 0

∑ (vα − vα ) w ( r ) + ρ K α j

i

j ≠i

ij

i

(2)

w ( rij ) / ∑ w ( rij ) j ≠i

where d is the number of dimension, n0 is the sum of weights under incompressible state, and rij is the distance between particle i and particle j. Here, particle movement is pre-calculated without the pressure term and then modified after the pressure is calculated with the pre-calculated particle movement. v* and r * in Figure 1 are a temporary velocity and a temporary position which are pre-calculated without pressure component in Eq.(1-6) in Figure 1. By considering the pressure component which we excluded, the temporary velocity v* and the temporary position r * of the particle are modified. Since the density of fluid is proportional to the particle number density, we calculate Poisson’s equation of pressure with Eq.(1-7) and the modified velocity with Eq.(3).

2d ln0

∑( p j ≠i

j

( )

− pi ) w rij = −

Dv′ d ρ i =− 0 Dt n

∑ j ≠i

p j − pi rij

2

ρi n − n 0 Δt 2

n0 (3)

( )

rij w rij

The velocity v and the position r of particle i at time step k + 1 are expressed with Eq.(1-8) in Figure 1. With the above process, the governing equation of fluid can be solved. 1.3. Blood Vessel Model

ρ

Dv α ∂σ αβ = + ρ K α ...Eq.(2 − 1) Dt ∂x β

1 ⎛ ∂uα ∂u β ⎞ + ⎟ ...Eq.(2 − 3) 2 ⎝ ∂x β ∂xα ⎠ σ αβ = λε γγ δ αβ + 2με αβ ...Eq.(2 − 2) α Dv ∂ 2u γ ∂ 2u α ρ = ( λ + μ ) α γ + μ β β + ρ K α ...Eq.(2 − 4) Dt ∂x ∂x ∂x ∂x

ε αβ = ⎜

Figure 2. Relationship among elastic equations

392

M. Nakagawa et al. / A Bloodstream Simulation Based on Particle Method

In this paper, we assume that blood vessel is an isotopic elastic body. Eq.(2-2) in Figure 2 are used as the governing equations of elastic body under the condition that ε is strain tensor (Eq.(2-3)), u is displacement, and λ and μ are lame constants. We obtain Eq.(4) by substituting Eq.(2-2) into Eq.(2-1) in Figure 2 and discritizing Eq.(2-4) in Figure 2.

ni − n 0 n0 Dv iα d p j − pi 2d = 0∑ rij w ( rij ) + μ 0 ρi Dt n j ≠ i rij 2 ln pi = λ

∑ ( u αj − u iα ) w ( rij ) + ρ i K α

(4)

j ≠i

1.4. Interaction between Blood and Blood Vessel We can simulate the repulsion from blood vessel particle to blood particle by solving Poisson’s equation of pressure without distinguishing blood particles and blood vessel particles. In addition, by calculating particle number density without distinguishing blood vessel particles and blood particles, we can calculate volume strain which includes pressure by collision. 1.5. Simulation Models We have generated some models for the simulation. Surgical tools, blood vessel and blood models consist of rigid, elastic and fluid particles, respectively. The model space is divided into some cells which length is the same as the particle radius, and the surgical tool is built by placing one particle at the center of the cell which center is contained in the polygon composing of the surgical tool. By converting CT image data into polygons with Marching Cubes [9], the polygons of a blood vessel are extracted, and then the above method is applied to build the blood vessel model. By using this method, the model can be built with arbitrary particle size, and the blood particles are also inserted into the blood vessel.

2. Results and Conclusions We have used the aorta model as the blood vessel in the simulation. The blood vessel is deformed by pouring blood particles into the blood vessel model and making contact with stick surgical tools. The initial state is shown in Figure 3(a). Two types of deformation with bloodstream and without bloodstream were simulated and compared. In case without bloodstream (Figure 3(b)), the blood vessel was severely crushed compared to the deformation with bloodstream (Figure 3(c)). Bleeding simulation has also been performed by cutting a part of the blood vessel, and blood particles were breaking out of the blood vessel (Figure 3(d)). It took 20ms to deform the blood vessel and to simulate bleeding with the model that is constructed with 15,880 particles for the blood vessel and 6,688 particles for the blood on a PC consisting of 2.8GHz Intel Core2 Quad CPU and NVIDIA GeForce9800GTX+ Graphics Card.

M. Nakagawa et al. / A Bloodstream Simulation Based on Particle Method

393

Figure 3. Blood vessel deformation

A bloodstream simulation has been performed with MPS method. Surgical tools, blood vessel and blood models are composed of rigid, elastic and fluid particles, respectively. Blood vessel model was generated as the aorta from CT image data. By comparing two types of simulation, we have confirmed that bloodstream is important for the deformation of blood vessels. We have also performed bleeding simulation, which is difficult with mesh method, by using a particle method. In this simulation, however, the bloodstream in the blood vessel was treated as steady flow of Newtonian fluid. In the further research, we plan to treat blood as pulsatile and non-Newtonian fluid.

Acknowledgements This research was supported by Japan Society for the Promotion of Science (Research No.21500125).

References [1] [2] [3] [4] [5] [6] [7] [8] [9]

J. Mosegaard, P. Herborg and T. Sorensen, “A GPU Accelerated Spring Mass System for Surgical Simulation”, Studies in health technology and informatics, Vol.111, pp.342-348, 2005 M. Nakagawa, N. Mukai and M. Kosugi, “A Fast Blood Vessel Deformation Method Considering Inside Pressure”, ITE report, Vol.63, No.3, pp.371-375, 2009 N. Mukai, M. Nakagawa and M. Kosugi, “Real-time Blood Vessel Deformation with Bleeding Based on Particle Method”, MMVR16, pp.313-315, 2008 T. Kawamura, C. Xian, T. Hisada, K. Tsukahara and K. Tsuchimoto, “Investigations of Mechanical Characteristics of Pulsatile Artificial Heart by Fluid-Structure Finite Element Interaction Analysis”, The Japan Society of Mechanical Engineers, Vol.14, pp.273-274, 2002 W. Li, Z. Fan, X. Wei and A. Kaufman, “GPU Gem2”, pp.687-702, 2005 Q. Wang, Y. Zheng, C. Chun, T. Fujimoto and N. Chiba, “Efficient rendering of breaking waves using mps method”, Journal of Zhejjang University SCIENCE A, Vol.7, No.6, pp.1018-1025, 2006 K. Tsubota, S. Wada, H. Kamada, Y. Kitagawa, R. Lima and T. Yamaguchi, “A Particle Method for Blood Flow Simulation, -Application to Flowing Red Blood Cells and Platelets-“, Journal of the Earth Simulator, Vol.5, 2006 S. Koshizuka, “Particle Method”, Maruzen, 2005 W. E. Lorensen and H. E. Cline, “Marching Cubes: A High Resolution 3D Surface Construction Algorithm”, ACM SIGGRAPH Computer Graphics, Vol.21, No.4, 1987

394

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-394

Laser Induced Shockwaves on Flexible Polymers for Treatment of Bacterial Biofilms a

Artemio NAVARRO, bZachary D. TAYLOR, cDavid BEENHOUWER, c David A. HAAKE, aVijay GUPTA, bWarren S. GRUNDFEST a UCLA Department of Mchanical Engineering b UCLA Department of Bioengineering c Los Angeles Veterans Administration

Abstract. Bacterial biofilm-related infections are a burden on the healthcare industry. The effect of laser generated shockwaves through polycarbonate, a flexible polymer, is explored for its ability to generate high peak stresses, and also for its ability to conform to complex wound surfaces. Shockwave pulses in Al coated polycarbonate substrates and a resulting peak stress of greater than 60 MPa was measured which should provide sufficient pressure to kill bacteria. Keywords. Shockwaves, biofilms, wound, laser

1. Introduction Wound infections and infected traumatic wounds impose a major burden on the healthcare system. Treatment of infected wounds can prolong hospitalization and dramatically increase the cost of patient care. Recent studies estimate that 5% of all surgical wounds become infected, and 5%-7% of all traumatic wounds require open therapy for management of infections. Bacterial persistence in wounds is facilitated by the production of biofilms. Biofilms are bacterial communities containing a thick matrix of mucopolysaccharides produced by most bacterial species [1-2]. Current procedures to treat biofilm infected wounds are relatively ineffective and invasive. In this work we explore flexible polymers that can tolerate high peak stresses and with applications to biofilm disruption.

2. Tools and Methods Laser-generated pulses impinging upon a thin metallic surface generate stress waves within the material. The laser energy ablates the thin metallic film, thereby causing a rapid thermal expansion of the film resulting in a compressive wave propagating through the substrate. The laser fluence, pulse width, and the substrate material properties contribute to the temporal characteristics of the stress wave. In this work polycarbonate was used due to its high failure strengths, high stiffness coefficients, and its ability to be manufactured into thin, flexible sheets, thus allowing it to conform to

A. Navarro et al. / Laser Induced Shockwaves on Flexible Polymers

395

the curved surface of the body. A 1,064 nm Nd:YAG laser with a pulse duration of 3~6 ns is focused onto 380µm-thick polycarbonate substrate that is coated with a 0.5 µm aluminum thin-film, and a 50 to 100 μm thick layer of Na2SiO3 (waterglass) as shown in Figure 1. The waterglass is transparent to the 1,064 nm laser and acts as a confining layer, thereby lowering the temporal width of the stress profile.

Figure 1. Displacement interferometer setup.

A novel technique was developed to measure the 100-150 ns duration shockwaves through the use of a Michelson displacement interferometer[3-4]. The system incorporates a 1,064 nm Nd:YAG to generate the stress wave and a frequency stabilized 632.8 nm, 1mW HeNe laser used to measure the free surface velocity as the compressive wave travels and reflects as a tensile wave. The sample is coated with aluminum on both sides; one side is ablated with an ND:YAG laser and the other is used to reflect the HeNe laser. The 632.8 nm HeNe laser is directed through a 50/50 beam splitter to a reference mirror and the free surface of the coated substrate sample. The beams recombine and are focused onto a high speed photodetector coupled to a 5 GS/s digitizer as shown in Figure 1. As the wave propagates within the substrate, the free surface will move, causing the recombined signal HeNe signal to modulate between constructive and destructive interference and produces a down-chirped-like waveform. This can be used to reconstruct the input stress provided by the 1064 nm laser. Figure 2(a) depicts the waveform and a fit which can be used to determine the free surface velocity and ultimately the stress profile and Figure 2(b) displays the reconstructed pulse from the sampled data.

396

A. Navarro et al. / Laser Induced Shockwaves on Flexible Polymers

(a)

(b)

Figure 2. (a) Scan obtained from the Michelson displacement interferometer and the Al coated polycarbonate sample with fit. (b) Reconstructed shockwave profile.

3. Conclusion/Discussion A system to disrupt bacteria using laser generated shockwaves has been developed and characterized. The system produces high instantaneous peak stresses by ablating thin films of aluminum on polycarbonate. A peak stress of >60 MPa was measured using a displacement interferometer. Future experiments will test the efficacy of laser generated pulses on bacterial biofilms by coupling these pulsed shockwaves to biofilms grown on agar plates. Observation of cell death as a result of this technique will be assessed to determine the effectiveness of the technique and eventually optimize the shockwave parameters in order to maximize the bacterial disruption.

4. Acknowledgments The authors would like to thank Ms. Tiffany Chen, Ms. Neha Bajwa, Mr. Anthony Matolek, and Mr. Miguel Rocha for their vast knowledge of the cultivating and handling bacteria. The authors would also like to thank Dr. E. Carmack Holmes and Mrs. Cheryl Hein at CASIT for their support of this project.

References [1] [2] [3] [4]

Leid JG, Shirtliff ME, Costerton JW, Stoodley AP. Human leukocytes adhere to, penetrate, and respond to Staphylococcus aureus biofilms. Infect Immun 2002;70:6339–6345. Post JC, Hiller NL, Nistico L, et al. The role of biofilms in otolaryngologic infections: update 2007. Curr Opin Otolaryngol Head Neck Surg 2007;15:347–351. V. Gupta, A.S. Argon, J.A. Cornie and D.M. Parks, “Measurement of interface strength by laser pulse produced spallation,” Materials Science and Engineering, A126 (1990) 105-117. V. Gupta, A. S. Argon, D.M. Parks, and J.A. Cornie, “Measurement of interface strength by laser spallation experiment,” Journal of the Mechanics and Physics of Solids, 40, 1 (1992) 141-180.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-397

397

Virtual Reality Haptic Human Dissection Caroline NEEDHAM, Caroline WILKINSON and Roger SOAMES Centre for Anatomy and Human Identification, University of Dundee, Scotland.

Abstract. This project aims to create a three-dimensional digital model of the human hand and wrist which can be virtually ‘dissected’ through a haptic interface. Tissue properties will be added to the various anatomical structures to replicate a realistic look and feel. The project will explore the role of the medical artist and investigate the cross-discipline collaborations required in the field of virtual anatomy. The software will be used to train anatomy students in dissection skills before experience on a real cadaver. The effectiveness of the software will be evaluated and assessed both quantitatively as well as qualitatively. Keywords. Haptic, dissection, virtual reality, anatomy, teaching, cadaver, hand and wrist, medical artist

Introduction Three dimensional anatomical models are increasingly used in the teaching of anatomy and in some instances replacing cadaveric dissection all together [1]. This is an area of some debate with arguments both for and against the retention of cadaveric teaching. Benefits of cadaveric dissection are said to include; understanding of structures in three dimensions, witnessing anatomical variability, the feel of different tissues, learning practical dissection skills, team-work, and exposure to death. There are additional benefits to cadaveric dissection which cannot be replicated by using computer models. However, research has shown that digital models can play an important role in anatomy education when used alongside traditional teaching methods [2]. The use of haptic devices to re-introduce the sense of touch to computer based anatomical models could further enhance their effectiveness by making the experience more comparable to the real thing. While haptic technologies are increasingly being used in several areas of medical science (for surgical and clinical skills training, clinical practice including diagnostics and surgery planning and simulation [3]) there has been little research in the area of virtual reality haptic dissection for the teaching of human gross anatomy. This research is being undertaken as a part time PhD at the University of Dundee with an expected submission date of June 2014. The first author currently works as a lecturer in medical art at the University of Dundee.

1. Methods and Materials This research is a work in progress. As such, the achievements to date are discussed followed by future plans.

398

C. Needham et al. / Virtual Reality Haptic Human Dissection

1.1. Achievements • • • • • • •

Dissection of the hand and wrist of two cadavers, one formalin fixed and one Thiel embalmed [4]. Notes and photographs for reference in replicating appearance, tactile experience and process have been taken. Data collected to model the bones. Sourced from CT scan of sixteen year-old hand from the Scheuer Collection of Juvenile bones at the University of Dundee. Data collected to model the muscles and skin. Sourced from The Visible Human Project (VHP), National Library of Medicine (NLM). Cryosection images taken at one-third of-a-millimetre intervals were used. Both data sets were segmented using Amira 5.2.2 to produce 3D reconstructions of each anatomical structure as separate 3D image file (fig 1). The individual structures have been compiled and refined in FreeForm Modeling (version 10, 64 bit) to create the hand and wrist model of the musculoskeletal system (fig 2). Vessels and nerves are being created in FreeForm Modeling and added to the model (fig 3) A crude, prototype dissection of the model is already possible through the FreeForm Modeling interface (fig 4).

Figure 1 (left). Segmentation in Amira. Figure 2 (centre). Musculoskeletal system modeled in FreeForm Modeling. Figure 3 (right). Musculoskeletal system with the addition of vessels, nerves and skin.

1.2. Future Developments The current model first requires completion in FreeForm Modeling. Tissue properties will then be programmed for each anatomical structure. Currently [5] this is achievable through a range of programming options. However, if it were possible to integrate this into existing software (such as FreeForm Modeling) as a menu option/tool it would open up this function to a wider range of individuals from different disciplines. Although it is currently possible to carve into the model, this could be improved to allow more realistic cutting, possibly utilizing a dual haptic interface. Finally a piece of bespoke software will be created for use by and testing of a relevant student population.

C. Needham et al. / Virtual Reality Haptic Human Dissection

399

2. Discussion Although good progress has been made so far, the most complex aspects of the project lie ahead. Collaboration with computer scientists is essential to complete the project to its highest potential. However, the prototype dissection which is already possible (fig 4), demonstrates that even without collaboration it will be feasible to create a pilot VR dissector within the FreeForm interface (minimizing all menus not required by the student to avoid distraction). 2.2. The Role of the Medical Artist The research will also explore the role of the medical artist and investigate the crossdiscipline collaborations required in the field of virtual anatomy. Developments in technology have affected artists since the first pigments were mixed to make paint. Artists have, for the most part, been both familiar with operating technology as well as using it to convey their message. However, as technology continues to advance, this is not always the case. The programming of VR worlds and objects is one such area where a multi-discipline collaboration is often required. The use of technology by the artist highlights an important distinction which must be made between the tool maker and the tool user. In this instance, the tool maker would be the computer programmer and the tool user would be the artist who uses these virtual tools (i.e. software programs) to create their art form. It is postulated that as well as being two different roles they may also usually require two different mindsets and therefore, frequently, two or more individuals.

Figure 4. Prototype dissection of the model is already possible within the FreeForm interface

References [1] [2] [3] [4]

[5]

MCLACHLAN, J., BLIGH, J., BRADLEY, P. & SEARLE, J. (2004) Teaching anatomy without cadavers. Medical Education, 38, 418-424. BIASUTTO, S. N., IGNACIO CAUSSA, L. & ESTEBAN CRIADO DEL RÍO, L. (2006) Teaching anatomy: Cadavers vs. computers? Annals of Anatomy - Anatomischer Anzeiger, 188, 187-190. FAGER, P. J. & WOWERN, P. V. (2005) The use of haptics in medical applications. The International Journal of Medical Robotics and Computer Assisted Surgery, 1, 36 - 42. GROSCURTH, P., EGGLI, P., KAPFHAMMER, J., RAGER, G., HORNUNG, J. P. & FASEL, J. D. H. (2001) Gross anatomy in the surgical curriculum in Switzerland: Improved cadaver preservation, anatomical models, and course development. Anatomical Record, 265, 254-256. MEIER, U., LOPEZ, O., MONSERRAT, C., JUAN, M. C. & ALCANIZ, M. (2005) Real-time deformable models for surgery simulation: a survey. Computer Methods and Programs in Biomedicine, 77, 183-197.

400

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-400

The Tool Positioning Tutor: A Target-Pose Tracking and Display System for Learning Correct Placement of a Medical Device Douglas A. NELSONa,c,1 and Joseph T. SAMOSKYa,b,c Department of Bioengineering, University of Pittsburgh b Department of Anesthesiology, University of Pittsburgh c Simulation and Medical Technology R&D Center, University of Pittsburgh a

Abstract. Safe and successful performance of medical procedures often requires the correct manual positioning of a tool. For example, during endotracheal intubation a laryngoscope is used to open a passage in the airway through which a breathing tube is inserted. During training it can be challenging for an experienced practitioner to effectively communicate to a novice the correct placement and orientation of a tool. We have implemented a real-time tracking and position display system to enhance learning correct laryngoscope placement. The system displays a 3D model of the laryngoscope. A clinical teacher can correctly position the laryngoscope to open the airway of a full-body simulator, then set this tool pose as the target position. The system displays to the learner the fixed, target pose and a real-time display of the current, “live” laryngoscope position. Positional error metrics are displayed as color-coded visual cues to guide the user toward successful targeting of the reference position. This technique provides quantitative assessment of the degree to which a learner has matched a specified “expert” position with a tool, and is potentially applicable to a wide variety of tools and procedures. Keywords. Human computer interaction, 3D visual guidance, real-time spatial tracking, simulation-based training, endotracheal intubation, laryngoscopy

Introduction In both clinical and simulation-based learning environments, communication of psychomotor skills from expert to learner for procedural training is typically done by example and by verbal guidance, but this may leave the learner lost when trying to duplicate the manipulations of the expert, and verbal guidance can be a less than optimal medium to specify a three-dimensional psychomotor outcome or how to achieve it efficiently. We developed the Tool Position Tutor to enhance the communication of optimal device positioning. The system provides 3D tracking of the pose of a laryngoscope during manipulation by expert and learner. A display of a virtual laryngoscope shows the desired position, and visual cues guide the learner to match the target pose. The novel foci of this work compared to previous research in tracking of laryngoscope pose 1 Corresponding author: Douglas A. Nelson, Simulation and Medical Technology R&D Center, University of Pittsburgh, 230 McKee Place, Suite 401, Pittsburgh, PA 15213; E-mail: [email protected].

D.A. Nelson and J.T. Samosky / The Tool Positioning Tutor

401

[1] are the real-time 3D visual models, associated target-pose interaction strategies, and superimposed visual cues designed to guide the learner toward expert performance.

Methods and Materials We employed a 3D electromagnetic tracking system (Aurora, Northern Digital Inc.) equipped with a 6 DOF sensor (Traxtal Dynamic Reference Body) to measure the position and orientation of a standard laryngoscope. A custom adapter, fabricated via stereolithography, enabled a press-fit attachment of the sensor to the end of the laryngoscope handle. The Aurora field generator was positioned under the head and neck of a full-body patient simulator commonly used for airway management training (SimMan®, Laerdal Medical AS). A 3D CAD model of the laryngoscope was created in SolidWorks. Software was developed that displayed the virtual laryngoscope tracking the pose of the real laryngoscope. “Pose Set” and “Pose Clear” functions can be specified via footswitches. The instructor positions the laryngoscope in the simulator’s airway, sets the pose, and the system captures the reference position, displayed as a static gray target. A second, dynamically tracked “learner” laryngoscope model is then displayed, which the learner endeavors to match to the target. The distance between the target and learner scope is measured at three locations (blade tip, hinge and handle top) and color-coded error bars corresponding to each distance error metric are displayed. The root-square of the three distance error measures is used as a criterion of closeness to target pose. The learner scope is dynamically color-coded to indicate closeness to the target. In addition, dynamic vectors (termed “bungee error cords”) that connect the three corresponding points on the learner and target scopes are displayed to help guide the learner toward matching the target.

Results We verified tracking within a workspace extending +/- 20 cm to the left and right of the mouth of the simulator, +/- 20 cm inferior and superior, and +20 cm anterior: the position of the laryngoscope during repeated intubation of the simulator’s airway was well within these limits. Figure 1a illustrates the set up and display interface of the system. The three screen shots of Figure 1(b,c,d) depict a learner moving the laryngoscope closer to the “expert” target position: the scope changes color from red to yellow to green (shown as dark to light in grayscale renditions of the Figure), and the associated visual error metrics decrease. At a user-specified error tolerance both a visual indicator and an auditory cue indicate a successful match with the target.

Discussion We are currently designing a trial to test the effectiveness of this system in aiding instruction in laryngoscope positioning. We plan to acquire expert and novice data [1][2], and to perform cluster analysis on the data to differentiate levels of proficiency. The clustered metrics could also form a database of reference positions that may help

402

D.A. Nelson and J.T. Samosky / The Tool Positioning Tutor

Figure 1. (a) Laryngoscope with attached tracking sensor. A 3D model of the laryngoscope is displayed tracking the pose of the real scope. (b-d) After the target position is fixed by an expert, the learner scope changes color (hue) to indicate proximity (or dark to light in grayscale). Dynamic visual cues indicate the distance error to three corresponding points on the target and learner scopes. When the learner and target scope are aligned within a preset threshold (d) the “goal light” illuminates with concomitant audio feedback.

support automated instruction and self-learning without the need for an expert instructor to be present. We are exploring a variety of visual cues that may help guide the user to successful matching of target positions. For example, preliminary tests offer promise that continuous-gradient, as opposed to discrete, color shading of the learner scope as a function of distance to the target scope may offer benefits in guiding the learner toward the target. We also envision the display incorporating 3D anatomic images in addition to the tracked scope to allow the learner to develop a mental model of the interactions between scope and internal anatomy that may not be visible to the learner during the procedure. We are also exploring applications of similar techniques to tool positioning in other procedures, including regional anesthesia and diagnostic ultrasound.

References [1] [2]

Delson NJ, Koussa N, Hastings RH, Weinger MB. Quantifying Expert vs Novice Skill In Vivo for Development of a Laryngoscopy Simulator. Stud Health Technol Inform, 2003. 94: p. 45-51. Stylopoulos N, Cotin S, Maithel SK, Ottensmeyer M, Jackson PG, Bardsley RS, et al. Computerenhanced laparoscopic training system (CELTS): bridging the gap. Surg Endosc, 2004. 18(5): p. 782-9.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-403

403

A Cost Effective Simulator for Education of Ultrasound Image Interpretation and Probe Manipulation S.A. NICOLAUa,1 A. VEMURIa, H.S. WUa, M.H. HUANGa, Y. HOa, A. CHARNOZ b, A. HOSTETTLER b, C. FOREST b, L. SOLER b and J. MARESCAUX b a IRCAD Taiwan, Medical Imaging Team, 1-6 Lugong Road, Lukang 505 TAIWAN b IRCAD Strasbourg, 1 place de l’hopital 67091 Strasbourg FRANCE

Abstract. Ultrasonography is the lowest cost no risk medical imaging technique. However, reading an ultrasonographic (US) image as well as performing a good US probe positioning remain difficult tasks. Education in this domain is today performed on patients, thus limiting it to the most common cases. In this paper, we present a cost effective simulator that allows US image practice and realistic probe manipulation from CT data. More precisely, we tackle the issue of providing a realistic interface for the probe manipulation with a basic haptic feedback. Keywords. Ultrasound image simulation, training simulator, optical tracking

Introduction Education of young practitioners in most medical specialties, as a first step, is approached using a phantom that simulates the human body. However, most of them are very expensive and provide limited realistic experience to young practitioners. Because of their ease of use and better educative value to young practitioners, software based simulators in medical field have gained more importance in recent years. Indeed, they can reduce the cost and allow education on different kinds of pathology. Our final goal is to provide a US simulator for education that would allow for practice of US abdominal image interpretation and probe manipulation on patient database. The simulator we propose would offer the following advantages. Firstly, the student could work wherever and whenever he wants without needing to go to the hospital. Secondly, the time spent by a medical expert would be reduced, thus decreasing the education cost. Finally, students could practice on rare pathologies. To provide an efficient simulator for US image interpretation and probe manipulation, there are three major constraints to fulfil: realistic US image simulation, realistic probe interaction and a minimal haptic feedback. Vidal, Forest, Ni, Blum and Magee propose a US simulator for practicing needle puncture on patient dependent data [12, 5, 10, 2, 9]. The relative localization of the fake US probe is realized either using Omni © Sensable haptic feedback system, an EM tracking system (Ascension ©) or an optical tracking system (ARTrack ©) which makes the system price prohibitive. Cynydd is the only one to propose a cost effective 1

Corresponding Author: [email protected]

404

S.A. Nicolau et al. / A Cost Effective Simulator for Education of Ultrasound Image Interpretation

solution to education of US image interpretation [3]. The probe interaction is performed using a wiimote Nintendo © (~30 Euros). However, the system provides no haptic feedback and the probe rotation interaction is limited since the wiimote must be oriented toward the infrared emitter below the visualization screen. In this paper, we present the development of a low cost simulator for US education devoted to abdomen and that can be used on a standard workstation with a webcam. The principle is to load patient CT data (the skin is automatically segmented), and to move interactively a virtual US probe in the virtual scene on the skin reconstruction. The interaction is realized thanks to a 3D optical tracking of a fake probe that the user holds and moves on the surface of a phantom ensuring a haptic feedback (a cardboard box can also be used). From the position of the virtual US probe on the skin model, we compute the CT slice viewed by the probe. The final corresponding US image is provided thanks to an algorithm that transforms the CT image to US modality (already published in a patent [7]). Although the software part of the project is almost finished, this work is still in progress since we still need to demonstrate the clinical benefit of the system for student education. 1 . Methods & Materials The system is composed of a PC (workstation or laptop) equipped with one web camera. The phantom can be either a basic foam phantom or a simple cardboard box on which several optical markers are stuck (cf. Fig 1). To track the probe and the phantom, we use the ARToolkit+ library [1] to extract the 4 corners pixel coordinates of each individual marker and to identify them. Then, we use openCV library to refine the corner extraction [8]. Each set of 4 corners are then used to compute the marker pose in the camera frame. We print 5 markers that we stick on each face of a plastic cube, which is attached to the fake US probe (cf. Fig 1). Patient data is a CT volume (thickness<5 mm) that comprises the organ of interest. No organ segmentation is necessary, only the skin position has to be identified (using a simple threshold).

Figure 1. Left: illustration of the equipment to track the fake US probe and the cardboard box. Right: the foam phantom with the definition of the two lines we use to register it in the CT reference frame.

In this section, we firstly detail how we accurately track the US probe using a cube on which 5 markers are stuck. Since this tracking method needs the a priori knowledge of relative positions of each marker on the cube, we then explain how we estimate it with an easy calibration step. Finally, we describe the method that allows tracking the phantom (or cardboard box).

S.A. Nicolau et al. / A Cost Effective Simulator for Education of Ultrasound Image Interpretation

405

1.1. US Probe Tracking We firstly calibrate the web camera with the method described in [13], which allows a quick calibration with very few understandings by the end-user. He only has to print a chessboard of markers (for instance 6x5 markers with 3 cm squares) and stick them on a planar surface [4]. The tracking of the fake US probe could be realized in practice using only one marker stuck on it (cf. [1]). However, this approach would limit the possible user interaction since it would prevent the user to make 180° rotation around the probe axis. To overcome this issue, we propose to use a cube with 5 optical markers on its faces. In practice, to know the relative position of all marker corners, we need either to build it, which is expensive, or ask user to calibrate it. 1.2. Calibration of a Cube with 5 Markers on Its Faces The idea is to move/rotate the cube in front of the calibrated camera so that all possible face pairs are visualized by the camera. Let’s say we want to compute all point coordinates in the frame linked to the top face F0 (cf. left Fig. 2). Using ISPPC [11], each time that F0 is visible together with the ith face Fi, we can compute both Tcam−F0 and Tcam−Fi. Therefore, for each frame containing F0 and Fi we can obtain one estimation of TF0−Fi = T−1 cam−F0 x Tcam−Fi. Since we will in practice have several video frame showing at the same time F0 and Fi, we will get several estimations of TF0−Fi that we average to obtain a more accurate estimation (see [6] for more details on averaging rotation). Finally, if we move the cube so that F1, F2, F3 and F4 are one after one visible with F0, we can estimate all point coordinates in the frame of the top face F0.

Figure 2. Left: illustration of the frames on the cube. Right: definition of the distance D and d for software.

After the cube calibration, the user has to stick it on his US probe. In order to avoid supplementary calibration step, the user should roughly put the cube on the probe so that the marker centre of F0 belongs to the symmetry plane of the probe (cf. Fig. 2). Then, the user has to measure the length D and d and provide it to the system. 1.3. Phantom Tracking In case only a cardboard box is available, we propose to stick on one side several optical markers printed on the same page (cf. Fig. 1). The idea is that the marker corner coordinates will be chosen so that they correspond to one side of the CT data volume like depicted in Fig. 3 (left). When the user is moving the probe on the box top side, the probe position is computed and we find the vertex at the same altitude Z which is the closest to the probe (cf. right Fig. 3). Then, we display the virtual probe on this skin vertex in the virtual scene (we keep the same orientation). This allows giving the user the feeling that the probe moves on the skin whilst he moves the probe on the box.

406

S.A. Nicolau et al. / A Cost Effective Simulator for Education of Ultrasound Image Interpretation

Figure 3. Left: the optical markers are stuck on one side of the box. Right: to compute the position of the virtual probe on the skin model, we find the vertex on the skin model which is the closest to the line between the slice center and the probe extreme point. On this example, the virtual probe is displayed on the red vertex.

In case the user has a foam phantom, the user has firstly to stick one marker on the phantom side which is in front of the web camera. Then, we ask the user to move the fake probe on the phantom and to record two specific lines: the cranio-caudal line on the highest phantom crest and the line in an axial plane roughly in the middle of the phantom (cf. Fig. 1). This information allows us to register approximately the phantom in the CT frame: we firstly match the intersection of the two lines with the top face centre of the CT data, the orientation is given by the middle axial line which defines a plane and that we match with the middle axial plane of the CT image. 2 . Results On Fig. 4, one can see the final user interface. On the right and centre windows, the user can see the patient skin model (segmented organs can be visualized, but the system does not depend on it) with the virtual probe. Two different points of view can be provided: the right window shows the original CT image corresponding to the probe position and the left one displays the US image simulated in real time from CT image.

Figure 4. Illustration of the software interface.

To evaluate whether a practitioner could use the simulator by himself, we have written a user’s guide that contains guidelines for each calibration steps: webcam, cube marker, cardboard and phantom. Then, we asked 10 residents to read it and perform each calibration. We provided them a cube, a fake probe, a cardboard and a phantom. For each calibration, we have recorded the reading and manipulation duration. They also filled a questionnaire to give a score between 0 and 5 for each step (5 = very easy,

S.A. Nicolau et al. / A Cost Effective Simulator for Education of Ultrasound Image Interpretation

407

0 = I cannot do it). We also asked them whether they consider the system realistic enough (US image and probe interaction: 5 = realistic, 3 = enough realism, 0 = not realistic). Results in Tab. 1 show that the simulator is easy to calibrate and that the realism of simulated US image has to be improved whereas the interaction is good. Table 1. Evaluation results of our simulator by ten novice users. Calibration step Average score (0 to 5) Average time (sec.)

webcam 4.6 304

cube marker 4.1 332

box 4.4 155

phantom 4.2 225

US image 2.8 --

interaction 4.4 --

3 . Conclusions We have presented a simulator for education of young practitioner to US image interpretation and probe manipulation. Our objective is to provide a cost effective simulator that allows basic haptic feedback and which can be easily used with a standard workstation. In this paper, we have showed how we reach our aim using a simple webcam and several planar optical markers printed on paper sheets. More precisely, we have firstly explained how it is possible to robustly track a fake probe using a web camera. Secondly, we have described simple methods to use a phantom or a box to mimic the patient position. Finally, an evaluation with ten novice users showed that the simulator is easy to calibrate and that the system realism is encouraging. However, some efforts still have to be spent on US image simulation. Our next step will be to improve the US image simulation following expert remarks and evaluate the simulator benefit for education of young residents.

References [1] Augmented Reality Toolkit: ARToolkit+, HITL: http://www.hitl.washington.edu/artoolkit/. [2] T. Blum, S.M. Heining, O. Kutter, and N. Navab. Advanced training methods using an augmented reality ultrasound simulator. In In Proceedings of ISMAR 09, pages 177–178, 2009. [3] L. Cynydd et. al.. Cost effective ultrasound imaging training mentor for use in developing countries. In Studies in Health Technology and Informatics 2009. IOS Press. Volume 142, pages 49–54. [4] M. Fiala and C. Shu. Self-identifying patterns for plane-based camera calibration. Machine Vision Applications, 19(4):209–216, 2008. [5] C. Forest et. al. Ultrasound and needle insertion simulators built on real patient-based data. In Studies in Health Tech. and Inform. 2007. IOS Press. Volume 125. pages 136–139. [6] C. Gramkow. On averaging rotations. Int. Journal of Computer Vision, 42(1/2):7–16, April 2001. [7] A. Hostettler, L. Soler, C. Forest, and J. Marescaux. Patent application title: Process and system for simulation or digital synthesis of sonographic images: http://www.faqs.org/patents/app/20090046912. [8] Intel OpenCV library: http://opencv.willowgarage.com/wiki/. [9] D. Magee et. al. An augmented reality simulator for ultrasound guided needle placement training. Medical and Biological Engineering and Computing, 45(10):957967, 2007. [10] D. Ni et. al. An ultrasound-guided organ biopsy simulation with 6dof haptic feedback. In In Proc. of Medical Image Computing and Computer-Assisted Intervention, page 551-559, 2008. [11] S. Nicolau et. al. An accuracy certified augmented reality system for therapy guidance. In European Conference on Computer Vision (ECCV’04), LNCS 3023, pages 79–91. Springer-Verlag, 2004. [12] F.P. Vidal et. al. Simulation of ultrasound guided needle puncture using patient specific data with 3d textures and volume haptics. Computer Animation and Virtual Worlds, 19(2):111–127, May 2008. [13] Z. Zhang. Flexible camera calibration by viewing a plane from unknown orientations. In Proceedings of Int.Conference on Computer Vision (ICCV 99), pages 666–673, 1999.

408

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-408

A Portable Palpation Training Platform with Virtual Human Patient Tyler NILES a,1 , D. Scott LIND b,2 , and Kyle JOHNSEN a,3 , a Virtual Experiences Lab, University of Georgia, Athens, GA b Surgical Oncology, Medical College of Georgia, Augusta, GA Abstract. Palpation (the application of touch to the surface of the body) is an essential clinical skill. Correct palpation is part of a complete physical examination and it assists a clinician in making an accurate diagnosis, while poor palpatory skills can lead to diagnostic errors. As with any clinical skill, palpation is best learned through repetitive practice with constructive feedback. Unfortunately, changes in healthcare provide fewer opportunities for hands-on learning of this essential skill. Unlike other clinical skills, palpation has no immediate feedback to the learner regarding their performance. For example, when students are learning how to insert an intravenous catheter, failure to perform the technique correctly results in no blood return in the catheter. However, students do not know if they are palpating an abnormality if they have never felt it before. This inherent difﬁculty makes expert feedback even more vital to learning correct palpation. Existing research tools have addressed some of these challenges through simulation techniques that do not require experts, and can provide feedback on palpation pressures and palpation patterns. We describe a novel computer-based palpation training system, leveraging existing approaches, with an emphasis on sensing accuracy, directed-feedback, portability, and user experience. Keywords. Clinical Breast Exam, Augmented Reality, Virtual Patient, Force R Sensing, Palpation, Mammacare

Introduction The acquisition of clinical skills, including palpation, is an on-going challenge in medical education. Learning clinical skills using the traditional apprentice-based methods is increasingly difﬁcult in the current healthcare environment. Simulation is an attractive addition to standard methods of teaching clinical skills. Computer-based simulation permits standardization, lowered costs, increased patient diversity, and improved feedback to provide a minimum level of student competence. We present a design for a portable, computer-based simulator that enables students to practice and receive feedback when performing a clinical breast examination (CBE). The simulator incorporates a virtual human for medical interview training and a force sensing platform for palpation training. The primary beneﬁts of this innovative design are palpation sensing accuracy, portability, visual feedback, compatibility, and provider-patient conversation. 1 [email protected] 2 [email protected] 3 [email protected]

T. Niles et al. / A Portable Palpation Training Platform with Virtual Human Patient

409

In the new millennium, breast cancer is the most common type of cancer diagnosed in women and the second most common cause of death from cancer in women [1]. Early detection of breast cancers is the most important method to improve survival rates. Early breast cancer screening research suggested that ﬁnding small masses by physical examination was largely serendipitous [2], motivating imaging techniques such as mammography. However, a more recent study involving nearly 300,000 women found that the combination of a mammography and CBE can improve early detection of breast cancer— provided correct execution of the CBE [3]. Furthermore, a CBE has a lower direct cost than mammography, and CBE provides opportunities for provider-patient communication that patients may not receive during mammography screening. Unfortunately, many providers feel that they are not properly trained in CBE techniques, limiting the use of proper CBE techniques [4,5]. To assist with CBE training, a number of inert mannequin simulators are available, R These mannequin simulators such as the silicone breast models from Mammacare. offer the opportunity for students to practice breast examination technique on standardized models. In addition, the breast models can be embedded with materials or inﬂatable devices to simulate breast abnormalities [6]. To improve upon the feedback, some have incorporated force sensors into the mannequins, alongside visualizations of the force being used [7]. Finally, an immersive simulator has been created that combines a fullsize mannequin and a virtual patient (a mixed reality human) to practice both CBE and medical history-taking simultaneously [8]. The goals of the simulator described in this work are similar to the aforementioned tools: to improve and standardize the manner in which breast examinations are conducted. Our simulator builds upon and leverages these existing works, ﬁlling a gap between sensor-augmented mannequins and mixed reality humans, and providing unique opportunities for training.

1. Methods & Materials Based on feedback from medical educators, several speciﬁc design criteria were identiﬁed. 1.1. Portability One of the primary motivations for this work was an emphasis on simulator portability. Portability in a simulator can provide lower cost, increased robustness due to lower system component count, and permits more units to be produced and used for a given amount of resources (e.g. money, physical space). A potential limitation of simulator portability is a reduction in realism in comparison to larger-scale simulators that may offer a more immersive experience. However, portable simulators are useful in large-scale training sessions running in parallel, and for community-based education and research. In addition, a portable simulator does not require a dedicated room or support staff. 1.2. Accuracy While the currently available breast simulators offer precise palpation measurements, they do not accurately measure in units (e.g. pound or kilogram). Precision is how small of a change can be detected by a sensor, whereas accuracy is how close a sensor reading

410

T. Niles et al. / A Portable Palpation Training Platform with Virtual Human Patient

is to the true value. For example, a sensor may be able to detect a difference between the palpation force used by two students, but may not be able to determine exactly how many pounds of pressure were applied by each student or quantify the magnitude of the difference. Accurate palpation sensing would be valuable in establishing a CBE competency model that can be applied across simulator platforms. Therefore a future goal of this work involves ascertaining how accuracy might contribute to a more focused and effective training method or concept. 1.3. Visual Feedback Repetitive practice with immediate feedback is more effective than passive lecture alone in learning any clinical skill, including CBE [9]. Recently published research demonstrates that providing visual feedback about palpation, such as the position and pressure of palpation (touch map) and pattern of a sequence of palpations (pattern map), improved student acquisition of CBE skills [8]. Such visualizations can be presented as an overlay (augmented reality) over a live video feed of the breast model, or presented to the student or instructor at the end of the interaction. In addition, the visualizations provide an opportunity for the student to see improvements in their performance by comparing the visualizations of palpation pressure, pattern, and position between multiple interactions over the course of a training regime. Student progress in a training regime with this visual feedback can also be saved, printed, and given to the student at the conclusion of each exam as a record of their personal progress. 1.4. Compatability An important design concept was to enable a wide variety of breast models to be used with the simulator without modifying the models. This effort leverages the existing body of available models, and does not modify the existing training properties of the models. R models could be used with the Mammacare R method of For example, Mammacare training, while the simulator provides additional visual feedback, accuracy, evaluation, and history-taking training. This concept of compatibility could also extend to palpation of other body parts that are included in physical examinations, such as palpation of the abdomen as well as pelvic and rectal examinations. 1.5. Provider-Patient Conversation History-taking is an essential part of any patient-provider interaction. A virtual human patient presents an important sociocultural dimension, particularly in provider-patient conversation. Every patient interaction presents an opportunity to discuss issues relevant to the patient’s current complaint, to establish trust between the provider and patient, and to identify other issues that may not be clear at the start of the interaction. Thus, open dialogue (not scripted or multiple choice) conversation with a virtual human patient is an important goal in simulator design. 1.6. Customization of the Training Experience An important facet of medical education is the ability to adjust to the unique learning needs of each student. One solution to this need is customization of the available guid-

T. Niles et al. / A Portable Palpation Training Platform with Virtual Human Patient

411

Figure 1. Block Diagram of the Simulation tool. The host PC generates the AR display and virtual human.

Figure 2. Photo of System-User Interaction.

ance provided by the simulator. For example, if students are being evaluated, the educator may not want to provide immediate feedback on palpation pressure during the examination. Thus, the real-time (i.e. during the experience) visualizations and other assistance provided by the simulator could be enabled or disabled by the trainer on a per-scenario basis. In either case, the visual feedback data is still available for both the student and educator at the conclusion of the training session.

2. Results 2.1. System Design Based on the design requirements, we constructed the Portable Breast Simulator (PBS). The PBS is a training simulator consisting of a palpation sensor platform, a breast model placed on the platform, and a computer interface for visualization and patient interaction (see Figures 1, 2). The palpation sensor is a novel design. Palpation consists of a set of force (pressure) positions on a particular area. Force is detected by measuring the total weight placed on the simulator platform, and subtracting the weight of the breast model. This approach is in contrast to other approaches that place small force sensors at distinct locations on the breast model. For the sensing platform, a Saltner-Brecknell OEM postage scale provided the base packaging and 11-lb (5kg) load cell employed for the force sensing platform. A

412

T. Niles et al. / A Portable Palpation Training Platform with Virtual Human Patient

Figure 3. Sample of the augmented reality display during a practice examination.

Figure 4. Print-out samples of post-exam: pattern and force map, pattern map, and force map (left-to-right).

custom circuit was then designed using a microcontroller to deliver force readings at a rate of 250Hz to the host computer over USB. The advantage to this approach is that it does not require augmenting a breast model with force sensors, and it provides a very accurate measurement of the force used. The disadvantage of this approach is that the position of the palpation must still be tracked. For position tracking, a boom-mounted camera (Playstation Eye Camera, 640x480, 30fps, 76-degree FOV) is used with a colortracking computer-vision algorithm written using the Open Computer Vision library. Currently, this requires that the user wear an unobtrusive piece of colored tape on their palpation ﬁnger (typically the middle ﬁnger), although other algorithms for ﬁnger tracking are being explored. The camera used for position tracking also allows for augmented reality overlay (using the Ogre3D rendering engine) of force, position, and pattern visualizations. Two forms of visual feedback are provided. The ﬁrst is the pattern map, which displays a connected line indicating the path taken by the student’s ﬁnger during the exam. The second is a force map, which provides a multicolor gradient visualization indicating the level of pressure applied. While visible, they are semitransparent so as to not occlude the student’s ﬁngers during the exam (see Figure 3). Print-outs and screenshots saved at the conclusion of the exam, and use solid color (no transparency) for quick and easy identiﬁcation and review (see Figure 4). As a customization feature, these visualizations can be visible or hidden by the student, and enabled or disabled by the instructor. A virtual human (see Figure 3) provides the embodiment of the patient. Students can interact with the patient (greeting, determine chief complaint, take history, etc.) through a typed interface. No cues are provided to the student as to what questions to ask, although a history of the interaction is displayed if the instructor allows. The artiﬁcial intelligence for the character is provided by Virtual People Factory [10]. Virtual People Factory is an online system for creating intelligent agents that can be used by other programs (e.g.

T. Niles et al. / A Portable Palpation Training Platform with Virtual Human Patient

413

Second Life). Through VPF, instructors write a script for the virtual human patient (what the virtual human says and does in response to user questions). 2.2. Testing and Calibration To effectively use the PBS, the force to visual feedback color mapping must be calibrated as desired (e.g. if the the force exerted during a palpation is greater than 4-lbs, display a red circle, if it is less than 4-lbs, display a green circle). The mapping used will also depend on the size of the particular breast model used. The model for our testing R was a prosthesis B-cup sized model cone, although other models such as Mammacare models could have been used. To determine the color ranges, we recorded the palpation force used by a medical educators at MCG when asked to palpate at light, medium, and heavy pressure. Once the calibration has been performed, the color of the visual feedback indicates a particular pressure being applied. Expressed in a multicolor gradient visualization, these colors correspond to ranges of 1.0–3.99, 4.0–10.9, and 11+ pounds-force, respectively (determined using a B-cup sized cone). If force is present but too light, the color will be white. If the force is beyond the heavy category, essentially causing undue pain to the patient, the color will be dark red. In preliminary testing, the force sensor was found to have a sensitivity of 0.0108 pounds, and is accurate to within 0.37% on average (maximum of 0.51% error). The position tracking is accurate to within sub-millimeter range of the colored tape location (average of 0.65mm, maximum of 1.8mm). Virtual People Factory has a listed accuracy of 75% in generating appropriate responses to user input, although the accuracy rate is dependent upon a number of factors including script authoring, and grammatical differences between users. Our testing script appears to achieve near this 75% accuracy, although further tests are needed.

3. Discussion The PBS ﬁlls a signiﬁcant gap in current palpation tools for learning CBE. Current tools force educators to make the choice between affordable, highly conﬁgurable models or expensive, sensor-augmented models. The PBS offers the advantages of palpation sensing and visualization for use with affordable models while offering the added beneﬁt of virtual patient interaction. 3.1. Limitations As currently designed, the PBS has the capability to detect only a single palpation at any given time. This means that it cannot accurately detect two palpations simultaneously (e.g. two hands). To detect simultaneous two-handed palpation, a grid of sensors could be used for force measurement instead of a single load-cell. In addition, the current loadcell has a range of approximately 0–11 lbs, including the model. This limits the types of examinations that can be accurately measured. For example, a very large breast model may require more than 11-lbs of force to adequately examine. To measure larger forces with the existing system, a wider range load-cell could be incorporated, at the expense of decreased sensitivity. Finally, another issue is that the virtual human patient is not life-size. Research suggests that the behavior of a medical student with a virtual human

414

T. Niles et al. / A Portable Palpation Training Platform with Virtual Human Patient

patient is more realistic if the virtual human patient is life-size [11]. Using a large-screen monitor to display the visualization may help mitigate this issue, although at higher cost. 3.2. Future Work The PBS has undergone an iterative design process for usability and educational features. A major focus for additional work is to construct and validate a scoring metric for breast examinations—that is, a competency model which could be used to automatically determine the skill level of a student. A validated scoring metric would provide increased immediate feedback to students during and after an interaction without requiring an expert’s valuable time to review performance. Furthermore, student scores could be automatically tracked over time and provided to an instructor for student and class assessment. We also intend to compare the usability and effectiveness of the PBS as a practice and feedback tool to other existing simulators. This is necessary to determine if the PBS adds value to existing training approaches. Ultimately, the direction of this work is to provide a highly effective, validated approach to training palpation that reduces training costs and errors associated with palpation-based clinical examinations. References [1]

[2] [3]

[4]

[5] [6]

[7]

[8]

[9]

[10]

[11]

CDC, U.S. Cancer Statistics Working Group. 2009. United States Cancer Statistics: 1999–2005 Incidence and Mortality Web-based Report. Atlanta: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention and National Cancer Institute. www.cdc.gov/uscs. Martin, John, Myron Moskowitz, John Milbrath. 1979. Breast Cancer Missed by Mammography. American Roentgen Ray Society. 132, pp. 737– 739. Chiarelli, A.M., Vicky Majpruz, Patrick Brown, Marc Theriault, Rene Shumak, Verna Mai. 2009. The Contribution of Clinical Breast Examination to the Accuracy of Breast Screening. J. National Cancer Institution. 101 (18), pp. 1236–1243. Iannotti, Ronald J., Lila J. Finney, Alice Anne Sander, Jessica M. De Leon. 2002. Effect of clinical breast examination training on practitioner’s perceived competence. Cancer Detection and Prevention. 26 (2), Pages 146–148. Wiecha, J. M., and Gann, P. 1993. Provider conﬁdence in breast examination. Family Practice Research Journal. 13 (1), pp. 37–41. Gerling, Gregory, Geb Thomas, Alicia Weissman, Edwin Dove. 2002. Dynamic Simulator for Training Clinical Breast Examination. Human Factors and Ergonomics Society Annual Meeting Proceedings, Medical Systems and Rehabilitation. pp. 1472–1476. Pugh, C.M., L.H. Salud, ZB Domont, and KM Blossﬁeld. 2008. A simulation-based assessment of clinical breast examination technique: do patient and clinician factors affect clinician approach? American Journal of Surgery. 195 (6), pp. 874–880. Kotranza, Aaron, D. Scott Lind, Carla M. Pugh, Benjamin Lok. 2009. Real-Time In-Situ Visual Feedback of Task Performance in Mixed Environments for Learning Joint Psychomotor-Cognitive Tasks. IEEE International Symposium on Mized and Augmented Reality 2009. October 19–22, Orlando, FL, pp. 125–134. Pilgrim, C., Lannon, C., Harris, R. P., Cogburn, W., and Fletcher, S. W. 1993. Improving clinical breast examination training in a medical school: a randomized controlled trial. Journal of General Internal Medicine. 8 (12), pp. 685–688. Rossen, B., David Scott Lind, Benjamin Lok. 2009. Human-centered Distributed Conversational Modeling: Efﬁcient Modeling of Robust Virtual Human Conversations, 9th International Conference on Intelligent Virtual Agents (IVA 2009). Amsterdam, Netherlands, Sept. 14–16. Kyle Johnsen, D. Scott Lind, Benjamin Lok. 2010. The Impact of a Mixed Reality Display Conﬁguration on User Behavior with a Virtual Human. Proceedings of Intelligent Virtual Agents 2010 (publishing pending).

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-415

415

A Development of Surgical Simulator for Training of Operative Skills using Patient-Speciﬁc Data Masato OGATA a,b,2 , Manabu NAGASAKA a,b , Toru INUIYA a,b , Kazuhide MAKIYAMA b , Yoshinobu KUBOTA b a Research and Development Division, Mitsubishi Precision Co., Ltd., Japan b Department of Urology, Yokohama City University School of Medicine, Japan Abstract. At the Advanced Medical Research Center at Yokohama City University School of Medicine, we have been developing a practical surgical simulator for renal surgery. Unlike already commercialized laparoscopic surgical simulators, our surgical simulator is capable of using patient-speciﬁc models for preoperative training and improvement of laparoscopic surgical skills. We have been evaluating the simulator clinically with the aim of using it in renal surgery training at Yokohama City University Hospital. The simulator can be applied to other types of laparoscopic surgery, such as gynecological, thoracic, and gastrointestinal. Here, we report on the technical aspects of the simulator. Keywords. surgical simulator, renal surgery training, ﬁnite elements method , parallel FEM, patient-speciﬁc model

1. Introduction The burden on patients has been greatly reduced by the use of minimally invasive procedures such as laparoscopic surgery. However, these types of surgery require surgeons to gain more experience and expertise than ever before. The prevailing training methods use ”training in mechanical handling with real medical equipment” and the so called ”wet labs” (training using laboratory animals such as pigs). The former is mainly training in equipment handling, not surgical training. The latter can be seen as a type of true surgical training, and it can simulate complex training. However, wet lab training has the substantial drawbacks of extensive preparation time, high cost of preparing the necessary number of animals, and anatomical diﬀerences between humans and animals. In addition, unlike with traditional laparotomy, on-the-job training in laparoscopic surgery is considered diﬃcult, mainly because the instructor cannot assist the operating resident. For 2 Corresponding

Author: R&D Mitsubishi Precision Co., Ltd.; 345 Kamimachiya, Kamaura city, 247-8505, Japan. E-mail: [email protected]: http://www-user.yokohama-cu.ac.jp/˜urology/surgical simulator/index.html

416

M. Ogata et al. / A Development of Surgical Simulator for Training of Operative Skills

these reasons, there is an urgent need to establish eﬀective and economical training methods. We have been studying the development of a practical surgical simulator since 2007, and we have now built a prototype system for urologic surgery. Advanced medical research center and the University hospital at Yokohama City University Medical School have been the nucleus of this cooperative work. The objectives of the project are: (1) development of a preoperative surgical simulator that uses a patient-speciﬁc model generated by surgeons; (2) use of both imagery and force to represent the subtle sensations felt through the forceps during the deformation of organs/vessels and the peeling of membranes; and (3) commercialization of the surgical simulator for use by certiﬁed urology physicians.

2. Previous Work Table 1 classiﬁes surgical simulators by a method analogous to that used in the classiﬁcation of ﬂight simulators. As indicated in the table, the objective of the part-task trainer is training only for the handling of mechanical tools during surgery. The task trainer uses a virtual human body created by virtual reality technology to train surgeons in surgical procedures and hand-eye coordination. The major objectives of the mission rehearsal simulator are to: (1) determine the risks of the real surgery in advance by using preoperative surgery training with a patient-speciﬁc model and thus improving surgical skills to minimize risk during the real surgery; and (2) train surgeons in surgical procedures and handeye coordination more precisely than a task trainer. In general, the technical diﬃculty and cost of the system increase from classiﬁcations (1) to (3). The major technologies used for surgical simulators are: segmentation: extraction of organs, real-time computational mechanics, and human interface. In the most important computational mechanics, the number of presentations relating to the mass-spring model[4,7,1] have decreased. On the other hand, the ﬁnite element method (FEM), the accuracy of which is guaranteed, dominates the presentations. Even an FEM that can handle large deformation has been presented[1]. A practical surgical simulator is composed of these element technologies. There have been few papers written on systems for surgical simulators[7,6], but the number of presentations on individual element technology[4,2,5,3] has been increasing. Task-trainer-type surgical simulators were realized by using the mass-spring model, which was a key early technology. The mission-rehearsal type of surgical simulator using FEM (classiﬁcation 3 in Table 1) has not yet been realized. As long as the mass-spring model is used for deformation, the system is unstable and does not satisfy the upper bound of the critical spring coeﬃcient:

n i mi Kc = , nπ 2 (t)2 where mi is mass, n is the number of mass points, and t indicates the time interval[8]. For real-time processing, t cannot be reduced. Only the number of mass points can be reduced. This reduction leads to poor geometrical similarity with patient anatomy. With this instability and the dissimilarity of the geometry,

M. Ogata et al. / A Development of Surgical Simulator for Training of Operative Skills

417

Table 1. Classiﬁcation of surgical simulators used for training purposes Class

Purpose of training

Engineering diﬃculty and characteristics

(1) Part-task trainer (2) Task trainer

Only for apparatus Basic laparoscopic surgical skills Hand-eye coordination Basic laparoscopic surgical skills Hand-eye coordination Laparoscopic surgical skills for patient-speciﬁc surgery

Relatively easy, with Relatively hard : Spring model Hard: FEM with no area Complicated multi Data generation is

(3) Mission rehearsal

actual operational tools

limitation organ interactions necessary

traditional simulators require complicated tuning of the data model by engineers. Surgeons cannot generate model data freely by themselves. 3. Developed Technologies We have developed a preoperative training simulator that uses a patient-speciﬁc model and ﬁts into classiﬁcation 3 in Table 1. Unlike existing task trainers, our simulator has the following characteristics: (1) accurate and stable simulation of deformation by using the FEM; and (2) a system that can generate a patientspeciﬁc model. Therefore, we can precisely represent the subtle sensations obtained from deformation of the vascular tissues and organs of each patient. This can be done only by using the large deformation parallel FEM that we have developed. 3.1. Patients-Speciﬁc Model Generation Instead of using standard data models prepared by each system developer, we can use the new data model generation system to directly generate patient-speciﬁc models from patients’ diagnostic CT or MRI images. Currently, the generation time for a model, including all processes, is about 3 hours. Our ﬁnal target is for surgeons or radiologists to be able to generate a model in less than 1 hour. Figure 1 shows the procedure used to generate a patient-speciﬁc model. The procedure consists of: (a) segmentation of simulated organs from CT or MRI as input data; (b) application of physical properties such as stiﬀness; and (c) division into ﬁnite elements with various levels of detail. Extraction of simulated organs uses a region-growing method with several teachers points that indicate the region on CT or MRI images that is extracted and the region that is discarded (Fig. 1(b) ). By combining the extracted data on these organs, the volume data to be simulated are generated. Physical properties are then given to the volume data. Finally, after the generation of surface patches to account for the level of detail, the volume data are divided into ﬁnite elements, i.e. tetrahedra. In the case of lymphatic and other organized tissues that do not appear on CT, the standard forms are applied diﬀerently, taking into consideration the orientation of blood vessels and the age, sex, weight, and height of the patient. 3.2. Mechanical Model for the Simulator The surgical simulator has to present large deformation in real time. Considering the need for a balance between the representation of reality and the calculating

418

M. Ogata et al. / A Development of Surgical Simulator for Training of Operative Skills

(a) Volume data of organs are extracted by using the diagnostic images. The volume data are then given physical parameters and ﬁnally separated into ﬁnite elements of tetrahedra.

(b) Segmentation of kidney: Set teacher points used for instruction on extracting or discarding regions.

Figure 1. Patient-speciﬁc model data generation. The organ-simulation data are extracted from the patient’s diagnostic images (e.g. CT or MRI) and then given physical properties, which are ﬁnally separated into tetrahedra.

time required, we have implemented a parallel pseudo-non-linear elastic model based on the co-rotated FEM[3], with improvements. For parallel processing, each calculation unit has diﬀerent organs, so the exchange of external forces between organs and forceps occurs by using the displacement of the contact points, with data communication. By repeating calculations based on the penalty method until the displacements of the contact points converges to some value, the system reaches equilibrium. The details are shown below. (1) Pseudo Non-Linear FEM Model To simulate soft tissue, continuum mechanics that treat spatially distributed computational centers are appropriate. Mathematically, it is necessary to solve the boundary value problem, deﬁned in partial diﬀerential equation Eq. (1), in realtime. The boundary values are determined from the structures of the muscles, membranes, and organs ﬁxed to the bones and from the external forces applied by the forceps: d2 u du ·T +ρ g− 2 +ν =0 , (1) dt dt where, T is Cauchy stress tensor, ρ is the density, g is the body force vector per unit volume, u is the displacement vector, and the ν is the viscosity. The physical meaning of the above equation is that the external force applied to the organs by the forceps breaks the equilibrium. This cases internal force in the ﬁrst term of Eq.(1), and the internal force aﬀects acceleration in the second term, and then causing viscous resistance in the third term. These forces are balanced and the system then reaches a new equilibrium. To numerically solve Eq.(1) accurately, the equation is converted into the following ﬁnite element equation: ¨ = F − C U˙ − RK(R−1 x − X) , MU

(2)

where, x is the current position vector combined with all nodal positions,X is the initial position vector, U is the nodal displacement vector, R is the combined

M. Ogata et al. / A Development of Surgical Simulator for Training of Operative Skills

419

matrix with each rotation matrix of the ﬁnite element, M is the mass matrix, F is an external force vector, C is the viscous matrix, and ﬁnally K is the stiﬀness matrix. If the stiﬀness matrix K is compensated by the rotation R as in Eq.(2), then the Cauchy tensor becomes invariant to the rotation. With this compensation we can solve a large deformation model in a linear FEM scheme[3]. Without this compensation, the volume of the vessel increases drastically when blood vessels are handled in the simulation. After time discretization, the above equation becomes: (M + tC + t2 K)U i+1 = (2M + tC)U i − M U i−1 + t2 F ,

(3)

where, superscript i indicates time and t indicates the time interval. The above equation can be arranged to become the following simultaneous equation: AU i+1 = F .

(4)

By solving the above equation by a conjugate gradient method, we can get a stable displacement U i+1 . To solve the equation for real time, the conjugate gradient method is implemented on the GPUs of GPU cluster. (2) Parallel Processing: Interaction of Organs For parallel processing, the calculation in Eq. (4) is separated into the calculation by each processing unit, as in Eq. (5): Aj U i+1 = Fj , j

(5)

where, j indicates the processing unit number. It is hard to calculate external forces directly in case of multiple collision of organs, i.e. by transfer of forces through other organs and by self-collision(when part of the transformed organ aﬀects itself). However, if we assume that the forceps place temporary forces on by solving Eq. (5). The temporary the organs, then we can get displacements U i+1 j force F j is calculated by using the interference depth d as in Eq. (6), i.e. by the penalty method: Fc = Kp d ,

(6)

where, Kp is a penalty coeﬃcient. Because the organs are located in physically separated processing unit, temporary forces aﬀecting to the other organs are transmitted via communication. By repeating the above calculation in Eq.(5) with the penalty method until the displacements of the contact points connected to the other objects converge to some value, the system reaches equilibrium, i.e. each unknown F j has been solved. 4. Evaluation We conducted two evaluations of the prototype: image quality and force representation accuracy. Figure 3(a)-(c) shows the surgical simulator developed by in-

420

M. Ogata et al. / A Development of Surgical Simulator for Training of Operative Skills

(a) A model data generation system

(b) A simulating system

(c) GPU Cluster

(d) Trocar simulation: Select position

(e) Peeling lymph ducts and connective tissues

(f) Completed peeling then clipping of vessels

Figure 2. A part of the simulated sequence of the surgery for removing a kidney: trocar positioning, peeling of the lymph ducts and connective tissues, clipping and cutting. The model data comprise of 33,581 ﬁnite elements, and the total number of node is 8,174, update ratio for simulation is 30Hz.

tegrating the key technologies described above. The system consists of two major sub-systems: a patient-speciﬁc model data generation system and a simulation system. Figure 3(d)-(f) shows the image quality of part of a sequence of virtual surgery simulating kidney removal. Figure 3(d) shows trocar simulation, Fig. 3(e) shows the removal of lymph ducts and connective tissues around the blood vessels, and Fig. 3(f) shows the clipping of vessels. The model data comprise 33,581 ﬁnite elements, and the total number of nodes is 8174. Figure 3 shows a sequence of collision between organs caused by removal of the lymph ducts and moving of the forceps in a series of urological operations. The time sequence of the force generated with the FEM model for the operations is depicted in Fig. 3(b). The ﬁgure shows that the force gradually increases with pulling on the lymph ducts and then disappears when the ducts are peeled out. Figure 3(c) and (d) show the time sequence of the force corresponding to the operational sequence of touching the vein (thus aﬀecting an adjacent vein) and then moving back to the initial position.

5. Conclusion Here, we have reported on a practical surgical simulator developed at the Advanced Medical Research Center of Yokohama City University School of Medicine for renal surgery. Unlike already commercialized laparoscopic surgical simulators, ours is capable of oﬀering training to improve laparoscopic surgical skills and preoperative training on patient-speciﬁc models. Currently, we have been per-

M. Ogata et al. / A Development of Surgical Simulator for Training of Operative Skills

(a) Pulling the lymph ducts

(b) Time sequence of the force generated when moving the lymph ducts

(c) Collision of forceps with vein A and vein B

(d) Time sequence of the force from multiple collision of organs

421

Figure 3. Time-sequence of the forces generated by removing the lymph ducts and from collision between organs.

forming clinical evaluations aimed at use of the product for renal and urologic surgery at Yokohama City University Hospital. The simulator can also be used for other laparoscopic procedures, such as gynecological, thoracic, and gastrointestinal surgery. References [1] [2]

[3]

[4] [5] [6]

[7]

[8]

Mendoza, C. and Laugier, C.: Tissue Cutting Using Finite Elements and Force Feedback, IS4TM, pp. 175–182 (2003). Meseure, P., Davenne, J., Hilde, L., Lenoir, J., France, L., Triguet, F. and Chailou, C.: A Physically-Based Virtual Environment Dedicated to Surgical Simulatotion, IS4TM, pp. 38–47 (2003). M¨ uller, M. and Gross, M.: Interactive virtual materials, Proceedings of Graphics Interface 2004 , Canadian Human-Computer Communications Society School of Computer Science, University of Waterloo, Waterloo, Ontario, Canada, pp. 239–246 (2004). Nakao, M., Minato, K., Kume, N., Mori, S. and Tomita, S.: Vertex-preserving Cutting of Elastic Objects, IEEE Virtual Reality, pp. 277–278 (2008). Petersik, A., Tiede, B. P. U., Hohne, K.-H. and Leuwer, R.: Realistic Haptic Interaction in Volume Sculpting for Surgery Simulation, IS4TM, pp. 194–202 (2003). Mukai, N., Harada, M., Muroi, K., Miyamoto Y., Uratani A., Yano, T.: Development of PC Based Real-Time Surgical Simulator Transaction of IEICE. D-II, Vol. 84, No. 6, pp. 1213–1221 (2004). Inoue, Y., Masutani, K., Ishii, H., Kumai, N., Kimura, F., Sakuma,I.: Development of Surgical Simulator with High Quality Visualization Based on Finite Element Method and Deformable Volume Rendering, Transaction of IEICE, Vol. 87, No. 1, pp. 271–280 (2004). Delingette, H.: Toward realistic soft tissue modeling in medical simulation, Proc. IEEE, Vol. 86, No. 3, pp. 512–523 (1998).

422

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-422

Virtual Reality Image Applications for Treatment Planning in Prosthodontic Dentistry Takumi OGAWAa, Tomoko IKAWAa, Yuko SHIGETAa, Shintaro KASAMAa, Eriko ANDOa, Shunji FUKUSHIMAa, Asaki HATTORIb and Naoki SUZUKIb. a Department of Fixed Prosthodontics Tsurumi University School of Dental Medicine b Institute for High Dimensional Medical Imaging, Jikei University School of Medicine, {ogawa-t, ikawa-tomoko, shigeta-y, kasama-shintaro, ando-eriko, fukushimas}@tsurumi-u.ac.jp, {hat, nsuzuki}@jikei.ac.jp

Abstract. For successful occlusal reconstruction, the prosthodontists must take several points into consideration, such as those involving issues with functional and morphological findings and aesthetics. They then must unify this information into a coherent treatment plan. In this present study we focused on prosthodontic treatment and investigated how treatment planning and simulation could be applied to two cases. The personal occlusion condition can be reproduced on the virtual articulator in VR space. In addition, various simulations can be performed that involve prosthetesis design. Keywords. Virtual reality, Treatment plannning, Prosthodontic

Introduction Dental or panoramic X-ray images have long since been used for diagnosis and planning. However, these images provide only a bi-dimensional view and can be difficult to understand in regards to the relative positions of the anatomical structures. Recently, computed tomography (CT) images have come into use and are being used for surgical simulation with 3-dimensional images. The computer aided simulation system mainly have been applied to surgical treatments [1][2] . To assist the prosthodontists with this task, we developed a Strategic Approach for a Prosthodontic Planning (SAPP) System that applied the patient’s personal data to the design of the treatment plan and the resultant prosthetesis. In this present study we focused on prosthodontic treatment and investigated how treatment planning and simulation could be applied to each case.

Methods & Materials The SAPP System can diagnose the occlusal relationship, and the cranial reference plane in relation to the occlusal plane on the virtual articulator. The potential applications of our system are reviewed in the following cases.

T. Ogawa et al. / Virtual Reality Image Applications for Treatment Planning

423

•

Step 1: Reconstruction of virtual dentition model

•

Step 2: Reproducing the relative position between the upper and lower dentitions

Results Case 1 was a 67 year-old female (Fig.1). Her chief complaints were related to an aesthetic problem and malocclusion. Her occlusion type was classified by Eichner's classification as C1. Her treatment plan was made via our system. We subsequently assessed the reproduced occlusal condition in VR space (Fig.2), and designed a removable denture for her (Fig. 3).

Figure 1. Intra-oral findings at first visit.

Figure 2. Design of magnetic attachment denture.

Figure 3. Inter-oral findings on magnetic attachment denture.

Case 2 was a 29 year-old female. She also complained of an aesthetic problem and malocclusion (Fig. 4). The extent of bite raising was estimated in the virtual model. In this simulation, the extent for prosthodontic treatment was considered with the rest position of the mandible and the clearance. From these simulation results, we made a provisional restoration (Fig. 5), then the provisional prosthesis was formed based on the our simulation results, and it was provided for her. Finally, we could provide the suitable prosthesis for her (Fig. 6).

424

T. Ogawa et al. / Virtual Reality Image Applications for Treatment Planning

Figure 4. Inter-oral findings on magnetic attachment denture.

Figure 5. VR simulation for prosthodontic treatment.

Figure 6. Post treatment Intra-oral findings.

Conclusions The personal occlusion condition could be reproduced on the virtual articulator in VR space. In addition, various simulations can be performed that involve prosthetesis design. Furthermore, this system proved useful as a communication tool between patients, dentists, and dental technicians.

References [1] Xia J, Samman N, Yeung RW, Shen SG, Wang D, Ip HH, Tideman H.Three-dimensional virtual reality surgical planning and simulation workbench for orthognathic surgery, Int J Adult Orthodon Orthognath Surg.,Winter;15:265-282,2000. [2] Gateno J, Xia JJ, Teichgraeber JF, Christensen AM, Lemoine JJ, Liebschner MA, Gliddon MJ, Briggs ME.: Clinical feasibility of computer-aided surgical simulation (CASS) in the treatment of complex cranio-maxillofacial deformities, J Oral Maxillofac Surg. Apr;65:728-34, 2007.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-425

425

The Initiation of a Preoperative and Postoperative Telemedicine Urology Clinic Eugene S. PARK, MDa,1 , Ben H. BOEDEKER, MD, PhDb,c, Jennifer L. HEMSTREETc, and George P. HEMSTREET, MD, PhDa,c a Department of Urology, University of Nebraska Medical Center, Omaha, NE b Department of Anesthesiology, University of Nebraska Medical Center, Omaha, NE c Research Service, VA Medical Center, Omaha, NE

Abstract. This work describes the establishment of a Telemedicine Urology Clinic at the VA Medical Center in Omaha, Nebraska to serve an underserved veteran population in rural Nebraska. Results from patient satisfaction surveys show that both the patient and the healthcare provider benefit from the telemedicine encounter for both the preoperative and the postoperative setting. Keywords. Preoperative, postoperative, telemedicine, VTC, telehealth

Introduction Telemedicine has brought about an array of innovations that continues to improve medicine and reduce healthcare expenditure. The establishment of a telemedicine urology clinic with the purpose of evaluating rural patient populations pre and postoperatively would benefit both patients and physicians by saving time, reducing costs, and facilitating an effective, virtual face-to-face consultation. Another potential outcome of this project is the ability to provide urologic care to underserved populations in Nebraska by reducing the need for emergency visits in underserved areas.

Methods & Materials Preoperative and postoperative evaluations will be performed using a VTC link to the provider at the Omaha VA Medical Center in Omaha, Nebraska. The Urology clinic at the Omaha VA Medical Center is the location where evaluations transpire, complimented by evaluations at the Urology clinic in Lincoln and Grand Island. The preoperative evaluations consist of a health history interview, a review of health records, and a review of any diagnostic testing ordered prior to surgery according to established practice guidelines. Patients scheduled for surgery also undergo preanesthesiology telemedicine evaluation in the urology clinic. After the evaluation, the patient completes a 15-item, 5-point Likert scale questionnaire (Table 1) pertaining 1 Corresponding Author: Eugene S. Park, MD, Section of Urologic Surgery, 982360 Nebraska Medical Center, Omaha, NE 68198-2360, USA; E-mail: [email protected]

426

E.S. Park et al. / The Initiation of a Preoperative and Postoperative Telemedicine Urology Clinic

to their perceptions of the telemedicine evaluation. Patients scheduled for postoperative evaluations in the Grand Island and Lincoln locations constitute the intervention group, and patients coming to the Omaha VA function as the control group. A second control group will be matched retrospectively and analyzed to avoid bias by the participating physicians who may undergo behavioral modification during the course of the study. Another survey (Table 2) has been developed for the clinics that will assess patient satisfaction for pre and post operative evaluations. This will be implemented in the future at the urologic clinics in all three cities.

Results Based on the telehealth survey results at the Omaha VA Medical Center, almost all of the patients were satisfied with preoperative teleconsultation. According to the results taken by 41 patients in the urology clinic, most related a positive attitude towards a telemedicine preoperative examination. Seventy percent (29 of 41) patients preferred a teleconsultation, and 85% (34 of 40) testified that it was as good as face-to-face consultation in the Anesthesia Pre-op clinic. Patients seen for postoperative care are given a post-visit survey to measure satisfaction. As an outcome of these results, it would be reasonable to state that the establishment of a preoperative and postoperative clinic in Urology could potentially reduce costs, improve time management, and increase patient satisfaction.

Conclusions Establishing a pre and post operative clinic in urology would facilitate many needs of patients and make consultation easier for practicing physicians. Establishing a telemedicine clinic should reduce the costs to patients who travel and in the case of a surgery, cancellation would prevent patients from having to travel for a preoperative evaluation. The results from the surveys strengthen the concept that teleconsultation is an effective and positive approach for implementing preoperative and postoperative examinations. The initial data confirms a positive effect on patient satisfaction. Table 1. VA Telemedicine Patient Satisfaction Survey Question: 1. I was able to clearly see the TV screen during the visit.

2. I was able to clearly hear the health care provider during the visit

3. The health care provider was knowledgeable, skillful, and courteous.

4. My needs were met during the telehealth visit.

5. Telehealth made it easier to receive my health care today.

6. I received good care today.

Response: 95% Agree 5% Disagree 0% Unsure 93% Agree 5% Disagree 2%Unsure 96% Agree 2% Disagree 2% Unsure 98% Agree 0% Disagree 2% Unsure 95% Agree 0% Disagree 5% Unsure 98% Agree

E.S. Park et al. / The Initiation of a Preoperative and Postoperative Telemedicine Urology Clinic

7. Next time, I prefer to see my health care provider “in person”, despite travel inconveniences. 8. I want to use telehealth again.

9. I would recommend telehealth to other veterans.

427

0% Disagree 2% Unsure 19% Agree 64% Disagree 17% Unsure 93% Agree 0% Disagree 7% Unsure 95% Agree 0% Disagree 5% Unsure

Table 2. Omaha VAMC Teleurology Clinic Evaluation Form Question: 1. I could talk freely to the examiner during the teleconsultation. 2. I could hear everything that was being said. 3. I could see the pictures on the screen clearly. 4. The examiner was able to ask me questions. 5. The picture quality on the screen was as good as on my TV at home. 6. It is an advantage to be seen at the Telemed clinic to prevent ravel to Omaha. 7. A teleconsultation could reduce stress on patients by preventing travel. 8. A teleconsultation could save me time. 9. A teleconsultation could save me money. 10. The TV camera made me feel uncomfortable. 11. I was embarrassed using the link to speak to the examiner. 12. The appointment took longer than expected. 13. I would prefer a teleconsultation. 14. I would prefer to see the examiner face to face in Omaha. 15. A teleconsultation is as good as going to Omaha Urology Clinic.

References [1] [2] [3]

B.H. Boedeker, W.B. Murray, B.W. Berg. Patient perceptions of preoperative anaesthesia assessment at a distance. J Telemedicine Telecare 13 (2007), 22-24. T. Broens, R. Veld, M. Vollenbroek-Hutten, H. Hermens, A. Halteren, L. Nieuwenhuis. Determinants of successful telemedicine implementations: a literature study. J Telemedicine Telecare 13 (2007), 303309. J. Ferguson. How to do a telemedical consultation. J Telemedicine Telecare 12 (2006), 220-227.

428

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-428

Modeling Surgical Skill Learning with Cognitive Simulation Shi-Hyun PARK1a, Irene H. SUH b, c, Jung-hung CHIENa,b,c, Jaehyon PAIKe, Frank E. RITTERf, Dmitry OLEYNIKOV c, d, Ka-Chun SIU a,b,c * a Nebraska Biomechanics Core Facility, University of Nebraska at Omaha; b College of Public Health, cCenter for Advanced Surgical Technology d Dept of Surgery, University of Nebraska Medical Center, Omaha, Nebraska, USA. e Department of Industrial and Manufacturing Engineering, Penn State University, f College of Information Sciences and Technology, Penn State University, University Park, Pennsylvania, USA.

Abstract. We used a cognitive architecture (ACT-R) to explore the procedural learning of surgical tasks and then to understand the process of perceptual motor learning and skill decay in surgical skill performance. The ACT-R cognitive model simulates declarative memory processes during motor learning. In this ongoing study, four surgical tasks (bimanual carrying, peg transfer, needle passing, and suture tying) were performed using the da Vinci© surgical system. Preliminary results revealed that an ACT-R model produced similar learning effects. Cognitive simulation can be used to demonstrate and optimize the perceptual motor learning and skill decay in surgical skill training. Keywords. Perceptual Motor Learning, Virtual Reality, Cognition, User Models

1.

Introduction

Competence for technical tasks has become an important issue within the medical profession in recent years. A benefit of VR training is to enhance surgical proficiency of novice surgeons from “pure novice” to “pre-trained novice” [1-3]. The virtual training environment allows the learner to attempt a well-defined task at a set difficulty level with opportunities for repetition and correction of errors. However, most VR trainers are only designed for a set of task difficulty levels without considering the experience of learners. Some learners may be frustrated or overwhelmed by the complexity of the training task, but others may become bored or not challenged enough to progress further. It is crucial to take individual surgical skill and experience into account during trainer development. One approach to make the trainer be more user-specific and adaptive is to explore the learning process and skill decay during training. Contemporary basic research on learning and forgetting has produced a number of findings with potential real-world implications for the training of medical professionals. For instance, researchers have typically described the course of forgetting during laboratory tasks as following a power function [4-5]. Similar mathematical functions have also been used to describe *

Corresponding Author: Email: [email protected] web: http://www.unmc.edu/cast/

S.-H. Park et al. / Modeling Surgical Skill Learning with Cognitive Simulation

429

the course of skill decay for both routine and complex procedural tasks in the workplace [6-7]. However, other researchers [8] note that skill decay may follow an exponential decay, and that individual rates of decay for different types of skills may be influenced by a number of moderating factors, many of which can potentially be modified as part of a training protocol. For instance, the methods used to test skill retention (e.g., recognition vs. recall) and the type of criteria used to judge that retention (e.g., recall of task-related knowledge on a written test vs. behavioral demonstrations on a simulated task) significantly influence estimates of decay in skilled performance. In this empirical study, a cognitive architecture model (ACT-R) was applied to investigate surgical skill learning and forgetting over time. ACT-R offers an approach for simulating human behavior, including learning and forgetting. This study used our understanding of the surgical tasks to create a simulated learner. We hypothesized that the ACT-R model could produce learning effects similar to experimental data.

2.

Methods

2.1 Subject Four young medical students (M1) from the University of Nebraska Medical Center participated in this study. 2.2 Experimental Protocol Participants performed four training tasks (bimanual carrying, peg transfer, needle passing, and suture tying) five times using the da Vinci© surgical system. The order of tasks was randomized. 2.3 Training Tasks The following four inanimate robotic surgical tasks were performed in this study: A. Bimanual carrying (BC), a “pick and place” task: picking up five 15 × 2-mm rubber pieces from a 30-mm metal cap with the right and left instruments, respectively, and carrying them to the opposite caps simultaneously (Fig. 1a). B. Peg transferring (PC), a “both hands coordination” task: picking up one ring from one peg, transferring it to the other hand in space, and then placed it on the peg located at the opposite area. Once participants transferred all rings from the nondominant hand to the dominant hand, they repeated the drill from the dominant hand to the non-dominant hand (Fig. 1b). C. Needle passing (NP), a “translational” task: passing a 26-mm surgical needle through six pairs of holes made on the surface of a latex tube (Fig. 1c). D. Suture tying (ST), a “precision navigation” task: passing a 150 × 0.5-mm surgical suture through a pair of holes made on the surface of a latex tube and making three knots using intracorporeal knots (Fig. 1d).

430

S.-H. Park et al. / Modeling Surgical Skill Learning with Cognitive Simulation

(a) Bimanual carrying

(c) Needle passing

(b) Peg transferring

(d) Suture tying

Figure 1. The tasks performed in the study using the da Vinci© Surgical System.

2.4 Description of the ACT-R Cognitive Models There are existing models of learning that can be used and have been used to examine different learning schedules [9, 10]. Also, it is possible to create a learning model and examine a broad range of training schedules. The ACT-R architecture [11-12] makes it possible to simulate cognitive and perceptual motor skill learning. The learning mechanisms in ACT-R predict that procedural and declarative knowledge are improved by practice in non-linear and not equivalent ways. The equations in ACT-R suggest that massing practice to make the declarative knowledge stronger right before it is proceduralized may make learning procedures more efficient [13]. Therefore, we investigated surgical skill learning over time through simulating a human learner. We analyzed four robotic surgical training tasks into components to implement ACT-R models. The components are listed on Table 1. The four robotic surgical tasks were decomposed with unit task components with motion states (Table 2). For instance, the bimanual carrying (BC) task has 1+2, 3+4 representing that the BC task was decomposed into a) moving to target with left and right hands, b) grasping object with left and right hands, and so on. The task analysis is a theoretical base to develop and test a computational model against complex and dynamic fundamental robotic surgical training tasks. Based on these decompositions of tasks, we implemented ACT-R models using Herbal/ACT-R compiler [13], and compared the results with experimental data.

S.-H. Park et al. / Modeling Surgical Skill Learning with Cognitive Simulation

Table 1. Decomposed unit motion states

1. Move to target (left hand). 2. Move to target (right hand)

431

Table 2. The combinations of unit motions

Task components with motion states BC

1+2, 3+4, 1+2, 5, 6+7

PT

1, 3, 1, 8, 4, 6, 2, 5, 7, 2, 4, 2, 8, 3, 7, 1, 5, 6, 1+2

NP

2, 4, 2, 5, 10, 1, 3, 11, 12, 8, 1, 6, 1+2

ST

2, 4, 2, 5, 10, 1, 3, 2+5, 13, 13, 8, 1+2, 4, 11+12

3. Grasp object (left hand) 4. Grasp object (right hand) 5. Position object 6. Release object (left hand) 7. Release object (right hand) 8. Orient object with both hands 9. Push suture/needle (left hand) 10. Push suture/needle (right hand) 11. Pull suture/needle (left hand) 12. Pull suture/needle (right hand) 13. Rotate suture (left hand) 14. Rotate suture (right hand)

2.5 Data Collection and Analysis Kinematics of the da Vinci© instruments was sampled and recorded at 100 Hz. Analysis of the experimental data included task completion time and the average speed of the instrument tip. Only the task completion time is presented in this paper.

3.

Results

As the learning curves generated by the ACT-R model show in Fig. 2, repetitive practice (iteration) had little learning effect on the BC task, because of the simplicity of the task. However, peg transfer, needle passing, and suture tying showed significant learning effects with practice because those tasks are composed of more complicated and less practiced unit tasks. These results were similar to the experimental data, which also showed that more complex tasks take longer time to reach a plateau effectively than a simple task.

432

S.-H. Park et al. / Modeling Surgical Skill Learning with Cognitive Simulation

Figure 2. Perceptual motor learning curves generated by ACT-R. (BC: Bimanual carrying, NP: Needle passing, PT: Peg transferring, ST: Suture tying)

4.

Conclusions

Our preliminary results revealed that ACT-R models predicted similar learning effects compared with the experimental data. In conclusion, a cognitive simulation model could be used to demonstrate the perceptual motor learning and skill decay in surgical skill training. This model could be used to examine how different learning regimens could have different effects on learning and retention. For example, it would be much easier to run the model 100 times with four different practice times than it would be to get medical residents (or students even) to try these different learning programs. 5.

Acknowledgement:

This work was supported by the Nebraska Research Initiative and the Center for Advanced Surgical Technology, University of Nebraska Medical Center, and ONR grant N00014-06-1-0164, and DTRA HDTRA1-09-1-0054.

References Gallagher et al., Ann Surgery, 241:364-372, 2005 Aggarwal et al., J Surg Res, 145:80-86, 2008 Mukherjee et al., Stud Health Technol Inform, 142, 210-4, 2009 Wixted & Ebbesen, Psychol Sci, 2: 409-415, 1991 Anderson et al., J Exp Psychol Learn Mem Cogn, 25: 1120-1136. 1999 Nembard & Uzumeri, International Journal of Industrial Ergonomics, 25: 315-326, 2000. Brannon & Koubek, Theoretical Issues in Ergonomic Science, 2: 317-355, 2001 Arthur et al., Human Performance, 11: 57-101, 1998 Pavlik, Instructional Science, 35: 407-441, 2007 Ritter & Bibby, Cognitive Science, 32: 862-892, 2008 Anderson et al., Psychological review, 111(4): 1036-1060, 2004 Anderson, How can the human mind exist in the physical universe? New York, NY: Oxford University Press. 2007 [13] Paik et al., Building large learning models with Herbal. In Proceedings of ICCM-Tenth International Conference on Cognitive Modeling, 187-191, 2010

[1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-433

433

Virtual Reality Stroop Task for Neurocognitive Assessment Thomas D. PARSONS,a,1 Christopher G. COURTNEY,a Brian ARIZMENDI, a and Michael DAWSONa2 a University of Southern California, Los Angeles, CA

Abstract. Given the prevalence of traumatic brain injury (TBI), and the fact that many mild TBIs have no external marker of injury, there is a pressing need for innovative assessment technology. The demand for assessment that goes beyond traditional paper-and-pencil testing has resulted in the use of automated cognitive testing for increased precision and efficiency; and the use of virtual environment technology for enhanced ecological validity and increased function-based assessment. To address these issues, a Virtual Reality Stroop Task (VRST) that involves the subject being immersed in a virtual Humvee as Stroop stimuli appear on the windshield was developed. This study is an initial validation of the VRST as an assessment of neurocognitive functioning. When compared to the paper-andpencil, as well as Automated Neuropsychological Assessment Metrics versions of the Stroop, the VRST appears to have enhanced capacity for providing an indication of a participant’s reaction time and ability to inhibit a prepotent response while immersed in a military relevant simulation that presents psychophysiologically arousing high and low threat stimuli. Keywords. Neuropsychological assessment, psychophysiology, ecological validity, virtual environment

Introduction The assessment of traumatic brain injury (TBI) has become a difficult challenge for the DoD medical health system. The reports are sobering: 12-20% of Service Members report symptoms of TBI in theater or during re-deployment [1-2]; 47% of all blast injuries in war zones affect the head [3]; two out of five injuries during OIF II were head, face, or neck injuries [4]; and blast injuries often produce symptoms similar to classical TBI, thereby complicating detection, diagnosis, and treatment [5]. Additionally, mild TBIs (mTBIs) often go undiagnosed when other life-threatening wounds occurred [6]. While many mTBIs resolve in a matter of days or weeks, some cases develop post-concussive syndrome, which includes a number of persistent behavioral, cognitive, and psychological symptoms. Such symptoms vary greatly in both severity and onset [7]. Given the prevalence of TBIs, and the fact that many mTBIs have no external marker of injury, there is a pressing need for innovative technology for initial assessment, treatment, and rehabilitation. The demand for TBI 1

Corresponding Author: Thomas D. Parsons, Ph.D., Director of NeuroSim Lab, University of Southern California, Institute for Creative Technologies, Los Angeles, CA. E-mail: [email protected]. 2 This research is partially supported by the U.S. Army Research Laboratory, Human Research & Engineering Directorate, Translational Neuroscience Branch (Aberdeen Proving Ground, MD).

434

T.D. Parsons et al. / Virtual Reality Stroop Task for Neurocognitive Assessment

assessment that goes beyond traditional paper-and-pencil testing has resulted in the use of automated neurocognitive testing for increased precision and efficiency; as well as the use of virtual environment technology for enhanced ecological validity and increased function-based assessment. Computerized testing batteries, such as the Automated Neuropsychological Assessment Metrics (ANAM; [8]), provide increased accuracy and efficiency (over traditional paper-and-pencil versions) for repeated administration [9]. ANAM has been used for TBI assessment with civilians, athletes, and military personnel [10]. While the ANAM has been found to have adequate predictive value, it does not replicate the diverse military environment in which Soldiers function. A study examining active duty military personnel with mTBI found that 46 percent experienced occupational impairment four to eight months after the injury [11]. Because such injuries can greatly interfere with the ability to perform complex cognitive and emotional processing tasks involved in optimal work performance, there is also a need for assessment technology with greater ecological validity. At varying levels of threat, Soldiers must be able to exercise control of executive functions including the ability to direct and maintain attention, organize incoming stimuli, reason about abstractions, problem-solve, self regulate, and coordinate psychomotor performance. For a neurocognitive measure to be relevant to the assessment of Soldier cognitive functioning, it should go beyond paper-and-pencil measures and provide some indication of a Soldier’s reaction time as well as the tendency to perseverate in a response despite external feedback within high and low threat settings. While the military has historically evaluated such cognitive abilities and related predictions of performance decrements through observation or as net outcomes (e.g., task or mission completion), much less focus has been applied to assessment of environmental and occupational challenges. Although the paper-and-pencil and ANAM Stroop are both validated neurocognitive assessments, there is a need for militarily-relevant tests. To address these issues, the Virtual Reality Stroop Task (VRST) was developed. Like the paper-and-pencil Stroop, the VRST assesses simple attention, gross reading speed, divided attentional abilities, and executive functioning. Like the ANAM, the VRST automates the paper-and-pencil Stroop task and allows for assessment of reaction time. The VRST goes beyond the ANAM and paper-and-pencil versions of the Stroop by replicating the diverse military environment in which Soldiers function.

1. Methods This study was designed as an initial validation of the VRST. A further goal of this study was to utilize psychophysiological measures to predict levels of threat and cognitive workload. In addition, we compared paper-and-pencil Stroop, ANAM Stroop, and VRST on behavioral measures such as reaction time (the time from stimulus onset to the first button press), response time (the time it took for the correct response to be made), and number of correct responses in each zone. Note, the paper-and-pencil version does not allow for assessment of reaction time of each individual response to a stimulus, so the analyses reflect percentage correct. The three main questions were: 1) Can threat level in the VRST be predicted using behavioral and physiological data? 2) Can Stroop difficulty in the VRST be predicted using behavioral and physiological data? 3) How do paper-and-pencil, ANAM, and VR versions of the Stroop behavioral results compare?

T.D. Parsons et al. / Virtual Reality Stroop Task for Neurocognitive Assessment

435

1.1. Participants and Procedure The University of Southern California’s Institutional Review Board approved the study. A total of 20 college-aged subjects participated in the study. After informed consent was obtained, basic demographic information was recorded. Presentation of the ANAM and VRST versions of the Stroop were counterbalanced. While experiencing the VRST, participant psychophysiological responses were recorded using the Biopac system. Electrocardiographic activity (ECG), and Electrodermal activity (EDA), were recorded simultaneously using a Biopac MP150 system and a computer running Acknowledge software. EDA was measured with the use of 8 mm silver-silver chloride electrodes placed on the volar surface of the distal phalanges of the index and middle fingers of the non-dominant hand. Electrodes were filled with a 0.05 molar isotonic NaCl paste to provide a continuous connection between the electrodes and the skin. Skin conductance responses were scored as the largest amplitude response beginning in a window of 1 to 3 s following stimulus onset. A response was defined as having amplitude greater than 0.01 μS. EDA was included because it tends to be sensitive to the presence of startling or threatening stimuli, and positive or negative emotional events [12]. ECG was recorded with use of a Lead 1 electrode placement, with one 8 mm silver-silver chloride electrode placed on the right inner forearm about 2 cm below the elbow and another placed in the same position on the left inner forearm. A third 8 mm silver-silver chloride electrode was placed on the left inner wrist to serve as a ground. Electrode sites were cleaned with alcohol prep pads in order to improve contact. Interbeat intervals (IBIs) were scored as the time difference in seconds between successive R waves in the ECG signal. A median interbeat interval was recorded during each of the same 5 second sampling periods used to assess skin conductance level. ECG was included because cognitive workload has been linked with changes in heart rate [13]. Further, in previous studies [14-16], greater sympathetic predominance in cardiac control in response to Stroop task demands has been observed. Following completion of the VRST protocol, subjects were assessed for simulator sickness. Notably, none of the subjects reported simulator sickness following VRST. 1.2. Paper-and-Pencil Stroop Task The paper-and-pencil Stroop test measures an individual’s ability to shift cognitive set and proffer a measure of a subject’s ability to inhibit a prepotent (i.e., an overlearned) response in favor of an unusual one. For the Stroop Color and Word Test, the subject was seated at a desk and presented a Word Page with color words printed in black ink, a Color Page with ‘Xs’ printed in color, and a color-Word Page with words from the first page printed in colors from the second page (the color and the word do not match). 1.3. The ANAM Version of the Stroop Task The ANAM Stroop is a computer automated version that requires the subject to press one of three computer keys to identify each color. Subjects were seated in front of a blank computer screen. The ANAM Stroop began with a computerized trial of practice words, during which time color words were presented on the otherwise blank screen and the subject was asked to respond to the color of these words by pressing an appropriate key on a keypad. For the interference task subjects pressed the key corresponding to the color of the letters rather than the color indicated by the word.

436

T.D. Parsons et al. / Virtual Reality Stroop Task for Neurocognitive Assessment

1.4. Virtual Reality Stroop Task The VRST involves the subject being immersed in a virtual Humvee as Stroop stimuli appear on the windshield. The VRST is a measure of executive functioning and was designed to emulate the paper-and-pencil as well as ANAM version of the Stroop test. The apparatus used for the virtual humvee included a Pentium 4 desktop computer with a 3 GHz Processor; 3 GB of RAM; and an nVidia GeForce 6800. Two monitors were used: 1) one for displaying the Launcher application which is used by the examiner administering the test; and 2) another for displaying the participant’s view of the virtual environment in the HMD. Like the ANAM version, the VRST requires an individual to press one of three computer keys to identify each of three colors, (i.e., red, green, or blue). Unlike the ANAM version, the VRST adds a simulation environment with military relevant stimuli in high and low threat settings. Participants wore an eMagin Z800 Head Mounted Display with an InterSense InteriaCube 2+ attached for tracking. A Logitech Driving Force steering wheel was clamped on to the edge of a table in front of the monitors. Accelerator and brake pedals was positioned under the table. To increase the potential for sensory immersion, a tactile transducer was built using a three foot square platform with six Aura bass shaker speakers (AST-2B-04, 4Ω 50W Bass Shaker) attached. The tactile transducer was powered by a Sherwood RX-4105 amplifier with 100 Watts per Channel x 2 in Stereo Mode. Development of the scenes, levels, and the virtual Iraqi/Afghani environment was done using Maya animation software. The environments were rendered in real time using the Gamebryo 3-D graphics engine with a fully customizable rendering pipeline, including vertex and pixel shaders, shadows, bump maps, and screenspace geometric primitives. The application also utilizes the NeuroSim Interface (NSI) developed in the Neuroscience and Simulation Laboratory (NeuroSim) at the University of Southern California. The NSI was used for data acquisition, stimulus presentation, psychophysiological monitoring, and communication between the psychophysiological recording hardware and the virtual environment. Configuration parameters were saved to files using the NSI and automatically loaded through its control module, allowing the experimenter to rapidly switch configurations in order to perform specific experimental sequences. The NSI also enabled the sending of event markers from the stimulus presentation computer to the data recording device. Finally, the NSI used compiled Matlab scripts to filter the incoming psychophysiological data in real-time. The software runs on Windows XP 32-bit, and requires 5 Gb of free Hard Drive space. 1.5 Stimuli and Design Participants were immersed in the VRST as psychophysiological responses were recorded. ECG and EDA were collected as participants rode in a simulated Humvee through alternating zones of low threat (i.e., little activity aside from driving down a desert road) and high threat (i.e., gunfire, explosions, and shouting amongst other stressors). The participants experienced 3 low threat and 3 high threat zones designed to manipulate levels of arousal (start section; palm ambush; safe zone; city ambush; safe zone; and bridge ambush). The order of threat levels was counterbalanced across participants. The VRST was employed to manipulate levels of cognitive workload, and was completed during exposure to the high and low threat zones. The VRST consisted of 3 conditions: 1) word-reading, 2) color-naming, and 3) interference. Each Stroop condition was experienced once in a high threat zone and once in a low threat zone.

T.D. Parsons et al. / Virtual Reality Stroop Task for Neurocognitive Assessment

437

Stimuli were presented for 1.25 seconds each, and participants were asked to respond as quickly as possible without making mistakes. 1.6 Data Analytics Two separate stepwise regressions were used to determine the efficacy of the psychophysiological and behavioral data for predicting levels of threat and cognitive workload. Specifically, average skin conductance level, heart rate (recorded as interbeat intervals), reaction time, response time, and number correct for each zone were entered as predictors for each dependent variable. In order to determine differences in participant response to the ANAM, VRST, and the paper-and-pencil Stroop test, a 3 (Stroop condition) by 3 (presentation type; ANAM, VRST, and paper and pencil Stroop) repeated measures ANOVA was employed for the percentage of correct answers in each condition. A Greenhouse-Geisser correction was used for all reported main effects and interactions with greater than one degree of freedom. Additionally, all significant main effects and interactions were followed up with paired-samples t-tests in order to determine the nature of these effects. A sequentially rejective test procedure based on a modified Bonferroni inequality was used on significant t-test results to prevent inflation of type 1 error rates [17].

2. Results Can threat level in the VRST be predicted using behavioral and physiological data? Results revealed that skin conductance level was the only significant predictor of threat level, β = 4.25, t(19) = 2.75, p < 0.01, with increased skin conductance levels in high threat zones. Behavioral data did not predict threat level. Can Stroop difficulty in the VRST be predicted using behavioral and physiological data? Heart rate and behavioral data were reliably predictive of cognitive workload. Response time significantly predicted workload as responses were slowed during the highest difficulty interference condition, β = 0.34, t(19) = 5.18, p < 0.001. Heart rate was also predictive of workload, as heart rate increased during the interference task, β = 0.19, t(19) = 2.84, p < 0.01. Subjects also gave more correct responses during the low cognitive load tasks of color naming and word-reading than during the high cognitive workload interference task, β = 0.17, t(19) = 2.74, p < 0.01. Thus, heart rate and response time increased during the highest difficulty interference condition, while the number of correct responses decreased How do paper-and-pencil, ANAM, and VRST behavioral results compare? The results of the analyses on the percentage of correct responses in each condition revealed a main effect of Stroop condition, F (2, 18) = 22.26, p < 0.001. This was the result of significantly fewer correct responses in the interference condition than either the colornaming condition, t (19) = 4.35, p < 0.001, or the word-reading condition, t (19) = 5.75, p < 0.001. The color-naming and word-reading conditions did not differ significantly. A main effect of presentation type was also exhibited, F (2, 18) =47.34, p < 0.001. A significantly greater percentage of correct responses were exhibited in the paper and pencil Stroop in comparison with either the ANAM, t (19) = 3.49, p < 0.01, or the VRST, t (19) = 7.07, p < 0.001. The ANAM also resulted in a greater percentage of correct responses than the VRST, t (19) = 7.24, p < 0.001. Finally, an interaction

438

T.D. Parsons et al. / Virtual Reality Stroop Task for Neurocognitive Assessment

between Stroop condition and presentation type was uncovered, F (4, 16) = 7.11, p < 0.01, which was due to significant differences between interference and both colornaming, t (19) = 3.67, p < 0.01, and word-reading conditions, t (19) = 4.48, p < 0.001, during the VRST only. The ANAM and paper and pencil Stroop tests failed to create significant differences in performance between the Stroop conditions.

Figure 1. Comparison of three versions of the Stroop

3

Discussion

Our goal was to conduct an initial pilot study to validate the VRST through comparison with paper-and-pencil and ANAM versions of the Stroop test. We believe that this goal was met. Results revealed that skin conductance level was the only significant predictor of threat level, with increased skin conductance levels in the high threat zones. These results seem to comport well with findings that electrodermal activity tends to be sensitive to the presence of startling or threatening stimuli, and positive or negative emotional events [12]. Heart rate and behavioral data were reliably predictive of cognitive workload. This appears consistent with other work in human computer interaction that has linked cognitive workload with changes in heart rate [13]. Finally, consistent with previous studies [14-16], a shift toward greater sympathetic predominance in cardiac control in response to Stroop task demands was observed, as heart rate and response time increased during the highest difficulty interference condition, while the number of correct responses decreased. As one would expect, high threat zones in the VRST resulted in a significantly smaller proportion of correct responses when compared to the paper-and-pencil and ANAM versions of the Stroop. Interestingly, the difference between ANAM interference and color-naming conditions were not significant. An interaction between Stroop condition and presentation type was uncovered, which is due to significant differences between interference and colornaming in the high threat zones, and the low threat zones. A main effect of presentation type was also demonstrated, which was due to significantly faster responses to the ANAM than the VRST’s low threat, and high threat zones. We recognize that the current findings are only a first step in the development of this tool. More steps are necessary to continue the process of test development and to fully establish the VRST as a measure that contributes to existing assessment

T.D. Parsons et al. / Virtual Reality Stroop Task for Neurocognitive Assessment

439

procedures for the diagnosis of mTBI. Although the VRST as a measure must be fully validated, current findings provide preliminary data regarding the validity of the virtual environment as a measure of executive functioning. Nevertheless, the fairly small sample size requires that the reliability and validity of the VRST be established using a larger sample of well-matched participants. This will ensure that current findings are not a sample size–related anomaly. In summary, the findings reported herein provide an initial validation of the VRST as a neurocognitive assessment of Soldier neurocognitive functioning. When compared to ANAM and paper-and-pencil versions of the Stroop, the VRST appears to provide an enhancement in that it has the capacity for providing an indication of a Soldier’s reaction time and ability to inhibit a prepotent response while immersed in a military relevant simulation that presents psychophysiologically arousing stimuli.

References [1] A.I. Schneiderman, E.R. Braver, & H.K. Kang, Understanding sequelae of injury mechanisms and mild Traumatic Brain Injury incurred during the conflicts in Iraq and Afghanistan: Persistent postconcussive symptoms and Posttraumatic Stress Disorder, American Journal of Epidemiology 167 (2008), 14461452. [2] T. Tanielian, & L.H. Jaycox, (Eds.) Invisible Wounds of War: Psychological and Cognitive Injuries,Their Consequences, & Services to Assist Recovery. Santa Monica, CA: RAND Corporation (2008). [3] K.H. Taber, D.L. Warden, & R.A. Hurley, Blast-related traumatic brain injury: what is known? Journal of Neuropsychiatry and Clinical Neuroscience 18 (2006), 141-145. [4] A.L. Wade, J.L. Dye, C. Mohrle, & M.R. Galarneau. Head, face, and neck injuries during Operation Iraqi Freedom II: results from the US Navy-Marine Corps Combat Trauma Registry. Journal of Trauma 63 (2007), 836-840. [5] J.M. Wightman, S.L. Gladish, Explosions and blast injuries, Annals of Emergency Medicine 37 (2001), 664-678. [6] E.M. Martin, W.C. Lu, K. Helmick, L. French, and D.L. Warden, (2008). Traumatic brain injuries sustained in the Afghanistan and Iraq wars, American Journal of Nursing 108 (2008), 40-7. [7] I. Cernak, J. Savic, D. Ignjatovic, M. Jevtik, Blast injury from explosive munitions, Journal of Trauma 47 (1999), 96-103. [8] R.L. Kane & G.G. Kay, Computerized assessment in neuropsychology: A review of test and test batteries. Neuropsychology Review 3 (1992), 1–117. [9] T. Roebuck-Spencer, W. Sun, A.N. Cernich, K. Farmer, & J Bleiberg, Assessing change with the Automated Neuropsychological Assessment Metrics (ANAM): Issues and challenges, Archives of Clinical Neuropsychology 22(S1) (2007), S79-S87. [10] T.D. Parsons, A. Notebaert, & K. Guskewitz, Application of Reliable Change Indices to Computerized Neuropsychological Measures of Concussion, The International Journal of Neuroscience 119 (2009), 492-507. [11] A.I. Drake, N. Gray, S. Yoder, M. Pramuka, & M. Llewellyn, Factors predicting return to work following mild traumatic brain injury: A discriminant analysis. Journal of Head Trauma Rehabilitation 15 (2000) 1103-1112. [12] T. Butler, H. Pan, O. Tueschent, et al., Human fear-related motor neurocircuitry. Neuroscience 150 (2007), 1-7. [13] Y. Hoshikawa and Y. Yamamoto, Effects of Stroop color-word conflict test on the autonomic nervous system responses, Heart and Circulatory Physiology 272 (1997), 1113–1121. [14] Y.N. Boutcher and S.H. Boutcher, Cardiovascular response to Stroop: effect of verbal response and task difficulty, Biological Psychology 73 (2006), 35–241. [15] J.P.A. Delaney and D.A. Brodie, Effects of short-term psychological stress on the time and frequency domains of heart rate variability, Perceptual and Motor Skills 91 (2000), 515–524. [16] K.J. Mathewson, M.K. Jetha, I.E. Drmic, S.E. Bryson, J.O. Goldberg, G.B. Hall, D.L. Santesso, S.J. Segalowitz, & L.A. Schmidt, Autonomic predictors of Stroop performance in young and middle-aged adults, International Journal of Psychophysiology 76 (2010), 123-129. [17] D.M. Rom. A sequentially rejective test procedure based on a modified Bonferroni inequality. Biometrika 77 (1990), 663–665.

440

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-440

Implementation of Virtual Online Patient Simulation V PATEL MRCS1, R AGGARWAL PhD MA MRCS, D TAYLOR Msc and A DARZI PC KBE HonFREng, FMedSci Division of Surgery, Department of Surgery and Cancer, Imperial College, London

Abstract. The development and use of virtual patients has become more expansive. Previous strategies to aid their development have been described to aid their formation. This study describes the development of a series of virtual patients following a methodology proposed by Posel et al [1]. Ten virtual patients with surgical pathology were developed using a reproducible framework. This article serves to guide virtual patient authors as a working description of virtual patient design in order to assist them for future virtual patient development.2 Keywords. Virtual Patient, Second Life, Virtual World

Introduction Virtual Patients may be defined as a simulation through which a user has to undertake communication, information gathering and apply diagnostic reasoning towards a computer generated patient. [2] Posel et al [1] postulated several authoring tips, which may serve as a guide to the development of virtual patients. Whilst these tips provide the basis for development there is an increasing variety of interfaces for presenting virtual patient cases, and in particular an interest in the possibility of using 3 dimensional virtual worlds as the primary platform on which to host them. [3] The aim of this study was to design and develop a series of virtual patients that could be implemented in the virtual world of Second Life. Furthermore we sought to use the guidelines stipulated by Posel et al [1] as a framework for virtual patient development. Whilst these guidelines are theoretical in nature this study will demonstrate their application in the development of our series of virtual patients with the intention of further guiding future virtual patient authors.

1 Corresponding

Author: Division of Surgery, Imperial College London, 10th Floor QEQM, St Mary's

Hospital, South Wharf Road, London W2 1NY, UK; E-mail: [email protected] 2 Acknowledgements:

The study was supported by the London Deanery under their Simulation Technology-enhanced Learning Initiative.

V. Patel et al. / Implementation of Virtual Online Patient Simulation

441

1. Methods & Materials 1.1. Outline The method of virtual patient design followed a specific path namely the design of paper based cases and data entry of this information into a case editor. This was followed by the application of the case data into a web based virtual patient "player" to test the case, and subsequently transferred to a controller that directed interaction with the user interface in the virtual world of Second Life.Below is a description of the design methodology according to some of Posel’s [1] tips. 1.2. Determining Case Content and Choosing a Design Model Three general surgical conditions were selected as a focal point for the clinical cases to be developed. These conditions were lower gastrointestinal bleed (Figure 1), acute pancreatitis and small bowel obstruction. An additional condition was selected-urinary retention-as a case that could subsequently be used as an orientation case for training. In order to plan the cases it was fundamental to establish the optimal design methodology. Previous design methods that have been described for virtual patient design include the linear string and the branching design method. [4] Whilst the branching design provides higher fidelity for the end user it is significantly more complex in terms of design. For the purpose of this study in order to ease design, either the linear string of pearls method was employed or a combination with a limited amount of branching was utilized.

Figure 1. Demonstrating the virtual patient experiencing a large per rectal bleed

442

V. Patel et al. / Implementation of Virtual Online Patient Simulation

1.3. Organisation of the Case For case planning VUE (Visual Understanding Environment) (Tufts University, Massachusetts), an open source Java program, was utilised to plan the case development. This program ensured the appropriate mapping out of a case with critical steps in case development defined as nodes. Arrows linked these nodes therefore ensuring the visualisation of the case progression (Figure 2). If a case required multiple steps for progression then a combination of the linear and branching methods was used. However if a case required a single step for progression the linear case was employed. Whilst alternatives to VUE may be used it is important to develop a basic network diagram for a case prior to further development.

Figure 2. Displaying the combination of the branch design and the linear string of pearls method for the design of the Acute Pancreatitis case for the 2nd Year Resident.

1.4. Storyboarding the Case All cases followed a generic outline in design. In order to follow the specific framework focusing on information elicitation, processing and decision making all cases were designed in the same format: history, examination, investigation and management. This was developed using Microsoft Word. This was to ensure a familiar process for the user so they would be able to recognise the disease process and initiate a management plan. This process of using a similar methodology to storyboard all cases ensures that information may easily be transferred between cases, therefore reducing the time for development of further variations of a case. The information needed to present the case in the virtual world included text, audio for breath and heart sounds and diagnostic images. The heart and breath sounds were

443

V. Patel et al. / Implementation of Virtual Online Patient Simulation

retrieved from an Imperial College Heart and Breath Sounds compact disc that is dispensed to undergraduate medical students. Diagnostic radiographs were obtained as jpeg images (Joint Photographic Image’s Expert Group). 1.5. Managing Case Complexity The development of this series of virtual patients incorporated cases of increasing complexity. The reason for this was the target end users would be at different stages of Postgraduate Surgical Training namely Intern, 2nd Year Resident and 4th Year Resident. In order to ensure the cases were designed to the specific training level a designated event was the focal point for each specified case (Table 1). Two specialist Surgical trainees decided these principal events. To aid simplicity in case design and development it is best to focus on a single event but ensure that this is appropriate for the end user and specifically appropriate for their training level. Table 1. Depicting the essential themes for each case

Scenario Lower GI Bleed

Training Level Intern 2nd Resident Not life threatening;

Year

4th Resident

Year

PLUS: Significant bleedrequires transfusion

PLUS: Persistent bleed + normal endoscopic investigations-requires definitive management

Requires monitoring Acute Pancreatitis

Simple Pancreatitis; No systemic upset

PLUS: Low urine output + exacerbation of COPDrequires intravenous rehydration + nebulisers

PLUS: Type II respiratory failure + acute renal failure-requires BIPAP and haemofiltration

Small

Small Bowel Obstruction; requires conservative management

PLUS: onset of strangulation-requires diagnosis and proceed to laparotomy

PLUS: Post-operative anastomotic leak. Requires diagnosis and appropriate intervention

Bowel Obstruction

1.6. Selection of the Appropriate Authoring Application The cases have been designed in a particular manner from planning to storyboard. However the vital step is transferring this case design into an authoring application, which is then interpreted by another piece of software (the controller) to produce interactions in the 3D user interface. The authoring application, Eclipse was used as a case editor, which is an open source multi-language development environment although the most frequent language used is Java. The purpose of the Eclipse editor was to produce XML files of data and store this within a retrievable database. The format for the case input was as similar as possible to the storyboard of the case within Microsoft Word. At appropriate steps in the case, the audio-visual media items and animations that would be visible to the user were uploaded and linked-in. These files, such as an abdominal radiograph were each assigned to a specific target where they would be visualised in the 3D virtual world; in this case a desktop monitor.

444

V. Patel et al. / Implementation of Virtual Online Patient Simulation

An additional requisite is the ability to test a case thoroughly by playing it through repeatedly and making changes as required. This was done using the web player; hosted on an Imperial College server. The virtual web player was able to display the logical sequence through a case. A message broker was developed to relay messages from the player through to the target objects in Second Life using http (hypertext transfer protocol). Each target was scripted so as to handle the data supplied to it by the controller in the most appropriate way. 1.7. Tackling Interactivity In order to ensure that the cases provided a high fidelity and realistic simulated experience for the user a design strategy was adopted to use a series of ward layouts, so that the clinician could encounter a different virtual patient in each. To ensure the fidelity of the simulation appropriate clinical environments such as an Accident and Emergency, a Clinical Ward and High Dependency Ward (Figure 3) were created to represent the potential locations of the virtual patient. Each of these environments contained an additional room, which would contain the blood and imaging request and retrieval service.

Figure 3. Displaying the High Dependency Unit

1.8. Anticipation and Navigation Whilst Second Life’s popularity in Higher Education and Research has grown since 2007, a student's familiarity with it could not be assumed. Therefore with the integration of the virtual patient player into Second Life a standard interface was used, based on nested menus to assist the user in navigation of the virtual patient case. As the menus were similar for all cases the successful navigation through one case meant that all of the subsequent cases could be successfully navigated and further justified using the same design methodology for all virtual patient cases in a series. An orientation program was developed so that all users would obtain a similar level of proficiency before commencing the assessed cases. This involved the creation of a short user manual describing the basics of using an avatar in Second Life and the navigation through the cases. This was accompanied by the orientation case focusing on urinary retention. It was to be a requisite that all end users would navigate through the urinary retention case prior to the other cases.

V. Patel et al. / Implementation of Virtual Online Patient Simulation

445

1.9. Integrate Evaluation The case series described are to be used for assessment of trainee’s performance. Regardless of the purpose of the virtual patients it is essential to integrate an evaluation process that could be utilised by a clinician to formulate an assessment. In place of a learning management system that would normally handle the administration for a class of end users, the web player incorporated an account creation page, where users details could be entered. Each user to be assessed was assigned a unique reference number. A phone was used in Second Life to represent a user logging in to the system. When clicked the phone requested their unique reference number, and subsequently used this to track their responses. This ensured that information could be retrieved regarding an end user’s performance through a case. A participant’s login and case progression would generate data being transferred, through comma separated value files, to the clinician’s computer as a Microsoft Excel file.

2. Results The total time required for the production of this series of virtual patients was 10 months. Below is a description of one of the virtual patients created with the focus on acute pancreatitis. This description incorporates all three training levels. 2.1. Virtual Patient Mr Hart is a 65 year old, man who presents with epigastric pain and 2 episodes of vomiting. His past medical history includes COPD and high blood pressure. His observations are normal but his abdominal examination reveals epigastric tenderness. His Amylase is 1024 but his modified Glasgow score is 1, due to his age. His breathing deteriorates where his Oxygen saturations reduce to 90%. His respiratory examination reveals a polyphonic wheeze. He also has experienced a reduced urine output, which is less than 30 ml over the last four hours. His Arterial Blood Gas reveals a pH of 7.33 with retention of carbon dioxide (6.5). His serum Creatinine level has risen to 142. He is subsequently transferred to the high dependency unit. His breathlessness deteriorates despite bronchodilator therapy. His oxygen saturations are now 87% on fiO2 of 0.24. He has stopped producing urine over the last four hours and his serum creatinine has further risen to 328 with a serum potassium level of 6.7. His arterial blood gas shows a mixed respiratory and metabolic acidosis with a pH of 7.24.

3. Discussion The primary aim of this study was to design and develop a series of virtual patients, with surgical pathology, that could be incorporated into a readily accessible virtual world platform. The secondary aim was to ensure that the majority of design objectives postulated by Posel et al [1] were met. This was achievable through the utilisation of allied web technologies; which linked together was able to produce the end product within a cost and time effective modality. The success of the production of the virtual

446

V. Patel et al. / Implementation of Virtual Online Patient Simulation

patient was hinged on the ability to outline the design and storyboard the cases and feasibly implement this into the editor. The case structure specifically the outline of the history, examination, investigation and management ensured the cases could be updated in a reproducible manner, which made subsequent case editing simpler following initial progress. The successful development of these virtual patients has ensured that they may be utilised by the training Surgeon. However to ensure that these virtual patients are relevant for clinical practise their face and construct validity must be determined.

References [1] [2]

[3] [4]

Posel N, Fleiszer D, Shore BM. 12 Tips: Guidelines for authoring virtual patient cases. Med Teach 2009;31(8):701-8. Triola MM, Campion N, McGee JB, Albright S, Greene P, Smothers V, et al. An XML standard for virtual patients: exchanging case-based simulations in medical education. AMIA Annu Symp Proc 2007:741-5. Conradi E, Kavia S, Burden D, Rice A, Woodham L, Beaumont C, et al. Virtual patients in a virtual world: Training paramedic students for practice. Med Teach 2009;31(8):713-20. Huwendiek S, De leng BA, Zary N, Fischer MR, Ruiz JG, Ellaway R. Towards a typology of virtual patients. Med Teach 2009;31(8):743-8.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-447

447

Patient-Specific Cases for an Ultrasound Training Simulator a

Kresimir PETRINECa,b,1, Eric SAVITSKY a,c and Cheryl HEIN a Center for Advanced Surgical and Interventional Technology (CASIT) b UCLA Computer Science Department, Los Angeles, CA 90095 b UCLA School of Medicine, Los Angeles, CA 90095

Abstract. We present a laptop-based ultrasound training simulator and a novel method for creation of patient-specific training datasets. The simulator consists of a laptop computer, a peripheral probe interface device with an embedded threedegree-of-freedom orientation sensor, a virtual US probe, a virtual patient model, a simulated US brightness scan (B-scan). Simulated B-scans are rendered from US volumes, which are synthesized using volume reconstruction methods from twodimensional US imagery captured in a data acquisition stage from clinical patients. This methodology enables creation of patient-specific ultrasound simulation datasets for training purposes. Keywords. Patient-specific, ultrasound, training, simulation, reconstruction

Introduction With the advent of low-cost, hand-carried ultrasound imaging systems (HCUS), it is now possible to use ultrasound at the point-of-care, to improve diagnostic and procedural guidance capabilities. Unlike CT and MRI, ultrasound can provide real-time multiplanar assessments. Ultrasound is the safest (i.e., no radiation risk or contrast toxicity) and most versatile cross-sectional imaging modality due to its real-time nature and multiplanar capabilities [1]. Millions of healthcare providers around the world (physicians and nurses) who are not traditional users of ultrasound now have the opportunity to leverage the versatility of ultrasound at the point-of-care, including in trauma and combat casualty care. The use of ultrasound is a well-documented method of improving patient safety [2], [3]. However, mastering the use of ultrasound for diagnostic and procedure guidance is not a trivial undertaking. Healthcare providers require hundreds of hours of didactic training, hands-on training and clinical practice to become proficient. The criteria for defining competence and credentialing are highly variable and have not been uniformly defined across specialties. Ultrasound training has a high opportunity cost and requires three scarce resources: an ultrasound machine, a patient model (most often a human volunteer, although some training is done using gel-tissue phantom models), and an expert sonographerinstructor. This dependence on scarce resources creates a serious bottleneck in training, making it impossible to rapidly train large numbers of healthcare providers in an 1

E-mail: [email protected]

448

K. Petrinec et al. / Patient-Speciﬁc Cases for an Ultrasound Training Simulator

efficient manner. Furthermore, the traditional training approach, which includes rotations in clinical settings, does not systematically expose trainees to an appropriate range of pathologies, relying instead on whatever cases arise during a given trainee’s clinical rotation. This further extends the timeline required to expose trainees to a sufficient number of abnormal pathologic conditions to ensure clinical competence in the use of HCUS technology. Simulation-based training has been identified as an effective method of training a wide array of clinical skills [4], [5], [6]. For training purposes, successful simulation of ultrasound must provide close correspondence between the perceptual motor task of using the simulator and the perceptual-motor task of using a real ultrasound machine. Our focus has been on the fidelity of the combination of probe motion and B-scan image interpretation. For training, it is important that no clinically-significant visual artifacts or features not normally present in clinical ultrasound scans be introduced. Some previous US simulators synthesize an approximation of US data directly from computed tomography (CT) scan instead of from real-patient US data [7], [8], [9]. However, those systems are not able to truly replicate US, because US speckles, which come from interference of the signal reflected by tissue micro-inhomogeneities are difficult to simulate given the resolution of CT source imagery [9]. This paper presents a new approach to ultrasound training. Volunteer patients and tissue phantoms are replaced with a virtual three-dimensional (3D) patient model, simulated on a standard laptop computer and representing the anatomy of any desired body part or organ. The experience of using an ultrasound machine and handling a probe are simulated through the coupling, via software, of volumetric ultrasound datasets with the virtual patient and a peripheral probe with an embedded three-degreeof-freedom orientation sensor. The US image on the simulator shows a B-scan corresponding to the location and orientation of virtual probe as it is shown on the virtual model. The image also corresponds to the orientation of the peripheral probe, which is the primary user input device to the simulator and drives the virtual probe orientation interactively, which, in turn, drives the B-scan imagery as it would in an ultrasound machine. Advantages of this approach for training ultrasound include minimizing the need for either a clinical ultrasound machine or a volunteer patient; deployment on a lowcost, widely available platform conducive to massively scalable distribution of training opportunities; and the ability to create and archive a large and extensible case library of patient-specific training datasets representing different procedures and disease states or conditions. These combine to offer all trainees the opportunity to practice to mastery in almost any environment (anywhere, anytime) and to experience a much broader range of case types than are available in standard clinical training rotations.

1. Methods & Materials 1.1. Data Acquisition Data acquisition is comprised of: a) determining the position and orientation of a typical 2D US probe during completion of a controlled clinical “volume scan” designed for the purpose of later volume reconstruction; b) to collect US images from a typical clinical ultrasound machine while completing the volume scan; and c) to timestamp all data collected for later post processing.

K. Petrinec et al. / Patient-Speciﬁc Cases for an Ultrasound Training Simulator

449

Figure 1. Data acquisition. A sonographer holds an US probe with motion tracker mounted on it.

A data acquisition system is shown in (Figure 1) and consists of: • •

SonoSite® M-Turbo™ portable ultrasound with C60 5-2 MHz curved array transducer, InertiaCube3™ (InterSense, Inc.) motion tracker with three degrees of freedom (3-DOF),

•

a laptop computer,

•

and a digital camera.

The motion tracker is attached to the US probe and connected to a laptop computer through a USB port. The tracker detects a full 360 degree range of motion about each of 3 axes (roll, pitch, and yaw). The tracker we selected provides a 180 Hz update rate, which virtually eliminates tracker-induced latency. Calibration is required to establish the rigid body transformation between the sensor and the B-scan, carefully controlling for the geometries and deformations involved in manual scanning and probe motion to achieve sufficient accuracy in scans. The computer asynchronously reads and stores the orientation of the probe and adds a timestamp, denoting the time at which the orientation is obtained. At the same time, ultrasound B-scans are being captured and stored in the Digital Imaging and Communications in Medicine (DICOM) standard format. 1.2. Volume Reconstruction A thorough description and grouping of the various freehand 3D reconstruction algorithms can be found in [10], where 3D volume reconstruction is classified into three categories based on implementation: Voxel-Based Methods (VBM), Pixel-Based Methods (PBM), and Function-Based Methods (FBM). In our approach, data reconstruction is based on PBM and consists of four stages: 1) Preprocessing of images and timestamps A number of scaling, alignment, offset and image modifications are applied to provide a consistent and organized baseline for further processing. This results in unique timestamps for each preprocessed image and an identified region of interest (ROI).

450

K. Petrinec et al. / Patient-Speciﬁc Cases for an Ultrasound Training Simulator

Figure 2. PBM reconstruction method. 3D voxel grid is shown as 2D grid symbolizing the centers of the voxels. 2D input images are illustrated as lines where the pixel centers are marked with circles.

2) Volume size computation The geometric relationship between the ROI and source data geometric parameters, including pivot points and orientation angles, is used to compute an affine transformation per image, which is then applied to each image to project it as the boundary of a slice in 3D Cartesian space in voxel units. 3) Pixel-based reconstruction Two blocks of memory are utilized in the reconstruction process: one for volume, and the other for a mask which will keep track of which voxels were set and how many times each was visited. For all images, two adjacent images are taken and projected into the volume by applying corresponding transformations to all image pixels. Also, all voxels on the line between corresponding pixels of the two images are linearly interpolated (Figure 2). Voxels visited multiple times are averaged. 4) Iterative reconstruction After pixel-based reconstruction there may be voxels that were not masked. Those voxels are set to the average value of adjacent voxels and the process is iterated until all voxels are assigned values. 1.3. Simulation and Visualization When using a reconstructed volume in the simulator, the virtual probe is positioned at the origin of reconstructed 3D volume and the peripheral probe (with embedded orientation sensor) is used to control yaw, pitch and roll angle of the virtual probe and drive the corresponding US image transformations, which are designed to locate the voxel corresponding to current probe locations, interpolate to obtain intensity, which determines the pixel value for the corresponding 2D US simulated image. The ultrasound simulator (Figure 3) is a custom designed application written in C++. The graphical user interface (GUI) consists of the virtual patient body, virtual probe, simulated US B-scan dataset, and a menu. The application begins with scene selection consisting of specific instances of these elements. A scene is configured using Extensible Markup Language (XML). A typical scene consists of one or more 3D models, probes and procedures. 3D models, defined in obj file format, may represent any of the graphical elements. All objects and images in the scene are rendered using OpenGL.

K. Petrinec et al. / Patient-Speciﬁc Cases for an Ultrasound Training Simulator

451

Figure 3. Ultrasound trainer.

Each model has a corresponding name and type. The type is an integer value ranging from zero to three. While zero represents solid static objects, one indicates dynamic objects such as probes, which can change position and orientation in the virtual environment. Two represents solid objects which may be transparent or opaque, and three indicates that an object needs to be tested for collision and is typically used for the topmost model such as skin along which the virtual probe may slide. Probes models are a vital part of ultrasound simulator. Each probe has a name, typical application and beam properties. There are two major categories of 2D transducers: linear array and curved array. Both categories can be described with four parameters: top width, bottom width, depth and top dip. Linear array transducers have equal top and bottom width. Probe parameters can be interactively changed during the simulation, including probe virtual model, type and depth, and the result is immediately visible on the screen showing the new probe or beam depth, as in real US machines.

2. Results Initially, we collected data from six volunteers using SonoSite® M-Turbo™ portable ultrasound with C60 5-2 MHz curved array transducer, and InertiaCube3™ (InterSense, Inc.) motion tracker mounted on the probe. The songrapher was instructed to find the optimal window and then to scan the body using the procedure developed for our data collection process. Each volunteer’s Morison’s Pouch, bladder and spleen were scanned several times resulting with 83 sweeps from which we reconstructed 83 volumes. Later, we used those volumes to create reconstructed datasets simulating Bscans for each scan window. Those reconstructed, simulated B-scans were compared to the original scans (Figure 4). The result shows minor artifacts due to discretization, and slightly lower sharpness due to data averaging of PBM. To quantitatively measure the result we used the root-mean-square (RMS) variation and standard deviation (STD). The average RMS variation was 4.3 %, and STD was 1.9 %. Qualitatively, these were judged clinically insignificant by sonography experts. In our next experiment we simulated two B-scans simultaneously: one from US and the other from CT volumes. For the US data acquisition we used a Philips iU22 ultrasound system with C5-2 broadband curved array transducer. The motion tracker was mounted on the probe and timestamps were asynchronously stored on a laptop.

452

K. Petrinec et al. / Patient-Speciﬁc Cases for an Ultrasound Training Simulator

Sagittal scans of the liver were performed several times. The outlines of volumes were displayed on laptop screen and visually inspected for artifacts and discontinuities (Figure 5). An example discontinuity in a volume simulated US B-scans is shown in Figure 6. This provides a preselection screening prior to reconstruction.

Figure 4. B-scans of Morrison’s Pouch. Comparison of original scan used in volume reconstruction (left) and scan simulated from reconstructed volume (right).

Figure 5. Visualization of swipe captured on a PC using motion tracker.

Figure 6. Examples of US B-scans of liver simulated from a volume with discontinuity (left) and from a volume without discontinuity (right).

K. Petrinec et al. / Patient-Speciﬁc Cases for an Ultrasound Training Simulator

453

The volume was manually registered with the Ultimate Human (UH) virtual model of human body (http://www.cgcharacter.com/ultimatehuman.html). The resulting datasets were evaluated in the simulator by two expert sonographers. Datasets were judged appropriate for clinical training on objective criteria related to clinical scan fidelity and subjective criteria related to simulator responsiveness and motion performance.

3. Conclusions This paper presents a new and efficient method for US training through a unique, extendable platform. The method is very flexible compared to other methods for constructing volumes from collected images, because it allows for unconstrained movement of the US probe. While this method requires complex data acquisition processes compared with previous simulation techniques, it enables acquisition of an unprecedented range of unique, patient-specific datasets. Future directions will include refinement of data acquisition methods to enable data acquisition at multiple clinical sites.

Acknowledgments This research was sponsored by the UCLA Center for Advanced Surgical and Interventional Technology (CASIT) and the Telemedicine and Advanced Technology Research Center (TATRC) of the U.S. Army. Their support is gratefully acknowledged.

References [1] [2]

Kalorama Information, “Medical Imaging Markets, Volume III: Ultrasound” (100 pages), 2006. Rothschild JM. Ultrasound guidance of central vein catheterization. In: Making health care safer: A critical analysis of patient safety practices http://www.ahrq.gov/clinic/ptsafety/chap21.htm. Agency for Healthcare Research and Quality. Available at http://www.ahrq.gov/. Accessed June 21, 2010. [3] K.G. Shojania, B.W. Duncan, K.M. McDonald, R.M. Wachter, and A.J. Markowitz, “Making health care safer: a critical analysis of patient safety practices”, Evid Rep Technol Assess (Summ), (43):i-x, pp. 1-668, 2001. [4] J.B. Prystowsk, G. Regehr, D. Rogers, et al. “A virtual reality module for intravenous catheter placement”, Am J Surg 1999;177, pp. 171-175. [5] R. Rowe, R.A. Cohen, “An evaluation of a virtual reality airway simulator”, Anesth Analg, 2002, 95, pp. 62-66. [6] S.D. Small, “Medical Education: Thoughts on Patient Safety Education and the Role of Simulation”, 2004. (Accessed September 9, 2004, at http://www.ama-assn.org/ama/pub/category/12059.html.) [7] T. Reichl, J. Passenger, O. Acosta, and O. Salvado, “Ultrasound goes gpu: real-time simulation using cuda,” M. I. Miga and K. H. Wong, Eds., vol. 7261, no. 1. SPIE, 2009, p. 726-116. [8] A. Hostettler, C. Forest, A. Forgione, L. Soler, and J. Marescaux, “Realtime ultrasonography simulator based on 3d ct-scan images,” in Medicine Meets Virtual Reality 13: The Magical Next Becomes the Medical Now, vol. 111. IOS Press, 2005, pp. 191 – 193. [9] J.L. Dillensegerab, S. Laguittonab, and E. Delabrousse, “Fast simulation of ultrasound images from a ct volume,” Computers in Biology and Medicine, vol. 39, no. 2, pp. 180 – 186, 2009. [10] O.V. Solberg, F. Lindseth, H. Torp, R.E. Blake, and T.A. Nagelhus Hernes, “Freehand 3d ultrasound reconstruction algorithms–a review,” Ultrasound in Medicine and Biology, vol. 33, no. 7, pp. 991 – 1009, 2007.

454

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-454

Stereo Image-Based Arm Tracking for In Vivo Surgical Robotics Eric PSOTA a , Kyle STRABALA b , Jason DUMPERT b , Lance C. PÉREZ a , Shane FARRITOR b and Dmitry OLEYNIKOV c a University of Nebraska-Lincoln, Department of Electrical Engineering b University of Nebraska-Lincoln, Department of Mechanical Engineering c University of Nebraska Medical Center, Department of Surgery Abstract. Motor-based tracking and image-based tracking are considered for threedimensional in vivo tracking of the arms of a surgical robot during minimally invasive surgery. Accurate tracking is necessary for tele-medical applications and for the future automation of surgical procedures. An experiment is performed to compare the accuracy of the two methods, and results show that the positioning error of image-based tracking is signicantly less than that of motor-based tracking. Keywords. Minimally invasive surgery, surgical robotics, stereo imaging

Introduction Minimally invasive surgery (MIS) is the use of long thin tools through several small incisions to perform surgical procedures that would otherwise require a large incision. MIS offers recognized benets over traditional surgical techniques, so much so that they have become the standard of care in certain procedures [5], [6], [12], [13]. These benets include shorter recovery time, less post-operative pain, and improved cosmetics. The use of a robotic platform to perform MIS procedures offers further benets including a more intuitive interface for the surgeon, the ability to scale motions, and motion ltering to remove tremors from the surgeon’s input [10], [4], [1], [2], [8]. The da Vinci Surgical System (Intuitive Surgical, Sunnyvale, CA) is a commercially available surgical robot that allows a surgeon, located at a remote workspace, to control tools that are inserted into the abdomen through small incisions [2], [8]. Other robots in development for surgical applications include the Raven [7] and CoBRASurge [9]. The robot prototype described in this paper was designed and built for MIS. As shown in Figure 1, it is composed of a torso with two arms, each with four degrees of freedom (DOF), three for translation and one for wrist rotation. At the end of each arm are interchangeable tools including a tissue grasper, hook electrocautery, and needle driver. R The robot is controlled remotely by a surgeon who manipulates two PHANTOM Omni (SensAble, Woburn, MA) controllers, one for each robot arm. To guide the robot’s arms during in vivo surgery, the surgeon relies on video feedback obtained from stereo digital cameras mounted on the robot’s torso. Each motor of the robot is controlled by a proportional-integral-derivative (PID) position control loop where the position is measured via a magnetic encoder on the back

E. Psota et al. / Stereo Image-Based Arm Tracking for In Vivo Surgical Robotics

455

Figure 1. Surgical robot visualized as a torso with two arms. A stereo camera pair is paritally visible at the center of the torso.

of the motor. The relative position and orientation of the robot arms, referred to as the the robot conguration, are computed from these encoder measurements and the robot’s forward kinematics. Along with each motor are two sets of gearing, one set for gear reduction and one set for obtaining the desired rotational motion of the robot. The gearing in the robot used in the experiments presented in this paper has some backlash. Backlash refers to the clearance between the mating teeth of two gears and results in a one-to-many mapping between the angular positions of the mating gears and the actual position of the robot arms. Due to backlash in the gearing of the robot, the difference between the actual robot conguration and the calculated conguration can result in a position error of the tool vertex of up to ±1.5 cm in any dimension. This error is signicant because the major dimension of the robot’s workspace is 20 cm. This paper describes an image-based alternative for tracking the robot’s arms using a stereo pair of cameras. Experimental results indicate that image-based tracking is capable of providing more accurate conguration feedback to the controller.

1. Methods and Materials 1.1. Motor-Based Tracking In this method, control of the robot arms is performed with position feedback calculated from the motor encoder measurements and the robot’s forward kinematics. A magnetic encoder is mounted to the back of each motor. The outputs of the encoder are two on/off signals that are 90 degrees out of phase with each other. These two signals represent a 2-bit number that can be used to determine both the angular distance and direction. An FPGA microcontroller inputs these signals and calculates the relative motion of the motor. The absolute angular position of each motor is calculated using the measured relative motion from a known ‘home’ conguration of the robot, which is estimated visually. Finally, the robot conguration is calculated from the absolute angular positions of the motors and the robot’s forward kinematics. The motors used in the robot are 8mm and 10mm diameter DC coreless motors available from MicroMo (Clearwater, FL). These motors are obtained as an assembly of the motor, a magnetic encoder with the trade name HEM3, and a planetary gearhead with a gear reduction of either 256:1 or 1024:1. The planetary gearheads have an advertised angular backlash of less than 3 degrees. Further spur and miter gearing is used to create

456

E. Psota et al. / Stereo Image-Based Arm Tracking for In Vivo Surgical Robotics

Figure 2. Surgical robot arms and the colored markers used to track the vertices of the left and right arm.

the desired rotational motion in the robot and has a gear ratio of approximately 1:1. The backlash in the spur and miter gearing is estimated at less than 8 degrees. The motors are supplied with a pulse width modulated (PWM) 15 Volt signal to move the motors, and position feedback is obtained with a 2-channel encoder via a NI 9505 motor driver and LabVIEW FPGA software on a Compact RIO (National Instruments, Austin, TX). The primary source for errors in the joint angle measurements is the gearing backlash, resulting in a position error at the tool vertex of up to ±1.5 cm in any dimension. 1.2. Image-Based Tracking This section examines an image-based method to track the movement of the left and right arms of the surgical robot using video from a stereo pair of cameras mounted to the torso of the robot. The image-based tracking method relies on easily distinguishable markers placed on both arms. Figure 2 shows the colored markers on both left and right surgical robot arms. The goal of image-based tracking is to use the three colored markers on each arm to estimate the location of the vertices of the arms. There are three primary steps to achieving this goal: 1) Identify the 2D centroids of the markers in both the left and the right images, 2) Compute the 3D coordinates of the centroids of the markers, and 3) Estimate the 3D coordinates of the left and right vertices. 1.2.1. Identifying the Centroids of the Markers The markers placed on the left and right arms were chosen to have highly saturated green and cyan color, respectively. These colors were chosen for their high contrast with organic matter, which is assumed to consist of varying shades of red. To isolate the markers from the rest of the image, a threshold is used to locate the colors of interest. First, let IL be the original left image and IR be the original right image. The pixel intensity for color c ∈ {R, G, B} at row m ∈ {1, . . . , M } and column n ∈ {1, . . . , N } is given by IL [m, n, c] ∈ {0, . . . , 255} in the left image, and by IR [m, n, c] ∈ {0, . . . , 255} in the right image. The four binary images GL , GR , CL , and CR are used for identifying pixels in IL and IR that contain the colors of interest, and are given by 1 if IL,R [m,n,G] > 2.5×IL,R [m,n,R] and IL,R [m,n,G] > 2.0×IL,R [m,n,B]

GL,R [m,n] =

0

otherwise

and CL,R [m,n] =

1 if IL,R [m,n,B] > 1.0×IL,R [m,n,G] and IL,R [m,n,G] > 1.5×IL,R [m,n,R] , 0 otherwise

E. Psota et al. / Stereo Image-Based Arm Tracking for In Vivo Surgical Robotics

(a) Left Image: Orig.

(b) Right Image: Orig.

(c) Left Image: GL , CL

457

(d) Right Image: GR , CR

Figure 3. Original images and a visualization of binary images from color isolation.

where GL and GR are used to identify the green markers and CL and CR are used to identify the cyan markers. Figure 3 shows a visualization of the binary images. The red background represents binary 0’s, and the green (blue) pixels represent binary 1’s, in GL (CL ) and GR (CR ). It is important to note that performing binary erosion followed by binary dilation can signicantly reduce noise within the binary images. After creating the binary images, the centroids of the three largest connected objects are identied using connected component labeling [11]. If the three markers are incorrectly identied, this error can be detected after computing the markers’ 3D locations (using the method given in Section II.B.2) by comparing the results to their known relative 3D locations. In the case of an error, the results are declared erroneous and the data obtained from the frames is not used. If the centroids accurately dene the center of the three markers, it is possible to exploit the restricted movement of the robot’s arms to identify the ‘top’, ‘bottom’, and ‘tip’ markers. The six conditions used to uniquely identify the 2D location of the markers in both left and right images are given by Ltip (2) > Ltop (2), Ltip (2) > Lbottom (2), Lbottom (1) > Ltop (1), Rtip (2) < Rtop (2), Rtip (2) < Rbottom (2), and Rbottom (1) > Rtop (1), where each coordinate is dened by the [row, column] location within the image. For example, Ltip (1) is the row location of the left ‘tip’ marker and Ltip (2) is the column location of the left ‘tip’ marker. 1.2.2. Compute the 3D Coordinates of the Markers After calibrating a stereo camera conguration, it is possible to obtain the 3D position of an object from its pixel location in both left and right images. Camera calibration is performed using the direct linear transform (DLT) method in [3] to obtain the camera calibration matrices PL and PR . A camera calibration matrix P allows for the projection of any homogeneous point in 3D space to a homogeneous 2D pixel location on the image plane of the camera. Alternatively, it is possible to obtain a set of 3D candidates for each 2D pixel coordinate from P, which forms a line in 3D space. To dene the line in 3D, two distinct points along the line are used. The rst point is the camera center C, dened by the right null space of P. The second point can be computed using the pseudo-inverse of P given by P+ = PT (PPT )−1 , where PP+ = I. For any 2D homogeneous pixel coordinate x, it is known that the point P+ x lies on the line that passes through x, because P(P+ x) = Ix = x. Using two 3D points C and P+ x, a 3D line in can be dened that contains all the points that project to pixel location x. If the 2D pixel coordinates of an object are known in two cameras with different camera centers, it is possible to compute the 3D coordinate of the object by nding the + intersection of the two lines dened by {CL , P+ L xL } and {CR , PR xR }. In practice, the two lines in 3D space typically do not have a perfect point of intersection because of

458

E. Psota et al. / Stereo Image-Based Arm Tracking for In Vivo Surgical Robotics

Figure 4. Visualization of the estimation of the vertices from the marker locations.

imperfect camera matrices or errors in the corresponding pixel locations. However, with carefully calibrated cameras and pixel correspondences that are reasonably accurate, it can be assumed that the two 3D lines come very close to intersecting. Thus, it is sufcient to compute an approximate point of intersection by projecting the 3D lines onto two orthogonal planes. First, the camera centers are rotated and translated so that they lie on the x-axis and the cameras point in -z direction. Then, the intersection of the lines when projected onto the xz-plane can be used to obtain the x and z coordinates, and those coordinates can be inserted into the line equations to obtain the y coordinate. Using this method, one can compute the homogeneous 3D locations of the markers, given by Ltop 3D , tip top tip bottom bottom L3D , L3D , R3D , R3D , and R3D . 1.2.3. Estimate the 3D Coordinates of the Vertices Using the 3D coordinates of the markers it is possible to estimate the coordinates of the vertices of the robot arms using triangulation. This is because the relative location of the markers and vertices in 3D space is static. To initially compute the relative location of the vertices to the markers, the coordinates of the vertices are selected manually in a single vertex pair of images to obtain Lvertex and R3D . After determining the relative position from 3D the manually selected points, this information is reused to automatically compute the 3D coordinates of the vertex for all future images. Figure 4 presents a visualization of the vertex point estimation. The endpoints and intersections of the white lines show the 2D pixel coordinates computed from the 3D bottom locations or the centroids of the markers and the vertices, given by PL Ltop , 3D , PL L3D tip top tip bottom PL L3D , PR R3D , PR R3D , and PR R3D . 2. Results This section presents experimental results for motor-based tracking and image-based tracking of the surgical robot’s arms. In order to compare the two tracking methods, it is rst necessary to establish a ‘ground truth’. Both tracking methods estimate the position of the vertex of the robot’s arms. If the true position of the arms is known, a positioning error can be computed using the average Euclidean distance between the real positions and the estimated positions. In this experiment, each of the robot arm vertices was positioned using the PHANR R controllers to touch ten different points on an object created with Legos , TOM Omni R which has well-known geometric properties. Using the fact that Legos have a precise width and depth of 8.0mm, and a height of 9.6mm, the ten 3D coordinates of the points touched by the robot arms are given, up to rotation and translation, by

459

E. Psota et al. / Stereo Image-Based Arm Tracking for In Vivo Surgical Robotics

30

25 Motor Image

Motor Image 20

20

Absolute Error (mm)

Absolute Error (mm)

25

15

10

10

5

5

0

15

0

5

10

15

20

25

30

35

40

45

0

0

5

10

(a) Left Arm

15

20

25

30

35

40

45

(b) Right Arm

Figure 5. Absolute error in the Euclidean distance between each of the ten points touched by the vertices of the surgical robot arms.

XLreal

⎡ 48.0 28.8 ⎢ 16.0 28.8 ⎢ 24.0 9.6 ⎢ 56.0 9.6 ⎢ ⎢ 16.0 0.0 = ⎢ 24.0 38.4 ⎢ ⎢ 32.0 0.0 ⎢ 8.0 19.2 ⎣

−24.0 ⎤ −24.0 ⎥ −16.0 ⎥ −16.0 ⎥ ⎥ −8.0 ⎥ , −32.0 ⎥ ⎥ 0.0 ⎥ ⎥ −24.0 ⎦ 32.0 28.8 −24.0 16.0 0.0 −8.0

real XR

⎡ 32.0 9.6 ⎢ 40.0 9.6 ⎢ 56.0 19.2 ⎢ 40.0 0.0 ⎢ ⎢ 8.0 9.6 = ⎢ 32.0 38.4 ⎢ ⎢ 48.0 9.6 ⎢ 56.0 19.2 ⎣

−16.0 ⎤ −16.0 ⎥ −24.0 ⎥ 0.0 ⎥ ⎥ −16.0 ⎥ . −32.0 ⎥ ⎥ −16.0 ⎥ ⎥ −24.0 ⎦ 40.0 28.8 −24.0 8.0 9.6 −16.0

Instead of attempting to align the coordinate systems used by the motor-based tracking and image-based tracking, the accuracy of the tracking methods were compared using the relative Euclidean distances between each of the ten measurements. In the following, D(Xi , Xj ) is the Euclidean distance between points Xi and Xj . The advantage of using relative Euclidean distance is that it does notchange with rigid transformations using only rotation and translation. A total of 10 = 45 dis2 tances were computed between each of the distances within the set of real distances, motor-based distances, and image-based distances. Figure 5 shows a plot of the absolute Euclidean distance error of both the motor-based tracking and image-based tracking real real motor motor real real given by |D(X(L,R),i , X(L,R),j ) − D(X(L,R),i , X(L,R),j )| and |D(X(L,R),i , X(L,R),j )− image image , X(L,R),j )|, for all 45 unique sets of points {i, j} such that i = j. The avD(X(L,R),i erage absolute Euclidean distance error using the motor-based tracking was 9.27mm for the left arm, and 6.92mm for the right arm. The average absolute Euclidean distance error using the motor-based tracking was 1.45mm for the left arm, and 1.01mm for the right arm. Thus, image-based tracking considerably improves the accuracy of position estimates when compared to motor-based tracking.

3. Conclusion This paper introduced a new stereo image-based method for computing the position of a robot’s arms and then compared the performance of this algorithm to conventional motorbased tracking. The experimental results show that image-based tracking offers signicantly improved performance over motor-based tracking. It is possible that a combina-

460

E. Psota et al. / Stereo Image-Based Arm Tracking for In Vivo Surgical Robotics

tion of image-based and motor-based tracking, where the image-based tracking measurements are used for real-time correction of the errors resulting from motor-based tracking, would perform even better. This hybrid method might be more robust in that it would allow tracking to continue in situations where image-based tracking is not possible, e.g., when blood and/or tissue occludes the markers on the robot arms. Future work includes the integration of image-based tracking with real-time threedimensional reconstruction of the in vivo surgical environment. This integration might allow surgeons to interact by instructing automated robot arm movement within the eld of view, thus eliminating much the human error associated with full manual controls.

Acknowledgements This work was funded in part by TATRC2 grant W81SWH-09-2-0185.

References [1] [2]

[3] [4] [5] [6]

[7]

[8]

[9] [10]

[11]

[12] [13]

G. H. Ballantyne. Robotic surgery, telerobotic surgery, telepresence, and telementoring. review of early clinical results., 2002. F. Corcione, C. Esposito, D. Cuccurullo, A. Settembre, N. Miranda, F. Amato, F. Pirozzi, and P. Caiazzo. Advantages and limits of robot-assisted laparoscopic surgery: preliminary experience. Surgical endoscopy, 19(1):117–119, 01/01 2005. M3: 10.1007/s00464-004-9004-9. R. I. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, ISBN: 0521540518, second edition, 2004. J. H. Kaouk, R. K. Goel, G.-P. Haber, S. Crouzet, and R. J. Stein. Robotic single-port transumbilical surgery in humans: initial report. BJU international, 103(3):366–369, 2009. P. Leggett, R. Churchman-Winn, and G. Miller. Minimizing ports to improve laparoscopic cholecystectomy. Surgical endoscopy, 14(1):32–36, 01/01 2000. M3: 10.1007/s004649900006. M. S. L. Liem, Y. van der Graaf, C. J. van Steensel, R. U. Boelhouwer, G.-J. Clevers, W. S. Meijer, L. P. S. Stassen, J. P. Vente, W. F. Weidema, A. J. P. Schrijvers, and T. J. M. V. van Vroonhoven. Comparison of conventional anterior surgery and laparoscopic surgery for inguinal-hernia repair. The New England journal of medicine, 336(22):1541–1547, May 29 1997. M. J. H. Lum, D. C. W. Friedman, G. Sankaranarayanan, H. King, K. Fodero, R. Leuschke, B. Hannaford, J. Rosen, and M. N. Sinanan. The raven: Design and validation of a telesurgery system. The International Journal of Robotics Research, 28(9):1183–1197, Sep. 1 2009. K. Moorthy, Y. Munz, A. Dosis, J. Hernandez, S. Martin, F. Bello, T. Rockall, and A. Darzi. Dexterity enhancement with robotic surgery. Surgical endoscopy, 18(5):790–795, 05/01 2004. M3: 10.1007/s00464003-8922-2. C. Nelson, X. Zhang, B. Shah, M. Goede, and D. Oleynikov. Multipurpose surgical robot as a laparoscope assistant. Surgical Endoscopy, 24:1528–1532, 2010. 10.1007/s00464-009-0805-8. J. P. Ruurda, I. A. M. J. Broeders, R. P. M. Simmermacher, I. H. M. B. Rinkes, and T. J. M. V. Van Vroonhoven. Feasibility of robot-assisted laparoscopic surgery: An evaluation of 35 robot-assisted laparoscopic cholecystectomies. Surgical Laparoscopy Endoscopy & Percutaneous Techniques, 12(1), 2002. ID: 00129689-200202000-00007. H. Samet and M. Tamminen. Efcient component labeling of images of arbitrary dimension represented by linear bintrees. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 10(4):579 –586, Jul. 1988. G. Stiff, M. Rhodes, A. Kelly, K. Telford, C. P. Armstrong, and B. I. Rees. Long-term pain: Less common after laparoscopic than open cholecystectomy. British Journal of Surgery, 81(9):1368–1370, 1994. R. Tacchino, F. Greco, and D. Matera. Single-incision laparoscopic cholecystectomy: surgery without a visible scar. Surgical endoscopy, 23(4):896–899, 04/01 2009. M3: 10.1007/s00464-008-0147-y.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-461

461

A Simulation Framework for Wound Closure by Suture for the Endo Stitch Suturing Instrument Sukitti PUNAK and Sergei KURENOV Roswell Park Cancer Institute

Abstract. Our simulation framework for wound closure by suture is designed for education and training purposes. Currently, it is designed specifically to support a simulation of the Endo Stitch™ suturing instrument by Covidien, and could be extended for other surgical instruments designed for intracorporeal suturing. The framework allows the user to perform a virtual wound closure by suture with real surgical instrument handles customized to fit on haptic devices. The wound simulation is based on a triangular surface mesh embedded in a linear hexahedral finite element mesh, whereas the suture simulation is based on a simplified Cosserat theory of elastic rods. The simulation utilizes a combination of physically-based and control-based simulations. Keywords. Endo Stitch, physically-based simulation, suture, wound closure

Introduction In laparoscopic surgery, the intracorporeal suturing is an important basic skill all surgeons learn in the modern medical era. The main component of the learning process is how to use instruments to perform the procedure properly and efficiently. Most suturing instruments, including robotic surgical instruments, are designed specifically to help facilitate intracorporeal suturing, but with a steep learning curve. An example of such instruments is the Endo Stitch suturing instrument by Covidien (Figure 1). It requires unique manipulations at the handle for the needle movement between the jaws during suturing, requiring first-time users to spend more than ten minutes to complete a simple suturing exercise [1]. This paper describes our simulation framework that has been extended from the simple simulation framework [1] to simulate a continuous closure of a wound by suturing using the Endo Stitch suturing instrument. Our simulation framework can also be easily adapted for other suturing procedures with other intracorporeal instruments.

1. Methods and Materials The developed framework consists of three main simulated components: i) the wound, ii) the suture, and iii) the surgical instrument. We start by describing how to modify and fit an Endo Stitch instrument to a haptic device, followed by how the suture is simulated, next how to construct and simulate the wound, and finally their interactions.

462

S. Punak and S. Kurenov / A Simulation Framework for Wound Closure by Suture

(a)

(b)

Figure 1. (a) An Endo Stitch suturing instrument is modified (b) to connect to the haptic device.

1.1. Simulation of the Endo Stitch Suturing Instrument The haptic device used in the simulation is a PHANTOM Omni® haptic device by SensAble Technologies, Inc. (Figure 1b). A real Endo Stitch suturing instrument is modified to connect to the haptic device [2] (Figure 1b). The movement of the haptic device controls the movement of the virtually reconstructed instrument. The instrument’s sliding levers position is translated into an electrical signal sent to the haptic device to control the simulation of the virtual instrument jaws. This allows the user to hold the real handle of the instrument to control the virtual instrument in the virtual world. 1.2. Simulation of the Suture Our simulated suture is a simplified computation version of the CORDE model [3] to make it run faster, however, at the expense of a slightly less accurate method to preserve computation time for other simulation tasks. The simulated suture is composed of a chain of point masses representing centerlines and a chain of quaternions representing orientations. As a result, the computation is similar to a computation of two mass-spring systems with the coupling of the two chains enforced by constraints. With the right adjustment of simulation parameters, the developed virtual surgical suture exhibits bending and twisting similar to a real suture. A semiimplicit Euler numerical integration is used to advance the simulated suture through time. The simulated suture is also augmented with a binary tree of hierarchy bounding spheres for collision detection, which is also used for self-collision detection. A knot recognition algorithm based on Dowker notation is also added to the suture simulation. 1.3. Simulation of the Wound The simulated wound is based on finite element method (FEM). The wound is simulated by a triangular surface mesh embedded in a linear hexahedral finite element method similar to the method in [4]. The method of embedding the surface mesh in the finite element (FE) mesh allows us to change the triangle mesh for the wound’s surface or the grid resolution of the FE mesh virtually independently of each other.

S. Punak and S. Kurenov / A Simulation Framework for Wound Closure by Suture

(a)

463

(b)

Figure 2. (a) The FE mesh and the instrument’s bounding cylinders and (b) the wound’s interpolated points.

The watertight triangle mesh, representing the wound’s surface, is created by modeling software – Blender [5] in our case. This surface mesh is then bounded by a rectangular volume. The bounding volume is divided into sub-rectangular volumes in the x, y, and z directions based on the detail needed (Figure 2a). The sub-volumes that are not intersected or not inside the surface mesh are deleted. The rest of the subvolumes then form the hexahedron elements of the FE mesh. The wound behavior is simulated based on the dynamic behavior of the linear hexahedral FE mesh. The wound’s surface deformation is updated according to the deformation of the FE mesh. An implicit Euler numerical integration is used to advance the simulated wound’s FE mesh through the time. 1.4. Simulation of the Wound Closure by Suture All of the simulated components have to interact with one another. Their intersections are resolved by collision detection (and response). During suturing, the needle of the Endo Stitch suturing instrument is used to puncture the simulated wound. As a result, parts of the suture and parts of the wound will be connected to one another. A connection is created by linking of the wound’s punctured point, which is a vertex on the wound’s surface mesh, to the suture’s point. The collision detection is based on bounding volume hierarchy collision detection. A binary tree of hierarchy bounding spheres is created for the wound’s triangular surface mesh. To control the instrument’s collision detection, the bounding cylinders are used without hierarchy, since the number of bounding volumes for the Endo Stitch suturing instrument or other instruments are small (Figure 2a). The collision response of the wound with the suture and the instrument is based on penalty forces. A realistic behavior of the interactions among the simulated objects can be done by physically-based simulation. However, it requires more computation time which can cause the simulation to be non-interactive. To keep the simulation running at an interactive rate, we have added control-based interactions which have replaced some physically-based interactions. Each control-based interaction is a rule-based script which uses less computation time than a physically-based interaction, but with less flexibility and accuracy as a trade-off. For example, instead of allowing a suture’s point to slide either forward or backward based on the interaction forces acting on it, the suture’s point is allowed to slide only forward.

464

S. Punak and S. Kurenov / A Simulation Framework for Wound Closure by Suture

1.5. Physically-Based and Control-Based Interactions The simulation of connection between a punctured point and a point on the suture is based on attractive forces. Since the wound is a volume object and the suture is passed from an entrance punctured point to an exit punctured point, connections inside the wound model with the suture have to be added for more realistic simulation (Figure 2b). These points are computed by interpolating along the straight line formed by the entrance and exit punctured points on the wound’s surface created by the path of the Endo Stitch instrument’s needle. The simulation framework records and maintains each connection of a wound’s punctured point – either a vertex on the wound’s surface or a created vertex inside the wound’s FE mesh – and a suture’s point. The suture’s point is marked as ‘slidable’ if the point is allowed to break free from the connection by sliding of the suture forward or backward. At each connection, two attractive forces are created: a force acting on the suture’s point and a force acting on the wound’s punctured point. The attractive force applied to the wound’s punctured point has to be translated into multiple point forces applied on the nodes of the finite element that the punctured point is embedded in. Therefore, the puncture point may be out of the target location compared to the suture point. The simulation has to run iteratively to allow the attractive forces to bring the points to the same location. As mentioned earlier, the simulation framework will run slower if all interactions are physically-based. With physically-based interactions, increasing the accuracy means increasing computation complexity and time. The simulation is, therefore, scripted with control-based interactions to reduce time computations for physically-based interactions. To reduce the number of simulation loops for maintaining consistent locations of the connections of the suture’s point and the wound’s punctured point, the wound’s punctured point is explicitly set to the suture’s point after the last simulation loop iteration. The locations of these point-pair will be consistently the same, regardless of what the number of the simulation loops is, however, the computed locations may be far from the correct and valid locations. This is a speed-accuracy trade-off. In addition, since the simulated procedure for the wound closure by suture does not require the user to pull the suture backward, the simulation is set to avoid the suture’s point to slide backward, reducing the simulation complexity for the physically-based computation. The main contribution to reduce the computation time and complexity of the simulation is by enforcing the user to follow the simulated procedure for the wound closure by suture, e.g., by not allowing the user to puncture on the same side of the wound consecutively – the side has to be alternate, either left first then right, left, right, or vice versa. Clue points (Figure 2b) can be added to help guide the user to stay within the scripted simulation. This means we can set the developed simulation framework to work and look realistic only for the target procedure, requiring less physically-based interaction complexity to keep the simulation running at an interactive rate.

2. Results and Discussion The developed framework was tested on a computer running Windows XP 32-bit OS, with an Intel® Core™ i7-940 (2.93 GHz) CPU. The suture was simulated with 65 points. The simulated wound’s triangular surface mesh was composed of 2,178 vertices and 4,352 triangles. The wound’s linear hexahedral finite element mesh was composed

S. Punak and S. Kurenov / A Simulation Framework for Wound Closure by Suture

465

of 500 nodes and 324 hexahedra. This simulation uses a combination of physicallybased and control-based simulations in order to continue running at an interactive rate. With two instruments – an Endo Stitch suturing instrument and a grasper – the simulation ran at approximately 20 fps when there were no or minor intersections and at approximately 10 fps with complex collisions. The simulation results (Figure 3) demonstrate that the user can perform the wound closure by suture with the instruments in the virtual world simulated by the developed simulation framework.

Figure 3. The simulated wound, suture, and Endo Stitch suturing instrument in actions.

We are planning to incorporate the framework into a course for educating and training medical residents on how to use an Endo Stitch suturing instrument to close a wound or stitch tissues together. Varieties of wound shapes and suturing methods will be added into the framework. The code was written in C++ with object-oriented programming (OOP), so that the core code can be reused. OpenGL and GLSL APIs were used for the graphics and rendering. wxWidgets was used for creating graphics user interface (GUI). Subsequently, we would like to create the wound’s surface more realistic by applying graphics rendering techniques, for example, by adding textures and more complex rendering to it. The next major step would be to extend the developed simulation framework to support cutting of the suture and the deformable model, and adding special effects, such as blood and water.

References [1] Kurenov, Sergei, Sukitti Punak, Jörg Peters, Constance Lee, and Juan Cendan. "Development and Initial Validation of a Virtual Reality Haptically Augmented Surgical Knot-Tying Trainer for the Autosuture ENDOSTITCH Instrument." Studies in Health Technology and Infomatics: IOS Press, 2009. 145-147. [2] Kurenov, Sergei, Sukitti Punak, Minho Kim, Jörg Peters, and Juan Cendan. "Simulation for Training with the Autosuture Endo Stitch Device." Surgical Innovation 13, no. 4 (2006): 283-287. [3] Spillmann, Jonas, and Matthias Teschner. "CORDE: Cosserat rod elements for the dynamic simulation of one-dimensional elastic objects." SCA '07: Proceedings of the 2007 ACM SIGGRAPH/ Eurographics symposium on Computer animation. San Diego: Eurographics Association, 2007. 63-72. [4] Müller, Matthias, Matthias Teschner, and Markus Gross. "Physically-Based Simulation of Objects Represented by Surface Meshes." CGI '04: Proceedings of the Computer Graphics International. Washington: IEEE Computer Society, 2004. 26-33. [5] Blender. Blender. 2010. http://www.blender.org/ (accessed October 20th, 2010).

466

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-466

Simplified Cosserat Rod for Interactive Suture Modeling Sukitti PUNAK and Sergei KURENOV Roswell Park Cancer Institute

Abstract. This paper presents a real-time simulation of a virtual surgical suture, which is a physically-based model adapted from the Cosserat theory of elastic rods. The focus is on achieving a physically plausible simulation of the suture that can be simulated in real time. With simulation parameters adjustment, the virtual surgical suture can be accustomed to exhibit bending and twisting similar to a real suture. It is simple to implement and easy to extend for collision detections and interactions with other virtual objects. Its simulation is similar to a simulation of a composition of two mass-spring chains – for positions and orientations. Test results show that the virtual surgical suture can be used to tie knots in real time. Keywords. Cosserat rod, physically-based simulation, suture

Introduction Commonly the focus of a simulation of a model is to balance the computation cost on the accuracy and on the efficiency. Whether to model an elastic rod-like object, e.g. a surgical suture, accurately or efficiently depends on what kind of application the model will be used in. In this paper, our focus is on a physically plausible simulation of a virtual surgical suture based on the Cosserat theory of elastic rods. We try to achieve a simulation of a virtual suture that behaves similar to a real suture and can be deployed in complex real-time applications designed specifically for surgical simulation where each model computation time is crucial. We found that the CORDE model [1] can behave similar to a real suture and can be simulated in real time. In this paper we purpose to simplify the CORDE model to further reduce its computation time at the expense of a slight accuracy reduction. With a robust contact and self-contact handling scheme, our simplified surgical suture model still exhibits the bending, twisting, buckling, and looping phenomena that are similar to real sutures.

1. Background In continuum mechanics, a deformable object is considered to consist of material points. The movement of each material point of the deformable object is governed by its kinetic and potential energy. The change of each material point results in a deformation of the object. The dynamic motion and deformation of the object are due to a continuous change of each material point through the time. The computational model can be physically-based or geometrically-based.

S. Punak and S. Kurenov / Simpliﬁed Cosserat Rod for Interactive Suture Modeling

467

Examples of geometrically-based surgical suture simulations are [2, 3]. The suture model called “Follow the Leader” in [2] is a chain of extensible rigid links. Each link length is maintained by the distance between its two end points. The movement of each point is dictated by the movement of its neighbors. This model is simple but also limited. Bending, twisting, and torsion properties are not included in the model. These physical properties can be included in a suture model by employing the differential geometric of curves [3]. However, the torsion is not the material torsion of the suture, but it is the geometric torsion of a curve. Examples of physically-based surgical suture simulations are [4, 5, 6, 7]. Recently, the physically-based Cosserat theory of elastic rods has been used to simulate suture [4], hair [8], cables [9], guide wires [10], and rod-like objects [1]. Simple physically-based suture models can be simulated as a linear mass-spring model [5]. Bending, twisting, and torsion can be included by adding spring structure for the behavior into the system [7]. However, adjusting a mass-spring model to exhibit physical properties of a real suture is not intuitive. The Cosserat theory of elastic rods can be used to model both torsion and twisting deformation by using constraints [4]. However, the computation has to solve a boundary value problem. As pointed out by [1] and [9], solving the boundary value problem by shooting methods makes handling external forces such as contact forces more difficult, less robust, and less stable. This makes the simulation of a suture model [4] based directly on the Cosserat theory of elastic rod run slower and hardly reach an interactive rate. Instead both [1] and [9] discretized their elastic rod models into a chain of positions and a chain of orientations and enforce the coupling between the positions and orientations by penalty forces from the constraint energy due to the coupling. Thus, avoiding solving the boundary value problem and the models can be run at an interactive rate. An interesting elastic rod model based on Kirchhoff rods by [11] can be run at an interactive rate, but due to its complexity, its computation time is higher than of [1]. However, the simulation results of both models are very similarly comparable. In order to preserve computation time as much as possible, we have chosen to adapt the elastic rod simulation from [1] for our surgical suture simulation and simplified it further to reduce its computation time while keeping its simulation behavior virtually intact.

2. Methods The Cosserat theory of elastic rods can be used to model a deformable object that is long and thin in two of its three dimensions. The theory assumes the object length is significantly longer than its cross-section radius. Therefore, the object can be modeled by its centerline [9] (or Cosserat curve [12]) that continuously runs along the center of its cross-section. Here we provide just an overview of the theory which can be applied to model surgical suture. The prime ;< > ? denotes the spatial derivative @;A@> and the dot ;B > ? represents the temporal derivative @;A@? . The centerline, which provides the suture position in CD , is defined as

E> FG2 > G3 > G7 > H E> I JKLM N CD O

(1)

468

S. Punak and S. Kurenov / Simpliﬁed Cosserat Rod for Interactive Suture Modeling

(a)

(b)

Figure 1. (a) The configuration and (b) the discretization of the simulated suture.

To express the torsion of the suture, each centerline is attached with a right-handed orthonormal basis. This basis is called a directors frame [9], which represents the material directions (or Cosserat directors [12]), and is defined to be

P> FQ2 > Q3 > Q7 > H P> I JKLM N CD O

(2)

These directors PR > are the columns of a rotation matrix S> T CD . For a suture, the effect of shearing deformation can be negligible. The crosssection of the suture can be assumed to always be perpendicular to the tangent of the centerline [12]. Therefore, the basis of directors is adapted to the curve [9]. PD > is defined to align with the tangential of the centerline (Figure 1a). Hence, PD is parallel to E< and E< AUE< U PD . This constraint shows the mechanical coupling of the centerline to its orientation. The rate of change of the position along the centerline E< , indicates the stretch of the centerline E at > . It is a strain vector E< G G<W > G

. Since shearing is neglected, we have G is the temporal derivative of the centerline, EB > . The angular velocity, _R , of the directors around the \ -th axis is _R PR ` _Z with _Z a DRbV PR [ PBR . Equilibrium states of a suture are found by differentiating the energy function e with respect to the coordinates [1]. Here the potential energy c is the cG
S. Punak and S. Kurenov / Simpliﬁed Cosserat Rod for Interactive Suture Modeling

469

quaternion l4 is placed halfway between the two centerlines E4 and E4mV (Figure 1b). To simplify the computation, the mass n4 a+oG W p4qV rp4 is lumped at the centerline E4 , where +, G, and p4 are the material density, cross-section radius, and the rest length of link s of the suture, respectively. The model becomes a coupling of a chain of centerlines (i.e. point masses) and a chain of orientations (i.e. quaternions). By using calculus of variations with Lagrangian multipliers t T CD and u T C for the constraints, the Lagrangian equation of motion for the suture model [1] is - wx -v wyB :

d

wx wy:

r

wz y:

r

w{ yB :

rt`

w|} wy:

ru

w|~ wy:

V

Z Q>

(3)

where y 4 T G2 G3 G7 Z V W D are the coordinates and are external forces and torques. , c, , and 1 are the kinetic, potential, dissipation, and constraint energies of the suture, respectively. To reduce the computation for the simulation of the suture model, we ignore the kinetic energy terms in Eq. (3) and compute an approximation of the constraint energy t ` @ A@y 4 r u @ A@y 4 due to E< AUE< U d PD K as a potential penalty energy j and put it back into the equation to replace the constraint energy terms [1]. By not adding the constraint directly into the equation, the movements of the suture’s centerlines can be computed separately from the movements of its directors, i.e. the orientation of the centerlines [1]. It means the need for solving the boundary value problem is reduced into solving two separate sets of a chain of semi-rigid bodies – centerlines and orientations. We simplify this model further by ignoring the damping energy term. Instead the previous velocity of each centerline is damped by a velocity damping constant - before the numerical time integration. Our final simplified equation is wz y:

r

w y:

wz y:

r

wz y:

r

w y:

V

Z Q>.

(4)

In discretized form, the computation is separated into two computations; one for forces applied on the centerlines and the other one for torques applied on the orientations: f JsM r JsM JsM and ig JM r i JM i JM.

(5)

The stretch forces f and bending ig torques are computed from centerlines and orientations, respectively. The constraint forces and torques i are computed from both centerlines and orientations. The stretch cf , bending cg , and constraint j energies and their discretized versions are derived same as in [1]. 2.2. The Solver The chain of point masses E4 representing the centerlines and the chain of quaternions l representing the material frames are loosely coupled by penalty forces and torques that constraint their movements towards a valid configuration. Since, the dynamic movement of the point masses is virtually decoupled from the dynamic movement of the quaternions, we can numerically time-integrate the point masses as point masses in a mass-spring system, and the orientation of quaternions as orientation of rigid bodies

470

S. Punak and S. Kurenov / Simpliﬁed Cosserat Rod for Interactive Suture Modeling

[1]. A semi-implicit Euler scheme described in [1] is used for the numerical time integration.

3. Results We use C++ with Object-Oriented Programming and OpenGL to implement the simulation of the surgical suture. Since the suture model computation is similar to a computation of two mass-spring chains, we could use GPU for the computation. We tried using NVIDIA CUDA for the computation of the suture simulation loop on GPU. To run the simulation on GPU, the simulation data had to be copied from the CPU memory to the GPU memory. For detecting and resolving the suture’s self intersection (including knot recognition) and interactions with tools and other objects in the scene we had to copy the result data from the GPU back to the CPU. Table 1 shows the average computation time (in ms) for a simulation loop (including collision detection) of our suture implementations on CPU, CPU+GPU via CUDA, and CPU+GPU via CUDA with CUDA Page-Locked Host Memory (PLHM). All simulations were tested on a desktop computer running Windows XP 32-bit OS, which is equipped with an Intel® Core™ i7-940 (2.93 GHz) CPU, 3 GB of DDR3 SDRAM, and an NVIDIA GeForce GTX 285 graphics card. A PHANTOM Omni® haptic device is used to manipulate the laparoscopic surgical tool. The suture collision detection including self intersection and knot recognition is based on a sphere bounding volume hierarchy. To resolve a collision, a penalty force is applied to exactly move the collided suture’s link or point away from the collided object or the other suture’s link due to self intersection. The penalty force is calculated as the penetration distance multiplied by mass and divided by the square of time step, which is similar to an inverse of the position based dynamics [13]. The results show that for small number of links, less than 512 which are used in our case, the computation on the CPU is faster. Therefore, we only continue developing our suture simulation on CPU. In the future, we may use GPU to simulate the suture if we can move all computations related to the suture’s intra- and inter-actions onto GPU. Our simulation gives an improvement in performance compared to the CORDE model [1] (Table 2), where FC and IT stand for force computation and integration time, respectively. The ratios in Table 2 shows the relative computation time based on 50 links. The simulation results show that our model still exhibits bending and twisting similar to a real suture (Figure 2).

Table 1. Average computation time (in ms) for a simulation loop (with collision detection). #links 32 64 128 256 512 … 4096

CPU 0.18 0.31 0.65 1.28 2.51 … 20.28

CPU + GPU 2.13 1.38 2.59 4.71 5.73 … 17.12

CPU + GPU (PLHM) 1.97 2.53 3.11 4.87 7.22 … 48.75

S. Punak and S. Kurenov / Simpliﬁed Cosserat Rod for Interactive Suture Modeling

471

Table 2. Computation time (in ms) of the CORDE model and our suture model. #links 50 100 1000

CORDE FC + IT [1] Ratio 0.069 1.000 0.131 1.899 1.24 17.791

Our suture model FC + IT Ratio 0.0354 1.000 0.0594 1.678 0.5448 15.791

Table 3. The parameter values used for the simulations. Suture parameter length p (mm) diameter Q G [ ] (mm) mass n (g) material density + (g/m3) tensile modulus jf (GPa) bending modulus jg (kg/s2) shearing modulus k (GPa) penalty constraint stiffness (unitless) velocity damping constant - (%) time step (s)

Normal suture 100 0.5 +oG W = 10 12.73 10000 2500 200 100 50 0.001

Stiff suture 100 0.5 10 2000 10000 5000 5000 500 50 0.001

Figure 2. Tying a double knot.

To render the suture, a combined chain of point masses and orientations is created from the chain of point masses and the chain of orientations. The combined chain is then subdivided twice by the Chaikin’s Algorithm, similar to [7]. A generalized cylinder is generated and rendered for the subdivision chain. Figure 2 shows the simulations of tying a double knot on a fixed rigid object using a surgical instrument similar to the one used in [14]. Table 3 provides the parameters used for the simulations. It is noticeable that the values in the table are higher than in the real world. This is due to scaling adjustments in the virtual world. A stiff suture can be created by increasing the material density, moduli, and the penalty constraint stiffness. However, increasing the material density has to be compensated by reducing the time step. Therefore, we break the rule of physics by increasing the material density without increasing the mass, so that we do not have to reduce the time step, which keeps the influence of forces on the position of centerlines strong. The stiffness of the suture can be changed by setting the material density, since it directly influences the rotation of the orientations – the higher the material density, the lesser the rotation of the orientations.

472

S. Punak and S. Kurenov / Simpliﬁed Cosserat Rod for Interactive Suture Modeling

4. Conclusion We have created a virtual surgical suture model based on our simplification of the CORDE model, which is in turn based on the Cosserat theory of elastic rods. The purpose of the simplification is to make the model run as fast as possible while keeping the model accuracy as much as possible. We tested the model with simple knot tying tasks on a rigid object. The test results show that our simplified model still exhibits bending and twisting similar to a real suture. It uses less computation time compared to the original CORDE model. We are working on adding interactions of the surgical suture model with other virtual objects in an environment designed for real-time surgical simulation.

References [1] J. Spillmann and M. Teschner. CORDE: Cosserat Rod Elements for the Dynamic Simulation of OneDimensional Elastic Objects. In Proceedings of the 2007 ACM SIGGRAPH/ Eurographics symposium on Computer animation, pages 63–72, Aire-la-Ville, Switzerland, Switzerland, 2007. Eurographics Association. [2] J. Brown, J.-C. Latombe, and K. Montgomery. Real-Time Knot-Tying Simulation. In The Visual Computer, volume 20(2-3), pages 165–179. Springer, May 2004. [3] J. Lenoir, P. Meseure, L. Grisoni, and C. Chaillou. A Suture Model for Surgical Simulation. In S. C. et D. Metaxas, editor, 2nd Int. Symp. on Medical Simulation (ISMS), volume 3078, pages 105–113, Cambridge M.A., June 2004. Springer Verlag. [4] D. K. Pai. STRANDS: Interactive Simulation of Thin Solids Using Cosserat Models. Computer Graphics Forum, 21(3), 2002. [5] J. Phillips, A. Ladd, and L. E. Kavraki. Simulated Knot Tying. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 841–846, 2002. [6] M. LeDuc, S. Payandeh, and J. Dill. Toward Modeling of a Suturing Task. In Proceedings of the Graphics Interface, pages 273–279. CIPS, Canadian Human-Computer Commnication Society, A K Peters, June 2003. [7] B. Kubiak, N. Pietroni, F. Ganovelli, and M. Fratarcangeli. A Robust Method for Real-Time Thread Simulation. In Proceedings of the 2007 ACM symposium on Virtual reality software and technology, pages 85–88, New York, NY, 2007. ACM. [8] F. Bertails, B. Audoly, M.-P. Cani, B. Querleux, F. Leroy, and J.-L. Lévêque. Super-Helices for Predicting the Dynamics of Natural Hair. In SIGGRAPH ’06: ACM SIGGRAPH 2006 Papers, pages 1180–1187, New York, NY, 2006. ACM. [9] M. Grégoire and E. Schömer. Interactive Simulation of One-Dimensional Flexible Parts. ComputerAided Design, 39(8):694–707, 2007. [10] L. Duratti, F. Wang, E. Samur, and H. Bleuler. A Real-Time Simulator for Interventional Radiology. In Proceedings of the 2008 ACM symposium on Virtual reality software and technology, pages 105–108, New York, NY, 2008. ACM. [11] M. Bergou, M. Wardetzky, S. Robinson, B. Audoly, and E. Grinspun. Discrete Elastic Rods. In SIGGRAPH ’08: ACM SIGGRAPH 2008 Papers, pages 1–12, New York, NY, USA, 2008. ACM. [12] D. Cao, D. Liu, and C. H.-T.Wang. Three-Dimensional Nonlinear Dynamics of Slender Structures: Cosserat Rod Element Approach. International Journal of Solids and Structures, 43(3–4):760–783, 2006. [13] M. Müller, B. Heidelberger, M. Hennix, and J. Ratcliff. Position Based Dynamics. Journal of Visual Communication and Image Representation, 18(2):109–118, 2007. [14] S. N. Kurenov, S. Punak, M. Kim, J. Peters, and J. C. Cendan. Simulation for Training with the Autosuture Endo Stitch Device. Surgical Innovation, 13(4):1–5, December 2006.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-473

473

A Design for Simulating and Validating the Nuss Procedure for the Minimally Invasive Correction of Pectus Excavatum Krzysztof J. RECHOWICZ a,1 , Robert KELLY b Michael GORETSKY b Frazier W. FRANTZ b Stephen B. KNISLEY c Donald NUSS b and Frederic D. MCKENZIE a a

Modeling, Simulation, and Visualization Department, Old Dominion University b Pediatric Surgery, Children’s Hospital of The King’s Daughters c Mechanical Engineering Department, Old Dominion University Abstract. Surgical planners are used to achieve the optimal outcome for a surgery, especially in procedures where a positive aesthetic outcome is the primary goal, such as the Nuss procedure which is a minimally invasive surgery for correcting pectus excavatum (PE) - a congenital chest wall deformity. Although this procedure is routinely performed, the outcome depends mostly on the correct placement of the bar. It would be beneﬁcial if a surgeon had a chance to practice and review possible strategies for placement of the corrective bar and the associated appearance of the chest. Therefore, we propose a strategy for the development and validation of a Nuss procedure surgical trainer and planner. Keywords. pectus excavatum, surgical planner

Introduction Pectus excavatum (PE) is a congenital chest wall deformity which is typically characterized by a deep depression of the sternum. The minimally invasive technique for the repair of PE (the Nuss procedure) has been proven to have a high success rate and satisfactory aesthetic outcome [1]. Although this procedure is routinely performed, the outcome depends mostly on the correct placement of the bar. It would be beneﬁcial if a surgeon had a chance to practice and review possible strategies for placement of the corrective bar and the associated appearance of the chest. Therefore, we present the design of a Nuss procedure surgical planner and a strategy for its validation, taking into account the biomechanical properties of the PE ribcage, emerging trends in surgical planners, deformable models, and visualization techniques. 1 Corresponding Author: Krzysztof J. Rechowicz, Modeling, Simulation, and Visualization Department, Old Dominion University, Norfolk, VA 23529, USA; E-mail: [email protected].

474

K.J. Rechowicz et al. / A Design for Simulating and Validating the Nuss Procedure

Figure 1. The core design of the Nuss procedure.

1. Methods The core of our Nuss procedure surgical planner is based on a black-box approximation of a ﬁnite element model (FEM) of the PE ribcage (ﬁg. 1) in order to ensure real-time performance which is not possible to obtain directly [2,3]. It includes development of a parametric model of the ribcage that can be deformed (based upon individual patient parameters obtained from CT slices to ﬁt the PE ribcage. The core of the system is implemented in a virtual environment so deformation, triggered by the bar, can be visualized in the surgical planner. An average shape is being used for evaluation of the plan developed by the surgeon during training. This average has been developed based on a sample of normal subjects surface scans [4]. We leverage the core model design of the surgical planner to create a Nuss procedure surgical trainer with the interaction forces fed back to the user through a haptic interface. The system is meant to provide intelligent performance feedback based on predicted shape outcomes and comparisons to an averaged normal shape and to known successful post-surgical results for a speciﬁc case. The user would utilize this system to pick up a virtual scalpel, make incisions on a virtual PE chest, choose and insert a pectus bar into the PE chest, then receive a performance score. All is to be performed while receiving visual and touch feedback. Evaluation will be performed by experienced surgeons from the pectus clinic at the Children’s Hospital of the King’s Daughters, who regularly practice the Nuss procedure. Validation of the system will also be performed by testing the planner with previously operated cases. A user would recreate a scenario, i.e., the ribcage geometry and location of the bar, and compare a simulated outcome with the actual result. In this way, different cases can be studied in order to prove that the solution accomplishes its intended results.

2. Results We evaluated an average shape by comparing it with a normal chest without PE and post-operative chest shape. Based on differences between two shapes (presented as the

K.J. Rechowicz et al. / A Design for Simulating and Validating the Nuss Procedure

475

Figure 2. Comparison of the pre- and post-operative surface scans.

colormap), it is possible to quantify results. Differences between an average and normal chest shape were small up to 4 mm due to muscle structure which is not typically present in PE. Comparison with a post-operative shape showed overcorrection due to positive difference up to 30 mm. The same approach is used for evaluation of the outcome of the planning process. Additionally, we have already performed a comparison of three pairs of pre- and post-operative scans using a displacement map projected on the surface scan obtained prior to the surgery (ﬁg. 2). For validation purposes, a similar approach will be used to compare simulated shapes with actual outcomes.

3. Conclusions In this paper, the approach for developing a real-time Nuss procedure planner has been presented. The proposed solution will utilize patient speciﬁc data, incorporate the biomechanical properties of the PE ribcage, and provide information about a post-operative shape of the chest based on the position of the bar. In addition, we have presented the initial outcome of before and after surface scans analysis as a means to validate results.

References [1]

A.D. Protopapas and T. Athanasiou, Peri-operative data on the Nuss procedure in children with pectus excavatum: independent survey of the ﬁrst 20 years’ data, Journal of Cardiothoracic Surgery 40 (2008). [2] G. San Vicente, C. Buchartand, D. Borro, J. Celigueta, Maxillofacial surgery simulation using a massspring model derived from continuum and the scaled displacement method, International Journal of Computer Assisted Radiology and Surgery 4 (2009), 89–98. [3] U. Obaidellah, Z. Radzi, N.A. Yahya, N.A.A. Osman, A.F. Merican, The Facial Soft Tissue Simulation of Orthognathic Surgery Using Biomechanical Model, presented at 4th Kuala Lumpur International Conference on Biomedical Engineering 2008, Kuala Lumpur, Malaysia (2008). [4] K.J. Rechowicz, R. Kelly, M. Goretsky, F.W. Frantz, S. Knisley, D. Nuss, F.D. McKenzie, Development of an average chest shape for objective evaluation of the aesthetic outcome in the Nuss procedure planning process, in IFMBE Proceedings: 26th Southern Biomedical Engineering Conference SBEC 2010 32 (2010), 528–531.

476

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-476

AISLE: an Automatic Volumetric Segmentation Method for the Study of Lung Allometry Hongliang REN and Peter KAZANZIDES 1 Dept. of Computer Science, Johns Hopkins University, Baltimore, MD USA {hlren,pkaz}@jhu.edu Abstract. We developed a fully automatic segmentation method for volumetric CT (computer tomography) datasets to support construction of a statistical atlas for the study of allometric laws of the lung. The proposed segmentation method, AISLE (Automated ITK-Snap based on Level-set), is based on the level-set implementation from an existing semi-automatic segmentation program, ITK-Snap. AISLE can segment the lung ﬁeld without human interaction and provide intermediate graphical results as desired. The preliminary experimental results show that the proposed method can achieve accurate segmentation, in terms of volumetric overlap metric, by comparing with the ground-truth segmentation performed by a radiologist. Keywords. AISLE, Automatic volumetric segmentation, Lung allometry

Introduction In order to develop a statistical atlas and study allometric laws of the lung, we need to extract the anatomical structures of the lung ﬁeld from a population of volumetric CT datasets. Therefore, it is highly desirable to have an accurate segmentation method with minimum human intervention. There are quite a few semi-automatic segmentation algorithms and software toolkits available, such as Analyze [3], MIPAV [2], ITK [4] and Osirix [5]. However, for a large-scale population study, it is still a labor-intensive process for most of the semi-automatic methods. Therefore, it is desirable to have a fully automatic pipeline to get the whole segmentation done without human intervention.

1. Methods We developed a fully automatic segmentation method based on the 3D activecontour-evolution algorithms [1] of ITK-snap [6]. The active contour refers to a 1 Corresponding Author: Peter Kazanzides, Department of Computer Science, Johns Hopkins University.

H. Ren and P. Kazanzides / AISLE: An Automatic Volumetric Segmentation Method

477

Figure 1. Two key steps (located at the purple circles) in the intensity feature based active-contour segmentation: thresholding and bubble placement.

Figure 2. Histogram plots of the selected slices from diﬀerent study sequences. The upper row is the original slice and the lower row is the corresponding histogram plots, where the deep dark histogram is the log-scale of the grey histogram.

closed contour that can evolve with time and space, and is driven by internal contour geometry and external force from the feature images. Hence, there are two key steps that need to be automated herein: creating binary intensity images after thresholding and initial contour (aka. bubble in ITK-snap) placement, as shown in Figure 1. The automatic thresholding step can be realized by utilizing a histogram-based Otsu ﬁlter, as it is good at identifying the threshold in contrast between lung ﬁeld and background from the statistical histogram in Figure 2. The initial bubble (contour) placement is done by calculating the geometric center of the lobes from a connected-component ﬁlter. In practice, we only need to place one bubble for one lobe. The radius of the bubble is arbitrary as long as it does not fall across the other lobe. The whole pipeline is shown in Figure 3.

Figure 3. The pipeline for the AISLE algorithm

478

H. Ren and P. Kazanzides / AISLE: An Automatic Volumetric Segmentation Method

Figure 4. The DICE metric between the AISLE segmentation and manual segmentation

2. Results We have run 13 fully-automatic segmentations, and the results were validated by comparing with the manual segmentation performed by the radiologist. The DICE volumetric overlap metric is used to evaluate the segmentation performance, and the results in Figure 4 demonstrate the proposed algorithm is capable of achieving very accurate segmentation (average DICE >0.99 ).

3. Conclusions The proposed AISLE segmentation method can achieve decent results without human interaction for lung ﬁeld extraction from volumetric CT datasets. The developed method is deployable in both Windows and Linux operating systems, and has an additional feature for showing the intermediate graphical results. The next step is to realize the edge-feature based automatic segmentation for the bronchial tree from thoracic images.

References [1] [2] [3] [4] [5] [6]

Vicent Caselles, Ron Kimmel, and Guillermo Sapiro. Geodesic active contours. International Journal of Computer Vision, 22(1):61–79, February 1997. http://mipav.cit.nih.gov/. http://www.analyzedirect.com/. http://www.itk.org/. http://www.osirix viewer.com/. P. A. Yushkevich, J. Piven, H. Cody, S. Ho, J. C. Gee, and G. Gerig. User-guided level set segmentation of anatomical structures with ITK-SNAP. Insight Jounral, 1, 2005. Special Issue on ISC/NA-MIC/MICCAI Workshop on Open-Source Software.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-479

479

Development of a Wireless Hybrid Navigation System for Laparoscopic Surgery Hongliang REN a , Denis RANK b , Martin MERDES b , Jan STALLKAMP and Peter KAZANZIDES a,1

b

a

Dept. of Computer Science, Johns Hopkins University, Baltimore, MD USA {hlren,pkaz}@jhu.edu b Fraunhofer Institute for Manufacturing Engineering and Automation, Stuttgart, Germany {rank,martin,stallkamp}@ipa.fhg.de Abstract. Navigation devices have been essential components for ImageGuided Surgeries (IGS) including laparoscopic surgery. We propose a wireless hybrid navigation device that integrates miniature inertial sensors and electromagnetic sensing units, for tracking instruments both inside and outside the human-body. The proposed system is free of the constraints of line-of-sight or entangling sensor wires. The main functional (sensor) part of the hybrid tracker is only about 15 mm by 15 mm. We identify the sensor models and develop sensor fusion algorithms for the proposed system to get optimal estimation of position and orientation (pose). The proof-of-concept experimental results show that the proposed hardware and software system can meet the deﬁned tracking requirements, in terms of tracking accuracy, latency and robustness to environmental interferences. Keywords. Image guided surgery, Surgical navigation, Electromagnetic tracking, Inertial measurement unit, Sensor fusion

Introduction Real-time tracking of surgical instruments inside the human body poses unique challenges in developing tracking devices for minimally invasive surgeries. Optical tracking (OPT), the gold standard for surgical navigation, is bulky and blind when its line-of-sight (LOS) between the cameras and the markers is occluded [7]. Electromagnetic Tracking (EMT) [4] is feasible for laparoscopic surgery but notorious for its susceptibility to surrounding metallic or conductive surgical tools [6] and its reliance on a wired connection to the markers (coils). In addition, both of them have a limited working volume and OPT further has the restriction of angle of view relative to the optical camera. Ultrasound based [9,5] navigation or 1 Corresponding Author: Peter Kazanzides, Department of Computer Science, Johns Hopkins University.

480

H. Ren et al. / Development of a Wireless Hybrid Navigation System for Laparoscopic Surgery

mechanical tracking [1] usually have handling inconveniences. Some prior work, such as [2][7], combined information from an optical tracker and EM tracker, which makes the operating room even more crowded with two bulky tracking systems, and can still suﬀer from the constraint of line-of-sight.

1. Objective Our objective is to develop a miniature tracking system that is easy-to-use, free of line-of-sight, angle-of-view, or cabling constraints, and with a reasonable working volume, without aﬀecting surgical workﬂow. The desired technical speciﬁcation is to achieve 6DOF target tracking accuracy of about 1 mm in position and 1 degree in orientation, with maximum latency of 100 milliseconds, minimum update frequency of 30 Hz, and with robustness against interference due to metallic objects, electrocautery, etc. Towards this end, we combine inexpensive miniature MEMS inertial sensors to compensate the distortions of the EM tracker, and to improve the dynamic behavior of the tracking system.

2. Material We employ a self-contained Inertial Measurement Unit (IMU), including accelerometer, gyroscope and magnetometer, to provide highly dynamic measurements with respect to global coordinates. For example, the accelerometer and magnetometer together can provide roll, pitch and heading measurements. The miniature electromagnetic sensor is used to provide an external reference to compensate the drift of the IMU. The prototype hardware is shown in Figure 1. It consists of two electromagnetic tracking systems (the commercial Aurora system from NDI [3] and a custom EMT system) and a sensor PCB which has integrated inertial sensors and electronics for signal processing. The commercial Aurora EMT system is included for the purpose of both validation and comparison. The rationale for developing a custom EM system is mainly due to the need for time synchronization and wireless operation. The custom EMT consists of a ﬁeld generator and up to three receiving coils on the sensor PCB, which is collocated with the inertial sensors. Communication between the sensor PCB and a PC is via Bluetooth (for wireless operation) or USB (for debugging or ﬁrmware updates). Note that in this paper, the sensor fusion experiments used the commercial NDI Aurora system, as we are still working on the custom EMT to get comparable performance. The three axis accelerometer, used for detecting movements in the x, y and z directions, is the ST331DLH from STMicroelectronics and its measurement range is set to ±2g with an internal cutoﬀ frequency of 780Hz. Two gyroscope sensors, the two-axis IDG300 and the single-axis ISZ300 from InvenSense, are used to measure the 3-axis angular rate. A three-axis AMR magnetometer, the Honeywell HMC1043 with sensitivity of each axis about 0.3mV/μT, serves as an electronic compass.

H. Ren et al. / Development of a Wireless Hybrid Navigation System for Laparoscopic Surgery

481

Figure 1. The hybrid tracking system consists of a hybrid tracker inside the handle of an endoscope (left), and an external ﬁeld generator (right). The hybrid tracker is composed of inertial sensors, electromagnetic coils and other supporting electronic components. The ﬁeld generator is a coil array with 4x4x3 transmitters.

3. Methods Because the raw measurements are from two diﬀerent sensor-coordinate systems (EMT and IMU), it is necessary to register these coordinate systems before performing sensor fusion. The coordinate registration includes a body-frame registration solved by an AX=XB formulation [8], and base-frame registration solved by a paired-orientation formulation [11]. Figure 2 deﬁnes the coordinate systems and illustrates the two unknown transformations, X and Y. Fcoil Ffg X Field Generator

Fimu Y

Fnav : Navigation frame (North East Down)

Figure 2. Deﬁnition of the coordinate systems. X and Y are the two unknown transformations, corresponding to body-frame and base-frame transformations, respectively.

The two streams of registered measurements are subsequently fed to a sensor fusion module, which consists of an orientation estimator (rot) and a position estimator (pos), both based on Kalman ﬁlters, as shown in the block diagram of Figure 3. For the orientation estimation in Figure 3, the measurements are obtained from the IMU and EMT, as both subsystems can provide orientation measurements. We are weighting them based on the acceleration of the tracker: when the acceleration is small, the measurements from the IMU are more accurate; otherwise, the measurements from the EMT are more accurate.

482

H. Ren et al. / Development of a Wireless Hybrid Navigation System for Laparoscopic Surgery fg

x

fg

q

Coil(s)

nav

Accel (x3) Magnetic (x3)

Gyro ( 3) (x3)

nav

nav

x g

q

q

Kalman Kalman filter (pos) fgR

~ x

fg

~ x

fg

q~

fg

~ q

nav

Attitude Meas. fgR nav

fggR

fg

fg

qˆ

Kalman filter (rot)

nav

Figure 3. Block diagram of the sensor fusion algorithm; x is position measurement, q is orientation measurement.

The orientation dynamics model is derived in time-derivative quaternion formulation as: q˙ =

1 [ω×] · q, 2

(1)

where q is the quaternion representation of the orientation, and [ω×] is a skew symmetric matrix of the angular velocity, ω, acquired from the gyroscope. We also include the gyroscope bias in the system dynamics, by assuming it is a random walk process. For position estimation, we are using the external position measurements from EMT as the reference and the dynamic model is derived from the kinematic relationship between position, velocity and acceleration, given by, x ¨nav = R · a − R · ω × x˙ nav − g,

(2)

where R is the rotation matrix from IMU frame to navigation frame, xnav , x˙ nav and x ¨nav are the position, velocity and acceleration vectors in the navigation frame, a is the acceleration measured by the accelerometer, ω is the angular velocity measured by the gyroscope, and g is the gravity vector.

4. Results We conducted a series of experiments to validate the proposed hybrid tracking system (HYB). First, we compared the orientation estimate between the hybrid tracker and the commercial Aurora EM tracker. Note that we are using the orientation estimates from the commercial NDI Polaris optical tracker (OPT) as the benchmark. Figure 4 shows the diﬀerence in the orientation estimate with respect to the optical tracker (i.e., HYB-OPT vs. EMT-OPT). The overall root-meansquare (RMS) tracking errors of the HYB for the 2 runs were 0.9 degrees, 0.8 degrees, 1.0 degrees, for roll, pitch and yaw, respectively. For EMT, the overall

H. Ren et al. / Development of a Wireless Hybrid Navigation System for Laparoscopic Surgery

483

Estimate difference wrt OPT: HYB vs. EMT

4

HYB EMT

3 2

degrees

1 0 -1 -2 -3 -4 roll

pitch

yaw

roll

pitch

yaw

runs

Figure 4. Measured orientation errors for proposed hybrid tracker and EMT, using optical tracking as ground truth.

RMS tracking errors were 1.9 degrees, 2.1 degrees, 2.1 degrees, for roll, pitch and yaw, respectively. We also compared the tracking performance in an earlier experiment [10] between HYB and EMT when the same metallic tool was moved around the tracker. The hybrid tracking method demonstrated its resistance to the environmental interference. Thus, the orientation estimation shows superior performance in terms of accuracy and robustness to metallic disturbance, compared to just the use of EM tracking. A trajectory tracking experiment is shown in Figure 5. The dynamic tracking performance of the hybrid tracker is better than just using the external EMT reference. Note that the hybrid system can obtain position measurements even when the EMT signals are missing for a short duration. In order to present the tracking information graphically, we implemented an OpenIGTLink [12] interface to enable rapid integration with IGT platforms such as 3D Slicer, as shown in Figure 6.

5. Conclusions & Outlook The proposed electromagnetic aided inertial navigation system demonstrated improved tracking performance in terms of tracking accuracy, data update rate, and tracking robustness. The integration of inertial sensing with an external reference, such as electromagnetic tracking, provides a promising solution for tracking surgical instruments during laparoscopic surgery. The external reference tracking system can provide

484

H. Ren et al. / Development of a Wireless Hybrid Navigation System for Laparoscopic Surgery HYB EMT

160 140 120

Y(mm)

100 80 60 40 20 0 −350

−300

−250 X(mm)

−200

Figure 5. Trajectory of position estimation from HYB (blue line) and EMT (red line); HYB demonstrated better dynamic tracking performance.

Figure 6. Graphical representation of the tracking results in 3D Slicer through OpenIGTLink interface

stable correction for inertial sensor drifts and, in turn, the inertial sensor can provide better dynamic tracking results. A important future work is to validate the custom EMT system, including the calibration, localization and integration of the EMT for sensor fusion.

Acknowledgment This project is a joint development between The Johns Hopkins University (JHU) and the Fraunhofer Institute for Manufacturing Engineering and Automation (IPA) and is supported by internal funds from both institutions. The authors gratefully acknowledge the contributions of our colleagues Russell Taylor, Elliot McVeigh, Iulian Iordachita, and Anton Deguet, at JHU.

H. Ren et al. / Development of a Wireless Hybrid Navigation System for Laparoscopic Surgery

485

References [1]

[2]

[3] [4] [5]

[6]

[7]

[8] [9]

[10]

[11]

[12]

J. Bax, D. Cool, L. Gardi, K. Knight, D. Smith, J. Montreuil, S. Sherebrin, C. Romagnoli, and A. Fenster. Mechanically assisted 3D ultrasound guided prostate biopsy system. Medical Physics, 35(12):5397–5410, 2008. W. Birkfellner, F. Watzinger, F. Wanschitz, R. Ewers, and H. Bergmann. Calibration of tracking systems in a surgical environment. IEEE Trans. on Medical Imaging, 17(5):737– 742, Oct. 1998. http://www.ndigital.com/medical. V. Kindratenko. A survey of electromagnetic position tracker calibration techniques. Virtual Reality, 5(3):169–182, Sept. 2000. H.-H. Lin, C.-C. Tsai, and J.-C. Hsu. Ultrasonic localization and pose tracking of an autonomous mobile robot via fuzzy adaptive extended information ﬁltering. IEEE Trans. on Instrumentation and Measurement, 57(9):2024–2034, Sept. 2008. C. Naﬁs, V. Jensen, and R. von Jako. Method for evaluating compatibility of commercial electromagnetic (EM) microsensor tracking systems with surgical and imaging tables. In SPIE Medical Imaging 2008: Visualization, Image-guided Procedures, and Modeling, volume 6918, pages 691820,1–15. SPIE, 2008. M. Nakamoto, Y. Sato, M. Miyamoto, Y. Nakamjima, K. Konishi, M. Shimada, M. Hashizume, and S. Tamura. 3D ultrasound system using a magneto-optic hybrid tracker for augmented reality visualization in laparoscopic liver surgery. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), pages 148–155, London, UK, 2002. F. Park and B. Martin. Robot sensor calibration: solving AX=XB on the Euclidean group. IEEE Trans. on Robotics and Automation, 10(5):717–721, Oct 1994. J. F. Quinlan, H. Mullett, R. Stapleton, D. FitzPatrick, and D. McCormack. The use of the Zebris motion analysis system for measuring cervical spine movements in vivo. Proceedings of the Institution of Mechanical Engineers – Part H – Journal of Engineering in Medicine, 220(8):889 – 896, 2006. H. Ren and P. Kazanzides. Hybrid attitude estimation for laparoscopic surgical tools: A preliminary study. EMBC 2009. IEEE International Conference on EMBS, pages 5583– 5586, 2009. H. Ren and P. Kazanzides. A paired-orientation alignment problem in a hybrid tracking system for computer assisted surgery. Journal of Intelligent and Robotic Systems, accepted, 2010. J. Tokuda, G. S. Fischer, X. Papademetris, Z. Yaniv, L. Ibanez, P. Cheng, H. Liu, J. Blevins, J. Arata, A. J. Golby, T. Kapur, S. Pieper, E. C. Burdette, G. Fichtinger, C. M. Tempany, and N. Hata. OpenIGTLink: an open network protocol for image-guided therapy environment. The International Journal of Medical Robotics and Computer Assisted Surgery, 5(4):423–434, 2009.

486

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-486

Visualization of Probabilistic Fiber Tracts in Virtual Reality Tobias RICK

Anette VON KAPRI a,b Svenja CASPERS c Katrin AMUNTS c,b,d Karl ZILLES c,b,e Torsten KUHLEN a,b a Virtual Reality Group – RWTH Aachen University b Jülich-Aachen Research Alliance (JARA-HPC, JARA-BRAIN) c Institute of Neuroscience and Medicine, INM-1, INM-2, Research Center Jülich d Department of Psychiatry and Psychotherapy, RWTH Aachen University e C. and O. Vogt Institute for Brain Research, Heinrich-Heine-University Düsseldorf a,b,1

Abstract. Understanding the connectivity structure of the human brain is a fundamental prerequisite for the treatment of psychiatric or neurological diseases. Probabilistic tractography has become an established method to account for the inherent uncertainties of the actual course of ﬁber bundles in magnetic resonance imaging data. This paper presents a visualization system that addresses the assessment of ﬁber probabilities in relation to anatomical landmarks. We employ real-time transparent rendering strategy to display ﬁber tracts within their structural context in a virtual environment. Thereby, we not only emphasize spatial patterns but furthermore allow an interactive control over the amount of visible anatomical information. Keywords. probabilistic tractography, virtual reality

Introduction Neuroscientiﬁc research aims at understanding the structure-function relationship in the brain. Networks of communicating brain areas are required to fulﬁl motor, sensory as well as all mental and cognitive activities. The structural basis of such networks are nerve ﬁbers connecting the participating brain areas. A profound knowledge about this connectivity structure is therefore necessary for understanding the computational activity of the brain. The mapping of nerve ﬁbers and ﬁber bundles in the brain is also required to further understand psychiatric or neurological diseases. Currently, diffusion tensor magnetic resonance imaging (DT-MRI) provides the most forward method for the assessment of white matter ﬁber tracts in the living human brain. Hereby, the course of the ﬁbers is estimated by measuring water diffusion in the brain. Based on their Brownian motion, water molecules prefer to move along directions with lowest resistance which in the brain is provided along the myelin sheaths. By applying magnetic ﬁeld gradients from different spatial directions, the uncertainty within the diffusion data can be estimated and used for consecutive analysis. From these DT-MRI 1 Corresponding

Author: [email protected]

T. Rick et al. / Visualization of Probabilistic Fiber Tracts in Virtual Reality

487

Figure 1. Solid surface representation of a ﬁber pathway, image created with FSL [4] (left). Our volumetric rendering shows the probability distribution within the ﬁber tract (right).

data an effective diffusion tensor can be estimated within each voxel. The quantities as mean diffusivity, principal diffusion direction and anisotropy of the diffusion ellipsoid can be computed from the elements of the diffusion tensor [1]. To reconstruct ﬁber pathways based on the diffusion data, two main methods are currently used: (1) deterministic tractography, and (2) probabilistic tractography. Deterministic tractography tries to ﬁnd the path from a seed to a target voxel based on the main diffusion direction within each voxel on the way. Hereby, uncertainty within the course of the ﬁber pathway cannot reliably be accounted for. In contrast, probabilistic tractography explicitly accounts for the uncertainty of the actual ﬁber tracts. For each voxel, a local probability distribution of the diffusion direction is calculated. A probabilistic tractography algorithm then tries to ﬁnd the most probable course of a ﬁber between a seed and a target voxel by deciding in each voxel which would be the most probable prosecution of the ﬁber, based on the local probability distribution and its prior course [2]. As a result of probabilistic tractography, no single ﬁber strand is provided, but a probability distribution of possible ﬁber pathways between seed and target voxels, ranging from voxels with a large number of passed traces to voxels with only a low number of passes. The visualization of the probability in three dimensions (3D) is an essential step for the registration of the most likely course of a ﬁber bundle. Furthermore, anatomical information is required to reveal the ﬁber in its structural context. In this paper we extent the ideas from [3] and address both the visualization of probabilistic ﬁber tracts with a special focus on how to provide the required degree of anatomical context. We embed our visualization system in a virtual environment which not only improves depth perception due to stereoscopic projections but enables the use of direct interaction techniques such that the user becomes an integral part of the visualization pipeline. We use direct volume rendering for structural as well as ﬁber information in order to provide semi-transparent renderings in real-time. The amount of visible anatomical context can be controlled by a so-called magic lens interaction metaphor which we will refer to as virtual ﬂashlight. The remainder of this paper is structured as follows. After brieﬂy reviewing previous work in Section 1, we will describe our visualization and interaction approach in Section 2. We present the results in Section 3 and conclude our work in Section 4.

488

T. Rick et al. / Visualization of Probabilistic Fiber Tracts in Virtual Reality

1. Related Work Deterministic streamline tractography emphasizes the course of the neuronal ﬁbers using the principal eigenvector of the diffusion tensor [5]. To visualize the 3D large scale structure, Kindlmann [6] applied direct volume rendering strategies to the anisotropy values to map from the diffusion tensor data to color and opacity. Other common visualizations often make use of glyph-based techniques that represent a single tensor as a geometric primitive or via streamline advection among the principal eigenvector of the local tensor. Chen at al. [7] for example, merge ellipsoids to show the connectivity information in the underlying anatomy while characterizing the local tensor in detail. Sherbondy et al. [8] implemented interaction techniques to place and manipulate regions to selectively display deterministic ﬁber tracts that pass through speciﬁc anatomical areas. However, due to the relatively low resolution of DTI data as compared to the diameter of an axon, only the main ﬁber direction within each voxel is accounted for. Therefore, a main methodological issue are crossing ﬁbers. Qazi et al. [9] successfully trace through regions of crossing ﬁbers deterministically by extracting two tensors at any arbitrary position. Nevertheless, streamline methods only represent a single ﬁber path between two points without indication of correctness. Current probabilistic tractography algorithms [2] model different courses of ﬁbers within each voxel using priors about the previous course of the estimated ﬁber tract and anatomical plausibility assumptions [10], thereby addressing the issue of crossing ﬁbers adequately. Conveying uncertainty in the rendering is an inherent requirement for neuroscientists to evaluate probabilistic tractographies. The high interest in uncertainty and ﬁber crossing is shown in the recent work of Descoteaux et al.[11]. Deterministic and probabilistic tractography are compared with respect to crossing and splitting ﬁber bundles. In most current visualizations uncertainty is only represented on two-dimensional (2D) slices. 3D representations of probabilistic ﬁber tracts are often generated by extracting opaque isosurfaces for certain probability ranges (see Figure 1 left). In addition to visualization, the exploration of data also requires interactive manipulation. Therefore, we introduce the so-called magic lens interaction metaphor. It was ﬁrst discussed by Bier et. al [12] as a 2D see-through user interface that changes the representation of content in a special window. A popular example is for instance a magnifying glass. In [13], Viega et. al extend the concept of magic lenses to virtual environments. They present the implementation of volumetric lenses that uses hardware clipping of geometric primitives to reveal the inner structure of objects. Whereas in [14] a magic box is used to present a higher-resolution of a ﬂow visualization in order to focus attention on these regions and investigate them in more detail. 2. Method A major issue of current 3D visualization techniques in common DTI analysis tools is that no indication of uncertainty in the ﬁber tracts is contained in the ﬁnal renderings. For instance in Figure 1 (left), the rendering of tracts is achieved by extracting an isosurface from the ﬁber tract but with no further clues to anatomical details or probability distribution within the ﬁber tract. However, anatomical context information is crucial for the registration of the most likely course of a ﬁber pathway in relation to structural landmarks.

T. Rick et al. / Visualization of Probabilistic Fiber Tracts in Virtual Reality

489

Figure 2. Three-dimensional brain with ﬁber bundle (left). Brain areas provide additional anatomical context (right).

Figure 3. A user deﬁned clipping region relates the ﬁber pathway with structural information.

2.1. Requirements To overcome such shortcomings, we formulate four conceptual requirements of our visualization system based on an interdisciplinary discussion with DTI domain experts as follows: (1) The visualization should emphasize spatial patterns and present the threedimensional physical structure in an intuitive fashion. (2) The ﬁnal rendering should convey the uncertainty within each ﬁber tract. (3) The location of ﬁber tracts within the human brain should easily be deduced by the anatomical context. (4) None of the above requirements must interfere with the interactivity of the visualization system. 2.2. Visualization Technique We employ a direct volume rendering as the underlying rendering technique for our visualization system. However, our visualization system requires the display of multiple and transparent volumetric (voxel-based) information simultaneously. Here, state-of-the-art techniques for direct volume rendering are no longer sufﬁcient for an interactive visualization in a virtual environment. The main reason for this is that the process of rendering transparent objects, which usually relies on either depth sorting or ray casting, is a complex process in general and becomes even more demanding the more objects are involved. However, we can exploit the fact, that most medical datasets are already registered in a common reference space. Therefore, we adapt classical slice-based volume rendering to efﬁciently handle multiple co-registered data sets as follows: First, we interleave the data

490

T. Rick et al. / Visualization of Probabilistic Fiber Tracts in Virtual Reality

Figure 4. Users in an immersive CAVE virtual environment (left). A spatial input device allows interaction directly in 3D (right).

sets into one vector-valued data ﬁeld. The proxy geometry (texture slices) is setup such that it represents the shared reference space and is rendered only once which alleviates the problems of depth-sorting multiple proxy geometries. Then, a special shader program handles each integration step of the individual data sets, separately. In a subsequent step the temporary integration values are interpolated according to user setting (e.g. maximum intensity or weighted sum). Hence, ﬁber probability and structural information can be classiﬁed according to separate transfer functions but form a consistent and correctly depth-sorted transparent image (cf. Figure 1 right). 2.3. Anatomical Context The anatomical context is provided by including a standardized reference brain (Figure 1). As illustrated in Figure 2, the opaque rendering of cross sections of the brain resembles the 2D slices the domain expert are familiar with from common DTI tools and provides an unbiased view on the original data. The rendering of functional or cortical deﬁned brain areas is used to give additional clues to the anatomical connection of ﬁber tracts. Additionally, the reference brain is volume rendered semi-transparently with userdeﬁned clipping regions in order to reduce visual cluttering. The clipping regions can either be simple axis-aligned planes or can directly be controlled by the user via a virtual clipping cone (virtual ﬂashlight) as depicted in Figure 3. 2.4. Emphasizing Spatial Patterns We use the virtual reality toolkit ViSTA [15] as basis for our implementation. This allows the deployment of our visualization system on common desktop computers as well as on immersive virtual environments (cf. Figure 4 left). Depending on the available hardware infrastructure, this also allows the combination of 3D rendering with user-centered projection (head-tracking) which increases the overall depth perception, signiﬁcantly. In addition to stereoscopic vision, direct interaction where the user takes an active role is an integral part of every interactive virtual reality system. We have incorporated a direct interaction metaphor into our visualization system, the virtual ﬂashlight. Similar to the beam of a ﬂashlight, the user can directly control the amount of visible anatomical structure by a 3D interaction device (cf. Figure 4 right). Interesting parts of the probabilistic ﬁber tracts can be revealed and referenced with the anatomical landmarks with reduced occlusion or visual clutter. This allows a more accurate inspection of the anatomic structure in the direct vicinity of ﬁber pathways. The concept is illustrated by the image series in Figure 5.

T. Rick et al. / Visualization of Probabilistic Fiber Tracts in Virtual Reality

491

Figure 5. The user can control the clipping region with a virtual ﬂashlight in the CAVE virtual environment.

3. Results The data used for all visualizations were obtained in the Institute of Neuroscience and Medicine of the Research Centre Jülich and the C. and O. Vogt Institute for Brain Research of the Heinrich-Heine-University Düsseldorf. The brain areas shown here were depicted from the Jülich-Düsseldorf cytoarchitectonic atlas [16]. All data were displayed on the standard reference brain of the Montreal Neurological Institute (MNI) as internationally used as common reference space (voxel resolution: 1mm3 ). Domain experts state that by combining anatomical information from the reference brain with overlaying ﬁber tracking results, the visualization gives ﬁrst hints to the anatomical context of the ﬁber tracts. Former visualization software most widely used in DTI tractography research only reconstructed ﬁber tracts in 3D as solid paths without any information about the uncertainty. Therefore, the coding of different probability values with different colors and transparencies allows a 3D impression of the ﬁber tract while still revealing its main direction and the uncertainty around it. Furthermore, the new visualization method allows interactive manipulation of the magnitude of anatomical information displayed which was hardly possible in former software packages. 4. Conclusion Our work addresses the visualization of probabilistic ﬁber tracts in the human brain. Here, the comprehension of the course of the ﬁber in relation to its conﬁdence is one of the most crucial steps. The interactive 3D visualization of probabilistic ﬁber tracts referenced with their anatomical landmarks allows the domain scientists to directly interpret their results in 3D. Hereby, reducing the additional mental workload previously required from judging 2D slices or missing uncertainty information in non-interactive 3D plots. We have embedded our visualization in a virtual reality application which increases the depth perception of structural patterns and enables direct interaction metaphors due

492

T. Rick et al. / Visualization of Probabilistic Fiber Tracts in Virtual Reality

to tracking of 3D input devices. Furthermore, the degree of anatomical information necessary in order to establish a relationship between nerve ﬁbers and structural landmarks can be controlled by the virtual ﬂashlight metaphor. Here, the user is provided with a ﬁne-grain control which parts of the structural information is cut away while the ﬁber tracts remain visible in the cone of the virtual ﬂashlight.

References [1] [2]

[3] [4] [5] [6] [7] [8] [9]

[10] [11] [12] [13] [14] [15] [16]

P. J. Basser, J. Mattiello, and D. Lebihan, “MR diffusion tensor spectroscopy and imaging,” Biophysical Journal, vol. 66, pp. 259–267, 1994. T. Behrens, H. Johansen-Berg, M. Woolrich, S. Smith, C. Wheeler-Kingshott, P. Boulby, G. Barker, E. Sillery, K. Sheehan, O. Ciccarelli, A. Thompson, J. Brady, and P. Matthews, “Non-invasive mapping of connections between human thalamus and cortex using diffusion imaging,” Nature Neuroscience, vol. 6, no. 7, pp. 750–757, 2003. [Online]. Available: http://dx.doi.org/10.1038/nn1075 A. von Kapri, T. Rick, S. Caspers, S. B. Eickhoff, K. Zilles, and T. Kuhlen, “Evaluating a visualization of uncertainty in probabilistic tractography,” K. H. Wong and M. I. Miga, Eds., vol. 7625, no. 1. SPIE, 2010, p. 762534. “FSL 4.1,” August 2008. [Online]. Available: http://www.fmrib.ox.ac.uk/fsl/ G. Kindlmann, “Visualization and analysis of diffusion tensor ﬁelds,” Ph.D. dissertation, School of Computing, University of Utah, 2004. G. Kindlmann, D. Weinstein, and D. Hart, “Strategies for direct volume rendering of diffusion tensor ﬁelds,” IEEE Transactions on Visualization and Computer Graphics, vol. 6, no. 2, pp. 124–138, 2000. W. Chen, S. Zhang, S. Correia, and D. F. Tate, “Visualizing diffusion tensor imaging data with merging ellipsoids,” IEEE Paciﬁc Visualization Symposium, vol. 0, pp. 145–151, 2009. A. Sherbondy, D. Akers, R. Mackenzie, R. Dougherty, and B. Wandell, “Exploring connectivity of the brain’s white matter with dynamic queries,” IEEE Transactions on Visualization and Computer Graphics, vol. 11, no. 4, pp. 419–430, 2005. A. Qazi, G. Kindlmann, L. O’Donnell, S. Peled, A. Radmanesh, S. Whalen, A. Golby, and C.-F. Westin, “Two-tensor streamline tractography through white matter intra-voxel ﬁber crossings: Assessed by fMRI,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops., 2008, pp. 1–8. T. Behrens, H. Johansen-Berg, S. Jbabdi, M. Rushworth, and M. Woolrich, “Probabilistic diffusion tractography with multiple ﬁbre orientations: What can we gain?” NeuroImage, vol. 34, no. 1, pp. 144–155, 2007. [Online]. Available: http://dx.doi.org/10.1016/j.neuroimage.2006.09.018 M. Descoteaux, R. Deriche, T. Knosche, and A. Anwander, “Deterministic and probabilistic tractography based on complex ﬁbre orientation distributions,” IEEE Transactions on Medical Imaging, vol. 28, no. 2, pp. 269–286, 2009. E. Bier, M. Stone, and K. Pier, “Enhanced illustration using magic lens ﬁlters,” IEEE Computer Graphics and Applications, vol. 17, no. 6, pp. 62–70, 1997. J. Viega, M. J. Conway, G. Williams, and R. Pausch, “3d magic lenses,” in UIST ’96: Proceedings of the 9th annual ACM symposium on User interface software and technology. New York, NY, USA: ACM, 1996, pp. 51–58. A. Fuhrmann and E. Gröller, “Real-time techniques for 3d ﬂow visualization,” in VIS ’98: Proceedings of the conference on Visualization ’98. Los Alamitos, CA, USA: IEEE Computer Society Press, 1998, pp. 305–312. I. Assenmacher and T. Kuhlen, “The ViSTA Virtual Reality Toolkit,” The SEARIS Workshop on IEEE VR 2008, Reno, 2008. K. Zilles and K. Amunts, “Receptor mapping: architecture of the human cerebral cortex,” Current Opinion in Neurology, vol. 22, no. 4, pp. 331–339, 2009.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-493

493

NeuroVR 2 - A Free Virtual Reality Platform for the Assessment and Treatment in Behavioral Health Care Giuseppe RIVA 1-3, Andrea GAGGIOLI 1-2, Alessandra GRASSI 1-2, Simona RASPELLI 1, Pietro CIPRESSO 1, Federica PALLAVICINI 1, Cinzia VIGNA1, Andrea GAGLIATI 3 Stefano GASCO 3, Giuseppe DONVITO 3 1 Applied Technology for Neuro-Psychology Lab, Istituto Auxologico Italiano, Milan, Italy 2 Psychology Department, Catholic University of Milan, Italy 3 Virtual Reality & Multimedia Park, Turin, Italy

Abstract. At MMVR 2007 we presented NeuroVR (http://www.neurovr.org) a free virtual reality platform based on open-source software. The software allows non-expert users to adapt the content of 14 pre-designed virtual environments to the specific needs of the clinical or experimental setting. Following the feedbacks of the 2000 users who downloaded the first versions (1 and 1.5), we developed a new version – NeuroVR 2 (http://www.neurovr2.org) – that improves the possibility for the therapist to enhance the patient’s feeling of familiarity and intimacy with the virtual scene, by using external sounds, photos or videos. More, when running a simulation, the system offers a set of standard features that contribute to increase the realism of the simulated scene. These include collision detection to control movements in the environment, realistic walk-style motion, advanced lighting techniques for enhanced image quality, and streaming of video textures using alpha channel for transparency. Keywords: Virtual Reality, Assessment, Therapy, NeuroVR, Open Source

1. Introduction The use of virtual reality (VR) in medicine and behavioral neurosciences has become more widespread. This growing interest is also highlighted by the increasing number of scientific articles published each year on this topic: searching Medline with the keyword “virtual reality”, we found that the total number of publications has increased from 45 in 1995 to 3203 in 2010, showing an average annual growth rate of nearly 15%. Although it is undisputable that VR has come of age for clinical and research applications [1-3] the majority of them are still in the laboratory or investigation stage. In a recent review [4], Riva identified four major issues that limit the use of VR in psychotherapy and behavioral neuroscience: • the lack of standardization in VR hardware and software, and the limited possibility of tailoring the virtual environments (VEs); • the low availability of standardized protocols; • the high costs (up to 200,000 US$) required for designing and testing a clinical VR application; • most VEs in use today are not user-friendly.

494

G. Riva et al. / A Free VR Platform for the Assessment and Treatment in Behavioral Health Care

To address these challenges, we presented At MMVR 2007 NeuroVR (http://www.neurovr.org) a free virtual reality platform based on open-source software [5]. The software allows non-expert users to adapt the content of 14 pre-designed virtual environments to the specific needs of the clinical or experimental setting. Following the feedbacks of the 1000 users who downloaded the first version, we developed a new version – NeuroVR 2 (http://www.neurovr2.org) – that improves the possibility for the therapist to enhance the patient’s feeling of familiarity and intimacy with the virtual scene, by using external sounds, photos or videos.

2. NeuroVR 2 Using NeuroVR 2, the user can choose the appropriate psychological stimuli/stressors from a database of objects (both 2D and 3D) and videos, and easily place them into the virtual environment. The edited scene can then be visualized in the Player using either immersive or non-immersive displays. Currently, the NeuroVR library includes 18 different virtual scenes (apartment, office, square, supermarket, park, classroom, etc.), covering some of the most studied clinical applications of VR: specific phobias, cognitive rehabilitation, panic disorders and eating disorders. Specifically, the new version now includes full sound support and the ability of triggering external sounds and videos using three different approaches: the keyboard, timeline or proximity. The VR suite leverages two major open-source projects in the VR field: Delta3D (http://www.delta3d.org) and OpenSceneGraph (http:// www.openscenegraph.org). Both are building components that integrates with ad-hoc code to handle the editing and the simulation.The NeuroVR2 Editor's GUI is now based on the QT cross-platform application and UI framework from Nokia (http://qt.nokia.com/) that grants an higher level of editing and customization over the editor functionalities, while the graphical rendering is done using OpenSceneGraph, an open source high performance 3D graphics toolkit (http://www.openscenegraph.org/projects/osg). All the scenes building can now be done by the therapists using a cleaner and simpler interface, and through a powerful "Action and Trigger" system and an easy to use interface exposed by the editor. The scene creator can now also define how the scene reacts to the patients behavior, when he is using the scene in the VR Player. The NeuroVR2 Player too has been largely rewritten to grant a more efficient workflow for the scenes playback and has a brand new startup interface written in QT. The whole suite is developed in C++ language, targeted for the Microsoft Windows platform but fully portable to other systems if needed. The key characteristics that make NeuroVR suitable for most clinical applications are the high level of control of the interaction with the tool, and the enriched experience provided to the patient. These features transform NeuroVR in an “empowering environment”, a special, sheltered setting where patients can start to explore and act without feeling threatened. Nothing the patient fears can “really” happen to them in VR. With such assurance, they can freely explore, experiment, feel, live, and experience feelings and/or thoughts. NeuroVR thus becomes a very useful intermediate step between the therapist’s office and the real world. Actually, NeuroVR is used in the assessment and treatment of Obesity [6], Alcohol Abuse [7], Anxiety Disorders [1], Generalized Anxiety Disorders [8] and Cognitive Rehabilitation [9; 10].

G. Riva et al. / A Free VR Platform for the Assessment and Treatment in Behavioral Health Care

495

3. Conclusions In this chapter, we introduced NeuroVR 2, the new version of an advanced platform designed for the creation and customization of highly flexible VEs for clinical psychology and behavioral neurosciences. A future goal is to provide software compatibility with instruments that allow collection and analysis of behavioral data, such as eye-tracking devices and sensors for psycho-physiological monitoring. Beyond clinical applications, NeuroVR provides the VR research community with a free “VR lab”, which allows the creation of highly-controlled experimental simulations for different of behavioral, clinical and neuroscience applications

4. Acknowledgments The NeuroVR development was partially supported by the European funded project “Interstress” – Interreality in the management and treatment of stress-related disorders (FP7-247685).

5. References A. Gorini and G. Riva, Virtual reality in anxiety disorders: the past and the future, Expert Review of Neurotherapeutics 8 (2008), 215-233. [2] T.D. Parsons and A.A. Rizzo, Affective outcomes of virtual reality exposure therapy for anxiety and specific phobias: A meta-analysis, Journal of Behavior Therapy and Experimental Psychiatry 39 (2008), 250-261. [3] G. Riva and A. Gaggioli, Virtual clinical therapy, Lecture Notes in Computer Sciences 4650 (2008), 90107. [4] G. Riva, Virtual reality in psychotherapy: review, Cyberpsychology & Behavior 8 (2005), 220-230; discussion 231-240. [5] G. Riva, A. Gaggioli, D. Villani, A. Preziosa, F. Morganti, R. Corsi, G. Faletti, and L. Vezzadini, NeuroVR: an open source virtual reality platform for clinical psychology and behavioral neurosciences, Studies in Health Technology and Informatics 125 (2007), 394-399. [6] G. Riva, M. Bacchetta, G. Cesa, S. Conti, G. Castelnuovo, F. Mantovani, and E. Molinari, Is severe obesity a form of addiction? Rationale, clinical approach, and controlled clinical trial, CyberPsychology and Behavior 9 (2006), 457-479. [7] E. Gatti, R. Massari, C. Sacchelli, T. Lops, R. Gatti, and G. Riva, Why do you drink? Virtual reality as an experiential medium for the assessment of alcohol-dependent individuals, Studies in Health Technology and Informatics 132 (2008), 132-137. [8] F. Pallavicini, D. Algeri, C. Repetto, A. Gorini, and G. Riva, Biofeedback, VR and Mobile Phones in the treatment of Generalized Anxiety Disorders: A phase-2 controlled trial, Journal of CyberTherapy & Rehabilitation 2 (2009), 315-328. [9] S. Raspelli, L. Carelli, F. Morganti, B. Poletti, B. Corra, V. Silani, and G. Riva, Implementation of the multiple errands test in a NeuroVR-supermarket: a possible approach, Studies in Health Technology and Informatics 154, 115-119. [10] G. Albani, S. Raspelli, L. Carelli, F. Morganti, P.L. Weiss, R. Kizony, N. Katz, A. Mauro, and G. Riva, Executive functions in a virtual world: a study in Parkinson's disease, Studies in Health Technology and Informatics 154, 92-96. [1]

496

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-496

Personal Health Systems for Mental Health: The European Projects Giuseppe RIVA 1-2, Rosa BANOS 3, Cristina BOTELLA 4, Andrea GAGGIOLI 1-2, Brenda K WIEDERHOLD 5 1 Applied Technology for Neuro-Psychology Lab, Istituto Auxologico Italiano, Milan, Italy 2 Psychology Department, Catholic University of Milan, Italy 3 University of Valencia, Spain 4 Jaume I University, Spain 5 Virtual Reality Medical Center Europe, Bruxelles, Belgium

Abstract. Since the European funded project VREPAR - Virtual Reality in NeuroPsycho-Physiology (1995) – different European research activities have been using virtual reality and advanced information and communication technologies to improve the quality of care in the treatment of many different mental health disorders including anxiety disorders, eating disorders and obesity. Now the European Commission funding is shifting from the traditional hospital-centred and reactive healthcare delivery model toward a person-centred and preventive one. The main outcome of this shift is the “Personal Health Systems” (PHS) paradigm that aims at offering continuous, quality controlled, and personalized health services to empowered individuals regardless of location. The paper introduces four recently funded projects – Interstress, Monarca, Optimi and Psyche – that aim at using PHS based on virtual reality, biosensors and/or mobile technologies to improve the treatment of bipolar disorders, depression and psychological stress. Keywords: Virtual Reality, Assessment, Therapy, Personal Health Systems, Biosensors, Mobile Technologies, Stress, Depression, Bipolar Disorders.

1. Introduction New Information and Communication Technologies (ICT) offer clinicians significant new abilities to monitor patients’ conditions, thereby enabling them to diagnose problems earlier and treat them more effectively. For this reason, since the European funded project VREPAR - Virtual Reality in Neuro-Psycho-Physiology (1995 – 4th Framework Programme) – different European research activities have been using virtual reality and advanced information and communication technologies to improve the quality of care in the treatment of many different mental health disorders: anxiety disorders, male sexual disorders, eating disorders and obesity. Recently, the European Commission focus shifted from the traditional hospitalcentered and reactive healthcare delivery model toward a person-centered and preventive one. The main outcome of this shift is the “Personal Health Systems” (PHS) paradigm that aims at offering continuous, quality controlled, and personalized health services to empowered individuals regardless of location [1].

G. Riva et al. / Personal Health Systems for Mental Health: The European Projects

497

PHS cover a wide range of systems including wearable, implantable or portable systems, as well as Point-of-Care (PoC) diagnostic devices. Typically, the functioning of PHS is related to three main blocks as shown below [2]:

Figure 1. The structure of a Personal Health System (from [3])

1.

2.

3.

Data Acquisition: Collection of data and information related to the health status of a patient or healthy individual, e.g., through the use of sensors and monitoring devices. Data Analysis: Processing, analysis and interpretation of the acquired data to identify what information is clinically relevant and useful in diagnosis, management or treatment of a condition. This entails processing of data at both ends: locally at the site of acquisition (e.g., with on-body electronics) and remotely at medical centres. Data processing and interpretation takes into account the established medical knowledge and professional expertise where appropriate. Patient/Therapist Communication: Communication and feedback between various actors, in a loop: from patient/individual to medical centre; from medical centre that analyses the acquired data to doctor/hospital; and back to the patient/individual from either the wearable/portable/implantable system itself or the doctor or the medical centre (e.g., in the form of personalised feedback and guidance to the patient, adjusted treatment via closed loop therapy, control of therapy devices).

2. Personal Health Systems for Mental Health The European Commission is supporting research in this area under the Seventh Framework Programme (FP7). FP7 funds are used to support research into monitoring systems for patients with chronic diseases. In particular, such tools should provide improved quality of life for chronically ill patients, enabling them to stay at home rather than have to be admitted to hospitals. With ICT systems able to monitor a range

498

G. Riva et al. / Personal Health Systems for Mental Health: The European Projects

of parameters related to the patient’s condition, medical professionals can take timely decisions on the most effective treatment. Automatic alerts ensure doctors are immediately made aware of changes in the patient’s condition and can respond to prevent severe deteriorations. This approach can also used to improve mental health treatment. While most of us immediately think of either drugs or traditional talk therapy as the primary tools for mental health problems, there is a long history of using technologies for the diagnosis and treatment of psychological disorders. Specifically, PHS help us to connect on a level never seen in history; and for individuals less likely to seek professional help, they provide a confidential self-paced avenue towards change. For these reasons, the FP7 decided to support ICT based research projects providing solutions for persons suffering from stress, depression or bipolar disorders. These projects should address the parallel development of technological solutions, as well as new management or treatment models based on closed-loop approaches. Emphasis will be on the use of multi-parametric monitoring systems, which monitor various metrics related to behavior and to bodily and brain functions (e.g. activity, sleep, physiological and biochemical parameters). More, the required systems should aim at (i) objective and quantitative assessment of symptoms, patient condition, effectiveness of therapy and use of medication; (ii) decision support for treatment planning; and (iii) provision of warnings and motivating feedback. In the cases of depression and bipolar disorders, the systems should also aim at prediction of depressive or manic episodes. The solutions should combine wearable, portable or implantable devices, with appropriate platforms and services. Finally, they should promote the interaction between patients.

3. Personal Health Systems for Mental Health: The Funded Projects After a very demanding selection, the Commission provided financial support to the following four projects– Interstress, Monarca, Optimi and Psyche – that aim at using PHS based on virtual reality, biosensors and/or mobile technologies to improve the treatment of bipolar disorders, depression and psychological stress. Below there is a short description of their contents. 3.1. Interstress “Psychological Stress” occurs when an individual perceives that environmental demands tax or exceed his or her adaptive capacity . According to the Cochrane Database of Systematic Reviews the best validated approach covering both stress management and stress treatment is the Cognitive Behavioural (CBT) approach. Typically, this approach may include both individual and structured group interventions (10 to 15 sessions) interwoven with didactics. It includes in-session didactic material and experiential exercises and out-of-session assignments (practicing relaxation exercises and monitoring stress responses). The intervention focuses on: - Learning to cope better with daily stressors (psychological stress) or traumatic events (post traumatic stress disorder), - and optimizing one's use of personal and social resources.

G. Riva et al. / Personal Health Systems for Mental Health: The European Projects

499

CBT has undergone a very large number of trials in research contexts. However it has been less efficacious in clinical contexts and it has become obvious that CBT has some failings when applied in general practice. INTERSTRESS aims to design, develop and test an advanced ICT based solution for the assessment and treatment of psychological stress that is able to address three critical limitation of CBT: • The therapist is less relevant than the specific protocol used. • The protocol is not customized to the specific characteristics of the patient. • The focus of the therapy is more on the top-down model of change (from cognitions to emotions) than on the bottom-up (from emotions to cognitions). To reach this goal the project will use a totally new paradigm for e-health - Interreality [4; 5] – that integrates assessment and treatment within a hybrid environment, bridging physical and virtual world. Our claim is that bridging virtual experiences – fully controlled by the therapist, used to learn coping skills and emotional regulation - with real experiences – that allows both the identification of any critical stressors and the assessment of what has been learned – using advanced technologies (virtual worlds, advanced sensors and PDA/mobile phones) is the best way to address the above limitations. These devices are integrated around two subsystems - the Clinical Platform (inpatient treatment, fully controlled by the therapist) and the Personal Mobile Platform (real world support, available to the patient and connected to the therapist) – that will be able to provide: (i) Objective and quantitative assessment of symptoms using biosensors and behavioural analysis; (ii) Decision support for treatment planning through data fusion and detection algorithms; and provision of warnings and motivating feedback to improve compliance and long-term outcome. By creating a bridge between virtual and real worlds, Interreality allows a full-time closed-loop approach actually missing in current approaches to the assessment and treatment of psychological stress: • The assessment is conducted continuously throughout the virtual and real experiences: it enables tracking of the individual’s psycho-physiological status over time in the context of a realistic task challenge. • The information is constantly used to improve both the appraisal and the coping skills of the patient: it creates a conditioned association between effective performance state and task execution behaviours.

3.2. Monarca Manic-depression psychosis also known as bipolar disease is a mood disorder characterized by alternating periods of mania and depression. The current methodologies of diagnosis of this disease are based on self-reported experiences, typically done after a crisis episode has elapsed, that intrinsically lack objectivity due to the patients’ depressive or manic condition. The treatment of bipolar disorder is based on pharmacological and psychotherapeutic techniques often characterized by low compliance from patients.

500

G. Riva et al. / Personal Health Systems for Mental Health: The European Projects

In this scenario, MONARCA‘s aim is to develop and validate a closed-loop, multiparametric approach to the treatment, management, and self-treatment of bipolar disorder disease and facilitate effective and efficient therapy that reduces costs and load of the health system while at the same time improving the quality of life of the patients. The main project objectives consist in: • Bipolar disorder events assessment based on objective, measurable data. • Continuous multi-parametric monitoring. • Warnings on “risky” behavior (prevention of crisis). • Increase of patients’ awareness through self- monitoring and timely personalized coaching. To reach these objectives, the MONARCA tools will be designed and tested for the assessment and prediction of episodes of bipolar disorder disease. The design and tests will be carried out with the patients and healthcare professionals involvement. The system will consist of 5 main components: • A sensor enabled mobile phone. • A wrist worn activity monitor. • A novel “sock integrated” physiological (GSR, pulse) sensor. • A stationary EEG system for periodic measurements. • A home gateway. Additionally, GPS location traces, physical motion information, and recognition of complex activities (nutrition habits, household activity, amount and quality of sleep) will be combined into a continuously updated behavioral profile that will be provided to doctors in a meaningful way to support treatment. The system will support both the patients through personalized interfaces, helping them to better manage their disease and the medical professionals to adjust the therapy.

3.3. Optimi Depression and Stress related disorders are the most common mental illnesses and the prevention of depression and suicide is one of the five central focus points in the European Pact for Mental Health and Well Being. Currently the main treatments for mental illness are pharmacological and evidence based Cognitive Behavioral Therapy (CBT). However little is being done to develop effective systems for prevention of the onset of the illnesses. OPTIMI (Online Predictive Tools for Intervention in Mental Illness) is based on the hypothesis that the central issue and starting point of longer-term mental illness depends on the individual’s capacity and ability to cope with stress. Many of us are lucky not to be subject to daily stressful conditions that ultimately will result in changes to our biology and personality. Some are fortunate be able to cope with enormous real pressure. Many however are in high-risk situations where despite their best efforts, they decompensate and develop a depressive disorder. With the aim of detecting the onset of a mental illness, OPTIMI: • will identify the occurrence of high stress in the individual on a daily basis. • will determine the ongoing effect of stress on the individual by studying the behaviour pattern over a longer period

G. Riva et al. / Personal Health Systems for Mental Health: The European Projects

•

501

will also make estimates of the base line changes in the person’s state of mind using measurements that closely link depression with cognitive, motor and verbal behavior.

OPTIMI will use wearable appliances based on EEG, EGG, Cortisol levels, Voice analysis, Physical Activity analysis and a self reporting Electronic Diary in order to identify stress coping behavior patterns. The smart identification sensors that capture stress, specific behaviors and test results, will be enhanced with a knowledge based rule system to interpret the data and provide a diagnostic tool for both pharmacological and CBT based preventative and intervening treatments. OPTIMI will augment two existing computerized CBT systems to use these tools in real time to optimize the treatment cycle. OPTIMI will conduct two phases of trials with volunteers at high -isk situations. The first phase being held in 3 countries (China, Switzerland, Spain) over 6 months will use the tools, develop and fine tune the algorithms against the gold standard of regular therapist interviews. The second phase in 2 countries (UK, Spain) will use the calibrated tools and a computerized CBT preventative treatment system to evaluate effectiveness in reducing the impact of stress to high risk people as well as the relapse after treatment for depression.

3.4. Psyche One of the areas of great demand for the need of continuous monitoring, patient participation and medical prediction is that of mood disorders, more specifically bipolar disorders. Due to the unpredictable and episodic nature of bipolar disorder, it is necessary to take the traditional standard procedures of mood assessment through the administration of rating scales and questionnaires and integrate this with tangible data found in emerging research on central and peripheral changes in brain function that may be associated to the clinical status and response to treatment throughout the course of bipolar disorder. In this scenario, PSYCHE project will develop a personal, cost-effective, multiparametric monitoring system with the aim to treat and predict depressive or manic episodes in patient diagnosed with bipolar disorder by combining wearable and portable devices, with appropriate platforms and services. PSYCHE project will develop a personal, cost-effective, multi-parametric monitoring system based on textile platforms and portable sensing devices for the long term and short term acquisition of data from selected class of patients affected by mood disorders. The project will develop novel portable devices for the monitoring of biochemical markers, voice analysis and a behavioral index correlated to patient state. Additionally, brain functional studies will be performed under specific experimental protocols in order to correlate central measures with the clinical assessment, and the parameters measured by Psyche platform. Specifically, will focus on the following objectives: • Integration of sensors for physiological and behavioral data into a monitoring system for patients affected by bipolar disorders. • Development of novel portable devices for the monitoring of biochemical markers, voice analysis and a behavioral index correlated to mental illness.

502

G. Riva et al. / Personal Health Systems for Mental Health: The European Projects

•

Implementation of an integrated system to collect data from bipolar patients. Bipolar patients in different states of the illness (mania or depression episodes, remission) will be considered.

4. Conclusions PHS is a relatively new concept, introduced in the 1990s, that place the individual citizen in the centre of the healthcare delivery process. PHS can bring significant benefits in terms of improved quality of care and cost reduction in patient management, especially through applications for remote patient monitoring and disease management. The paper introduced four recently funded projects – Interstress, Monarca, Optimi and Psyche – that aim at using PHS based on virtual reality, biosensors and/or mobile technologies to improve the treatment of bipolar disorders, depression and psychological stress. The expected end outcome of these projects are : • Increased mental health practitioners productivity (i.e. reduced patient unit cost through remote monitoring and self care). • Reduced in-patient costs (i.e. due to delay of the time between when a disease becomes complex and chronic and the end of life or to the elimination altogether of the development of pre-morbid conditions into a full-blown disease); • Decreased diagnostic and treatment costs as less visits will be needed as a result of both preventive monitoring and chronic disease management.

5. Acknowledgments This paper was supported by the FP7 European funded projects “Interstress Interreality in the management and treatment of stress-related disorders” and ”Optimi Online Predictive Tools for Intervention in Mental Illness”.

6. References [1]

[2]

[3] [4] [5]

I. Iakovidis, Consultation Workshop on Personal Health Systems, in: PHS 2010 consultation, Brussels, Belgium, 2010, p. online: http://ec.europa.eu/information_society/activities/health/docs/events/phs2010wkshp/phs2010consult_w orkshop_report.pdf. C. Codagnone, D5.1 Consolidated Roadmaps Report - FP7-IST-2007- 215291, in: Roadmapping Personal Health Systems: Scenarios and Research Themes for Framework Programme 7th and beyond, Consorzio per l’innovazione nella gestione delle imprese e della Pubblica Amministrazione (MIP), Milan, Italy, 2009. G. Loukianos, Objective 5.1: “Personal Health Systems”, in: ICT WP 2011-12 - Challenge 5, ICT for Health, DG Information Society & Media, European Commission, Bruxelles, Belgium, 2010. G. Riva, D. Algeri, F. Pallavicini, C. Repetto, A. Gorini, and A. Gaggioli, The use of advanced technologies in the treatment of psychological stress, J CyberTher Rehab 2 (2010), 169-171. G. Riva, Interreality: A New Paradigm for E-health, Stud Health Technol Inform 144 (2009), 3-7.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-503

503

An Intelligent Virtual Human System for Providing Healthcare Information and Support Albert A. RIZZOa1, Belinda LANGEa, John G. BUCKWALTERa, Eric FORBELLa, Julia KIMa, Kenji SAGAEa, Josh WILLIAMSa, Barbara O. ROTHBAUMb, JoAnn DIFEDEc, Greg REGERd, Thomas PARSONSa and Patrick KENNYa a University of Southern California - Institute for Creative Technologies; bEmory University, cWeill Cornell Medical College; dMadigan Army Medical Center Army Abstract. Over the last 15 years, a virtual revolution has taken place in the use of Virtual Reality simulation technology for clinical purposes. Shifts in the social and scientific landscape have now set the stage for the next major movement in Clinical Virtual Reality with the “birth” of intelligent virtual humans. Seminal research and development has appeared in the creation of highly interactive, artificially intelligent and natural language capable virtual human agents that can engage real human users in a credible fashion. No longer at the level of a prop to add context or minimal faux interaction in a virtual world, virtual humans can be designed to perceive and act in a 3D virtual world, engage in spoken dialogues with real users and can be capable of exhibiting human-like emotional reactions. This paper will present an overview of the SimCoach project that aims to develop virtual human support agents to serve as online guides for promoting access to psychological healthcare information and for assisting military personnel and family members in breaking down barriers to initiating care. The SimCoach experience is being designed to attract and engage military Service Members, Veterans and their significant others who might not otherwise seek help with a live healthcare provider. It is expected that this experience will motivate users to take the first step – to empower themselves to seek advice and information regarding their healthcare and general personal welfare and encourage them to take the next step towards seeking more formal resources if needed. Keywords. SimCoach, Virtual Humans, Military Healthcare, Barriers to Care

Introduction Over the last 15 years, a virtual revolution has taken place in the use of simulation technology for clinical purposes. Technological advances in the areas of computation speed and power, graphics and image rendering, display systems, tracking, interface technology, haptic devices, authoring software and artificial intelligence have supported the creation of low-cost and usable PC-based Virtual Reality (VR) systems. At the same time, a determined and expanding cadre of researchers and clinicians have not only recognized the potential impact of VR technology, but have now generated a significant research literature that documents the many clinical targets where VR can add value over traditional assessment and intervention approaches (1-5). To do this, VR scientists have constructed virtual airplanes, skyscrapers, spiders, battlefields, ___________________________ 1 Albert Rizzo, University of Southern California, Institute for Creative Technologies, 12015 Waterfront Dr. Playa Vista, CA. 90064, [email protected]

504

A.A. Rizzo et al. / An Intelligent VH System for Providing Healthcare Information and Support

social settings, beaches, fantasy worlds and the mundane (but highly relevant) functional environments of the schoolroom, office, home, street and supermarket. And this state of affairs now stands to transform the vision of future clinical practice and research in the disciplines of psychology, medicine, neuroscience, physical and occupational therapy, and in the many allied health fields that address the therapeutic needs of those with clinical disorders. This convergence of the exponential advances in underlying VR enabling technologies with a growing body of clinical research and experience has fueled the evolution of the discipline of Clinical Virtual Reality. This paper presents the design vision for a Clinical VR project called SimCoach that aims to create intelligent virtual human agents to serve the role of online healthcare guides/coaches for military Service Members, Veterans and their significant others in an effort to break down barriers to care.

1. Virtual Humans in Clinical VR These shifts in the VR technological and scientific landscape have now set the stage for the next major movement in Clinical VR. With advances in the enabling technologies allowing for the design of ever more believable context-relevant “structural” VR environments (e.g. homes, classrooms, offices, markets, etc.), the next important challenge will involve populating these environments with Virtual Human (VH) representations that are capable of fostering believable interaction with real VR users. This is not to say that representations of human forms have not usefully appeared in Clinical VR scenarios. In fact, since the mid-1990’s, VR applications have routinely employed VHs to serve as stimulus elements to enhance the realism of a virtual world simply by their static presence. More recently, research and development has appeared in the creation of highly interactive, artificially intelligent and natural language capable virtual human agents. No longer at the level of a prop to add context or minimal faux interaction in a virtual world, these VH agents are designed to perceive and act in a 3D virtual world, engage in face-to-face spoken dialogues with real users (and other VHs) and in some cases, they are capable of exhibiting human-like emotional reactions. Previous classic work on virtual humans in the computer graphics community focused on perception and action in 3D worlds, but largely ignored dialogue and emotions. This has now changed. Intelligent VH agents can now be created that control computer generated bodies and can interact with users through speech and gesture in virtual environments (6). Advanced virtual humans can engage in rich conversations (7), recognize nonverbal cues (8), reason about social and emotional factors (9) and synthesize human communication and nonverbal expressions (10). Prototype-level embodied conversational characters have been around since the early 90’s (11) but significant advances have occurred more recently working systems used for training (12), intelligent kiosks (13) and virtual patients for clinical training (14). Both in appearance and behavior, VHs have now evolved to the point where they are usable tools for a variety of clinical and research applications.

2. Breaking Down Barriers to Care in Military Healthcare Research suggests that there is an urgent need to reduce the stigma of seeking mental health treatment in Service Members (SM) and Veteran populations. While US

A.A. Rizzo et al. / An Intelligent VH System for Providing Healthcare Information and Support

505

military training methodology has better prepared soldiers for combat in recent years, such hesitancy to seek treatment for difficulties that emerge upon return from combat, especially by those who may need it most, suggests an area of military mental healthcare that is in need of attention. Moreover, the dissemination of healthcare information to military SMs, Veterans and their significant others is a persistent and growing challenge. Although medical information is increasingly available over the web, users can find the process of accessing it to be overwhelming, contradictory and impersonal. At the same time, the need for military-specific health information is growing at an astounding rate. In this regard, the reports over the last few years of a surge in U.S. Army suicide rates have again thrust the challenges of military mental health care into the public spotlight. With annual suicide rates steadily rising since 2004, the month of Jan. 2009 saw 24 suspected suicides, compared to five in Jan. of 2008, six in Jan. of 2007 and 10 in Jan. of 2006 (15). In spite of a Herculean effort on the part of the U.S. Department of Defense (DOD) to produce and disseminate behavioral health programs for military personnel and their families, the complexity of the issues involved continue to challenge the best efforts of military mental health care experts, administrators and providers. Since 2004, numerous blue ribbon panels of experts have attempted to assess the current DOD and Veterans Affairs (VA) healthcare delivery system and provide recommendations for improvement. For example, the American Psychological Association Presidential Task Force on Military Deployment Services for Youth, Families and Service Members (16) poignantly stated that they were, “…not able to find any evidence of a well-coordinated or well-disseminated approach to providing behavioral health care to service members and their families.” The APA report also went on to describe three primary barriers to military mental health treatment: availability, acceptability and accessibility. More specifically: Well-trained mental health specialists are not in adequate supply (availability), the military culture needs to be modified such that mental health services are more accepted and less stigmatized, And even if providers were available and seeking treatment was perceived as more acceptable, appropriate mental health services are often not readily accessible due to a variety of factors (e.g. long waiting lists, limited clinic hours, a poor referral process and geographical location).The overarching goal reported in this and other reports is to provide better awareness and access to existing care while concurrently reducing the complexity and stigma in seeking psychological help. In essence, new methods are needed to reduce such barriers to care.

3. SimCoach Design Approach While advances in technology has begun to show promise for the creation of new and effective clinical assessment and treatment approaches, from Virtual Reality to computerized prosthetics, improvements in the military health care dissemination/delivery system are required to take full advantage of these evolving treatment methodologies, as well as for promoting standard proven intervention options. In response to the clinical health care challenges that the conflicts in Iraq and Afghanistan have placed on the burgeoning population of service members and their families, the U.S. Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury (DCoE) have recently funded our development of an intelligent, interactive, online Virtual Human (VH) healthcare guide program currently referred to as SimCoach. The SimCoach project that aims to address this need by developing

506

A.A. Rizzo et al. / An Intelligent VH System for Providing Healthcare Information and Support

virtual human support agents to serve as online guides for promoting access to psychological healthcare information and for assisting military personnel and family members in breaking down barriers to initiating the healthcare process. The SimCoach experience is being designed to attract and engage military SMs, Veterans and their significant others who might not otherwise seek help. It aims to create an experience that will motivate users to take the first step – to empower themselves to seek information and advice with regard to their healthcare (e.g., psychological health, traumatic brain injury, addiction, etc.) and general personal welfare (i.e., other nonmedical stressors such as economic or transition issues) – and encourage them to take the next step towards seeking more traditional resources that are available, when the need is determined. Rather than being a traditional web portal, SimCoach will allow users to initiate and engage in a dialog about their healthcare concerns with an interactive VH. Generally, these intelligent graphical characters are being designed to use speech, gesture and emotion to introduce the capabilities of the system, solicit basic anonymous background information about the user’s history and clinical/psychosocial concerns, provide advice and support, direct the user to relevant online content and potentially facilitate the process of seeking appropriate care with a live clinical provider. An implicit motive of the SimCoach project is that of supporting users determined to be in need, to make the decision to take the first step toward initiating psychological or medical care with a live provider. It is not the goal of SimCoach to breakdown all of the barriers to care or to provide diagnostic or therapeutic services that are best delivered by a real clinical provider. Rather, SimCoach will foster comfort and confidence by promoting users’ efforts to understand their situations better, to explore available options and initiate treatment when appropriate. Coordinating this experience will be a VH SimCoach, selected by the user from a variety of archetypic character options (See Figures 1-3), who will answer direct questions and/or guide the user through a sequence of user-specific questions, exercises and assessments. This interaction between the VH and the user will provide the system with the information needed to guide them to the appropriate next step of engagement with the system or to initiate contact with a live provider.

Figures 1-3. SimCoach Archetypes – Female Aviator, Battle Buddy, Retired Sergeant Major

The SimCoach project is not conceived to deliver diagnosis or treatment or as a replacement for human providers and experts. Instead, SimCoach will aim to start the process of engaging the user by providing support and encouragement, increasing awareness of their situation and treatment options, and in assisting individuals, who may otherwise be initially uncomfortable talking to a “live” care provider, in their efforts to initiate care.

A.A. Rizzo et al. / An Intelligent VH System for Providing Healthcare Information and Support

507

Users can flexibly interact with these VHs by typing text, clicking on character generated menu options and have some limited speech interaction during the initial phases of development. The feasibility of providing the option for full spoken natural language dialog interaction on the part of the user will be explored in the later stages of the project. Since this is designed to be a web-based system that will require no downloadable software, it is felt that voice recognition is not at a state where it could be reliably used at the current time. The options for SimCoach appearance, behavior and dialog is being designed to maximize user comfort and satisfaction, but also to facilitate fluid and truthful disclosure of medically relevant information. Based on the issues delineated in the initial interview, the user will be given access to a variety of general relevant information on psychology, neurology, rehabilitation, When relevant, users will also be directed to experts on specific areas such as stress, brain injury, marriage counseling, suicide, rehabilitation, reintegration and other relevant specialties the military healthcare system, and also to other SMs and Veterans by way of a variety of social networking tools (e.g., 2nd Life, Facebook, etc.). The user can progress through the system at their own pace over days or even weeks as they feel comfortable and the SimCoach will be capable of “remembering” the information acquired from previous visits and build on that information in similar fashion to that of a growing human relationship. The persistence of the SimCoach’s memory for previous sessions will require the user to sign into the system with a user name and password. However, that is optional for use of the system. Interspersed within the program will be the option to allow the user to perform some simple neurocognitive and psychological testing to inform the SimCoach’s creation of a model of the user to enhance the reliability and accuracy of the SimCoach output to the user, to support user selfawareness, and better guide the delivery of initial referral options. Users will also have the option to print out a summary of the computerized sessions to bring with them when seeking clinical care to enhance their comfort level, armed with knowledge, when dealing with the “real” human clinical care providers and experts. Software authoring tools are also being created that will allow other clinical professionals to create SimCoach “content” to enhance the likelihood that the program will evolve based on other care perspectives and emerging needs in the future. A fundamental challenge of the SimCoach project will be to better understand the diverse needs of the user base such that appropriate individual user experiences can be delivered to promote effective healthcare access. At the most basic level, there are immense differences in the needs of service members and their families. Further, there are likely large differences in the level of awareness that users will have of existing resources and in their own need/desire to engage such resources. Within the service member population there is a high likelihood that individual users will have had very diverse combat experiences, help-seeking histories and consequent impact on significant others. The net result of attempting to engage such a diverse user base is that the system will need to be able to employ a variety of general strategies and tactics to be relevant to each individual user. Focus groups and “Wizard of OZ” user studies are currently in progress in order to prepare the SimCoach interaction system for a wide range of potential dialog. In this regard, the SimCoach project is employing a variety of techniques to create the user experience. One relevant clinical model is the PLISSIT therapeutic framework (Permission, Limited Information, Specific Suggestions, and Intensive Therapy) (17), which provides an established model for encouraging help-seeking behaviors in persons who may feel stigma and insecurity regarding a clinical condition. In the

508

A.A. Rizzo et al. / An Intelligent VH System for Providing Healthcare Information and Support

SimCoach project, the aim is to address the “PLISS” components, leaving the intensive therapy component to live professionals to which users in need of this level of care can be referred. Another source of knowledge is social work practice. Such models take a case management approach, serving both as an advocate and a guide. The SimCoach development team is also leveraging knowledge from the entertainment/gaming industry. While knowledge from this community is not typically applied towards healthcare, a primary aim by this community is in the explicit attraction and engagement of individuals’ attention. As we work to develop this web-based VH interactive system we are working closely with experts in all three of these models to achieve our goal of engaging and focusing this unique user base on the steps to initiate care as needed. Additionally, all interactions will be consistent with findings that suggest that interventions with individuals with PTSD and other psychosocial difficulties achieve the following: 1) promotion of perceptions of self-efficacy and control 2) encouragement of the acceptance of change; 3) encouragement of positive appraisals; and 4) an increase in the usage of adaptive coping strategies (18). These principles of intervention will be implicit in all of the interactions between the SimCoach and its users.

4. Conclusions The systematic use of artificially intelligent virtual humans in Clinical Virtual Reality applications is still clearly in its infancy. But the days of limited use of VH’s as simple props or static elements to add realism or context to a VR application are clearly in the past. In this paper we have presented our general approach to the design and development of the SimCoach VH project envisioned to serve as an online clinical healthcare guide or coach. This work is focused on breaking down barriers to care (stigma, unawareness, complexity, etc.) by providing military SMs, Veterans, and their significant others with confidential help in exploring and accessing healthcare content and for promoting the initiation of care with a live provider if needed. This work will also afford many research opportunities for investigating the functional and ethical issues involved in the process of creating and interacting with virtual humans in a clinical context. While the ethical challenges may be more intuitively appreciated, the functional technology challenges are also significant. However, although this project represents an early effort in this area, it is our view that the clinical aims selected can still be usefully addressed in spite of the current limits of the technology. As advances in computing power, graphics and animation, artificial intelligence, speech recognition, and natural language processing continue to develop at current rates, the creation of highly interactive, intelligent VHs for such clinical purposes is not only possible, but probable.

References [1] [2] [3]

M.K. Holden, Virtual Environments for Motor Rehabilitation: Review, CyberPsychology and Behavior 8, 3 (2005), 187-211. T. Parsons & A.A. Rizzo, Affective Outcomes of Virtual Reality Exposure Therapy for Anxiety and Specific Phobias: A Meta-Analysis, Jour. of Behav. Therapy & Exper. Psychiatry 39 (2008), 250-261. M. Powers & P.M.G. Emmelkamp, Virtual reality exposure therapy for anxiety disorders: A metaanalysis, Journal of Anxiety Disorders 22 (2008), 561-569.

A.A. Rizzo et al. / An Intelligent VH System for Providing Healthcare Information and Support [4] [5] [6] [7] [8]

[9] [10] [11]

[12] [13] [14] [15]

[16]

[17] [18]

509

G. Riva, Virtual Reality in Psychotherapy: Review, CyberPsychology and Behavior 8, 3 (2005), 220230. F.D. Rose, B.M. Brooks & A.A. Rizzo, Virtual Reality in Brain Damage Rehabilitation: Review, CyberPsychology and Behavior 8, 3 (2005), 241-262. J. Gratch, et al., Creating Interactive Virtual Humans: Some Assembly Required, IEEE Intelligent Systems July/August (2002), 54-61. D. Traum, J. Gratch, et al., Multi-party, Multi-issue, Multi-strategy Negotiation for Multi-modal Virtual Agents, 8th International Conference on Intelligent Virtual Agents, Springer, Tokyo, Japan, 2008. L.P. Morency, I. de Kok et al., Context-based Recognition during Human Interactions: Automatic Feature Selection and Encoding Dictionary, 10th International Conference on Multimodal Interfaces, IEEE, Chania, Greece, 2008. Gratch & S. Marsella, A domain independent framework for modeling emotion, Journal of Cognitive Systems Research 5, 4 (2004), 269-306. M. Thiebaux, A. Marshall, et al., SmartBody: Behavior Realization for Embodied Conversational Agents. Intern. Conf. on Autonomous Agents and Multi-Agent Systems Porto, Portugal, 2008. T. Bickmore & J. Cassell, Social Dialogue with Embodied Conversational Agents. In Advances in Natural, Multimodal Dialogue Systems (J van Kuppevelt, L Dybkjaer and N Bernsen, Eds.), Kluwer Academic, New York, 2005. D. Evans, M. Hern, M. Uhlemann & A. Lvey, Essential Interviewing: A Programmed Approach to Effective Communication, (3rd Ed), Brooks/Cole Publishing Company, 1989. L. McCauley & S. D’Mello, A Speech Enabled Intelligent Kiosk, In IVA 2006 (J. Gratch et al. Eds), Springer-Verlag, Berlin, Germany, 132-144, 2006. P. Kenny, T. Parsons, G. Reger, C. Pataki & A. Rizzo, Virtual Patients for Future Leaders, Proceedings of the 2008 IITSEC, Orlando, FL., 2008. P. Jelinek & K. Hefling, AP Report: Army suicides at record high, passing civilians, Downloaded on 1/29/2009 at: http://www.google.com/hostednews/ap/article/ALeqM5jrRijfpxg8ZdUbcDpGbmnEpYPH9wD9616BB 80 American Psychological Association Presidential Task Force on Military Deployment Services for Youth, Families and Service Members. The Psychological Needs of U.S. Military Service Members and Their Families: A Preliminary Report. Retrieved 04/18/2007, from: http://www.apa.org/releases/MilitaryDeploymentTaskForceReport.pdf J. Annon, Behavioral Treatment of Sexual Problems, Harper-Collins. NY, NY, 1976. J.M. Whealin, J.I. Ruzek & S. Southwich, Cognitive-behavioral theory and preparations for professionals at risk for trauma exposure, Trauma Violence Abuse, 9, 100-113, 2008.

510

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-510

Virtual Reality Applications for Addressing the Needs of those Aging with Disability Albert RIZZOa1, Phil REQUEJOb, Carolee J. WINSTEINc, Belinda LANGEa, Gisele RAGUSAd, Alma MERIANSe, James PATTONf,g, Pat BANERJEEg & Mindy AISENb a University of Southern California - Institute for Creative Technologies; bRancho Los Amigos National Rehabilitation Hospital; cUniversity of Southern California – Division of Biokinesiology and Physical Therapy; dUniversity of Southern California – Rossier School of Education, eUniversity of Medicine and Dentistry of New Jersey, f Northwestern U., gUniversity of Illinois at Chicago Abstract. As persons with disabilities age, progressive declines in health and medical status can challenge the adaptive resources required to maintain functional independence and quality of life [1]. These challenges are further compounded by economic factors, medication side effects, loss of a spouse or caregiver, and psychosocial disorders [1-2]. With the gradual loss of functional independence and increased reliance on others for transportation, access to general medical and rehabilitation care can be jeopardized [2]. The combination of these factors when seen in the context of the average increase in lifespan in industrialized societies has lead to a growing crisis that is truly global in proportion. While research indicates that functional motor capacity can be improved, maintained, or recovered via consistent participation in a motor exercise and rehabilitation regimen [3], independent adherence to such preventative and/or rehabilitative programming outside the clinic setting is notoriously low [1]. This state of affairs has produced a compelling and ethical motivation to address the needs of individuals who are aging with disabilities by promoting home-based access to low-cost, interactive virtual reality (VR) systems designed to engage and motivate individuals to participate with “game”-driven physical activities and rehabilitation programming. The creation of such systems could serve to enhance, maintain and rehabilitate the sensorimotor processes that are needed to maximize independence and quality of life. This is the theme of the research to be presented at this MMVR workshop. Keywords. Virtual Reality, Aging, Disability, Technology

Introduction This MMVR workshop brings together researchers, users and industry partners to present and discuss the issues relevant to advancing the science and practice for using VR applications with those aging with and into disability. The session will commence with a user panel that will discuss the activity limitations experienced by aging adults with disabilities, followed by researchers presenting state of the art work using VR technologies for maintaining and enhancing sensorimotor and cognitive functions across the lifespan. Access to such systems by users will be the theme of the closing panel that will be made up of industry leaders, users and researchers. The objectives of this session include: 1. Promote awareness of the unique challenges and needs of adults with disability for maintaining and enhancing functional independence at this point in the lifespan; 2. Educate the public and professionals about the centers supported by the ___________________________ 1 Albert Rizzo, University of Southern California, Institute for Creative Technologies, 12015 Waterfront Dr. Playa Vista, CA. 90064, [email protected]

A. Rizzo et al. / VR Applications for Addressing the Needs of Those Aging with Disability

511

National Institute of Disability and Rehabilitation Research (NIDRR) that addresses these challenges using VR and associated technologies; 3. Clarify the issues involved in promoting equal access to VR and game-based applications for older adults with disability.

1. Virtual Reality Rehabilitation Over the last 15 years, a virtual revolution has taken place in the use of Virtual Reality (VR) simulation technology for clinical purposes. Technological advances in the areas of computation speed and power, graphics and image rendering, display systems, body tracking, interface technology, haptic devices, authoring software and artificial intelligence have supported the creation of low-cost and usable VR systems capable of running on a commodity level personal computer. At the same time, a determined and expanding cadre of researchers and clinicians have not only recognized the potential impact of VR technology, but have now generated a significant research literature that documents the many clinical targets where VR can add value over traditional assessment and intervention approaches [4-8]. To do this, scientists have constructed virtual airplanes, skyscrapers, spiders, battlefields, social events populated with virtual humans, fantasy worlds and the mundane (but highly relevant) functional environments of the schoolroom, office, home, street and supermarket. This state of affairs now stands to transform the vision of future clinical practice and research in the disciplines of psychology, medicine, neuroscience, physical and occupational therapy, and in the many allied health fields that address the therapeutic needs of children and adults with disabilities. This convergence of the exponential advances in underlying VR enabling technologies with a growing body of clinical research and experience has fueled the evolution of the discipline of Clinical Virtual Reality. This is expected to have significant impact on promoting access to VR technology for addressing the needs of persons aging with disabilities.

2. Theoretical Model for VR Applications for Successful Aging with Disabilities In 2008, the NIDRR made a 5-year award to the University of Southern California (USC) and Rancho Los Amigos National Rehabilitation Center (RLANRC) to establish a unique Rehabilitation Engineering Research Center (RERC)—“Optimizing Participation through Technology” (OPTT). The overall purpose of OPTT RERC is focused on those aging with and into disability. Over the first two years, we built a strong interdisciplinary infrastructure engaged in a set of research and development activities at the nexus of biomedical engineering and technology, sensorimotor systems rehabilitation, and gerontology and aging (see http://www.isi.edu/research/rerc/). Central to our research and development activities is the creation and delivery of VR simulation technologies for enhancing targeted skills (e.g., dexterity, balance) and exercise in those who are aging into and with disability. One of the biggest challenges we face with the research and development of VR applications is in maintaining a proper balance between usability, inherent flexibility to allow adaptation to the user’s needs, and cost, all while assuring the most efficient and appropriate means toward important rehabilitation goals and one that is compatible with an ever-changing technology. There is no doubt that many of the so called off the

512

A. Rizzo et al. / VR Applications for Addressing the Needs of Those Aging with Disability

shelf ‘rehab’ games are enjoyable, engaging and can even foster social interaction among family and friends—all important for supporting healthy minds and bodies, but it is less clear (and one of our major concerns) if important rehabilitation goals that are tailored to the special needs of those aging with a disability are being achieved. To this end, we have attempted to create a theoretically defensible, evidence-based, conceptual model as a means to guide the development of VR simulation technologies for rehabilitation in the context of OPTT-RERC. Our model contains three overlapping elements: Skill Acquisition, Capacity Building, and Motivational Enhancements. VR game-based rehabilitation provides the glue for achieving the aims of the model. A pathway from impairment reduction (i.e. physiological loss) to functional capability (e.g., instrumental activities of daily living, self-care, mobility) to more general function in real-world contexts (e.g., independent living, social participation) is more often implicit but less frequently operationalized in VR games-based therapeutic intervention protocols. Task-specific practice is considered to be the most important element of any behavioral training program, particularly when improved functional skills are sought (e.g., cognitive and physical). In fact, the effects of practice are often underestimated and all too often, programs fail to be effective because either ample practice time was not prescribed or compliance was poor. Recently, our work has shown that in addition to time-on-task practice, the practice structure (e.g., variable or constant) is important for optimizing consolidation and motor learning [9]. The scientific rationale and evidence for impairment mitigation (capacity) comes from a growing body of work showing the importance of fundamental impairments including strength and control for restoration of function (e.g., individuals post-stroke, elderly atrisk for falls) [10]. Similarly, the scientific rationale and evidence for motivational enhancements (intrinsic drive) as well as the pursuit of meaningful goals for sustainable behavioral change (e.g., cognitive behavioral intervention such as ‘Matter of Balance’, comes from a growing body of work showing the importance of selfregulation, self-management, and self-efficacy for behavioral change that supports beneficial outcomes [11-14]. In most cases, the motivational enhancements strengthen self-confidence and support participant control or autonomy (intrinsic motivation). As well, providing choice in the context of effective intervention programs engages the learner and supports adherence. The active ingredients of an effective task-oriented VR game likely consists of interactions that are: 1) Challenging enough to require new learning and active attentional engagement to solve a motor problem; 2) Progressive and optimally adjustable, such that over practice, the task demand is optimally adapted to the user’s capability and the environmental context. Extending the environment outside the laboratory or clinic to the home is an important aspect of an optimal consumer-centered program. 3) Interesting enough to promote active participation to engage a ‘particular type of repetition’ that Bernstein referred to as ‘problem-solving’. For more details, we elaborate and provide examples of intervention programs based on this conceptual model in the context of stroke rehabilitation in two recent publications [15-16]. Once the particular task or set of tasks to-be-trained has been chosen, the VR simulation game can be embedded into the fully-defined, task-oriented training program. VR simulation technology affords certain key design features that map nicely onto the active ingredients for an effective program. These include: 1) Focus on a specific skill and involve data-based and task-specific training (skill/practice); 2) Have adjustable difficulty levels from something simple for the user to accomplish, to a level representing normal or skilled performance (capacity building); 3) Be quantifiable in

A. Rizzo et al. / VR Applications for Addressing the Needs of Those Aging with Disability

513

order to assess progress (assessment, motivation); 4) Be administered repetitively and hierarchically to allow enough practice with the right amount of challenge (motivation, skill acquisition/practice); 5) Provide the user with feedback as to the outcome of performance (builds confidence); 6) Have some relevance to real world function (meaningful, skill/task-based, motivating); 7) Motivate and engage the user (enhances compliance). With this brief overview of our theoretical model, the following sections summarize the rationale and results from NIDRR-supported VR research and development efforts targeting three core rehabilitation domains relevant for persons aging with or into disability: upper extremity sensorimotor function, cognitive processing, and a VR software package for designing customized exercise applications.

3. VR Simulations for Recovery of Upper Extremity Function Sensorimotor impairments and participation restrictions remain a pervasive problem for patients post stroke, with recovery of upper extremity function particularly recalcitrant to intervention. 80% to 95% of persons demonstrate residual upper extremity impairments lasting beyond six months after their strokes (17). One of the issues that may contribute to less than satisfactory outcomes for the upper extremity is the complexity of sensory processing and motor output involved in normal hand function. There is a vital need to develop rehabilitative training strategies that will improve functional abilities and real-world use of the arm and hand in order to increase independence [18-19]. To address this need, we have developed an exercise system that integrates robotic-assisted arm training with complex VR gaming simulations [20]. We are using this system in several innovative ways. First, the system allows us to utilize current neurophysiological findings regarding the importance of repetitive, frequent and intensive practice for skill development and motor recovery to train the hemiparetic upper extremity of people post stroke [21-22]. After a two-week period of VR training, participants were able to more effectively control the upper limb during reaching and hand interaction with a target as demonstrated by improved proximal stability, smoothness and efficiency of the movement path. This improved control was in concert with improvement in the distal kinematic measures of fractionation (ability to move fingers one at a time) and improved speed. These changes in robotic measures were notably accompanied by robust changes in the clinical outcome measures. Because of the systematized, objective nature of this system, it allows us to test hypotheses regarding the most efficacious therapeutic interventions. It is controversial whether training the upper extremity as an integrated unit leads to better outcomes than training the proximal and distal components separately. During recovery from a lesion, the hand and arm are thought to compete with each other for neural territory. Therefore, training them together may actually have deleterious effects on the neuroplasticity and functional recovery of the hand. However, neural control mechanisms of arm transport and hand-object interaction are interdependent. Therefore, complex multi-segmental motor training is thought to be more beneficial for skill retention. We are investigating these competing theories to determine if and how competition among body parts for neural representations stifles functional gains from different types of training regimens. Lastly, we are also exploring how this promising therapeutic strategy may actually change neural connections in the brain as a patient’s motor functions improve. Animal and human research suggests that functional recovery is dependent on neural reorganization. We have developed an innovative MRI-compatible VR system that

514

A. Rizzo et al. / VR Applications for Addressing the Needs of Those Aging with Disability

tracks bilateral hand movement and uses these measurements to drive motion of virtual hand models during an fMRI experiment. Our preliminary data suggest that, indeed, robot-assisted training in VR may be beneficial for functional recovery after chronic stroke. Further, our data suggest that this functional recovery may be attributed to increased functional connectivity in bilateral sensorimotor cortex.

4. VR Simulation for Recovery of Cognitive Function A key aspect of rehabilitation recovery is deliberate, repetitive practice that resembles functional activity in some way. This can involve tool use, and can be accomplished via interactive VEs that incorporate robotics that can render haptic feedback. Our recent work has evaluated several clinically promising uses of these haptic/graphic environments (HGE) in stroke recovery, which often involves exploiting the features of HGE by distorting the visual and mechanical fields in which users operate. Our MMVR talk will present this work as an extension of our neurorehabilitation research [23-24] with inpatients having moderate-severe impairments following a TBI, in which a key deficit is attention. Our approach is believed to be distinctive in that we use the HGE in an extremely minimal environment -- a cursor and target in the dark with no other distractions. This concept emerged from our observations that patients in the earliest stages of recovery exhibit severe impairment of both focused and sustained attention. It was hypothesized that since VR allows for complete control of the task environment and difficulty level, it might lead to short-term gains that in turn might lead to longer-term benefits in these users. Our initial study provided an assessment of the tolerance of a VR intervention for attention remediation in persons with severe TBI [25]. A small sample of patients with severe TBI in the early stages of recovery received acute inpatient rehabilitation along with a minimalistic interactive VR reaching task. Six TBI patients and three healthy controls were tested while reaching for successive visual targets. Initial results showed the system to be well-tolerated and engaging, and users focused on the task without being distracted by environmental stimuli while showing gains in proficiency (increased target acquisitions). Encouraged by this preliminary work, our next study tested a more comprehensive 2-day treatment protocol to evaluate how haptic cues might be beneficial [26]. Users visited the laboratory for two successive days, and on each day they executed 6 blocks of training that included three haptic feedback conditions: 1) no haptic forces, 2) resistive haptic forces (giving subjects a "breakthrough" sensation as they acquired the target) and 3) facilitative forces (giving subjects a 250 ms “nudge” toward the target whenever a 1 second period of near-zero speed was detected). We hypothesized that haptic feedback would refocus the patient's attention as well as increase time-on-task for subsequent movements. Overall, 19 of 22 patients were able to tolerate the task and target acquisition time on a 3D cancellation task showed improved performance across trials. The haptic nudge condition resulted in superior performance compared to the breakthrough condition. Subjects with posttraumatic amnesia demonstrated improved performance across trials, with carryover between day 1 and day 2 indicating improved procedural memory despite the fact that subjects lacked the capacity for declarative learning. We are now developing a prolonged 2-week clinical intervention while moving to transfer this technology to a viable system on a smaller and more affordable scale to promote better patient access. This study will include incremental task difficulty adjustment features so that as

A. Rizzo et al. / VR Applications for Addressing the Needs of Those Aging with Disability

515

subjects improve, the task challenges will engage patients in both the earliest stages of recovery and in those whose progress requires more demanding exercises.

5. VR Simulation Architecture for Remote Exercise The application of VR in rehabilitation engineering can be broadly categorized into two main areas: motor rehabilitation and cardiovascular exercise. Fortunately, both motor rehabilitation and cardiovascular exercise share almost the same technologies and devices. The primary difference being that cardiovascular exercise requires a larger range of movement, higher frequency and larger intensity. The challenge is to harness the technology optimally for efficient utilization for these differing aims. To address this challenge, we have created a software architecture, known as REGAL (Remote Exercise and Game Architecture Language) to design virtual exercise environments for people with lower body disabilities. The software also facilitates the development of new virtual exercises by other developers who are interested in creating fitness and rehabilitation applications. New exercises can be uploaded to the architecture and be played by users similar to the pre-loaded exercises we developed, and the REGAL architecture also supports 3D PC games developed using other SDKs and game engines. The first version of the architecture and a “throwing” demo received feedback via questionnaire from 26 participants [27]. Users reported that they were easily able to experience a 3D perspective in the virtual environment (VE); were satisfied with the consistent and responsive natural VE interaction (but expected more); and users with lower body disabilities showed higher satisfaction with all aspects of the VE than users without disabilities. In the current version of the architecture, we have designed two rowing demonstrations: the first is based on PhysX and Coin3D with a very simple 3D scene, while the second is based on a FarCry demo (http://farcry.us.ubi.com/) to show how to reuse currently available VR games [28]. In the FarCry demonstration, the acceleration data from a Wii remote was used to successfully drive the navigation and avatar movement. We are evolving the software such that users can define their own gestures for particular exercise equipment and the architecture will record user-specific interaction gestures that will be recognized later in the application. Such flexible options built into the REGAL software are expected to promote individualization of the exercise program based on the needs and equipment available to the user.

6. Conclusions In this workshop session, we will present the challenges faced in the creation and delivery of VR simulation technologies for addressing barriers faced by individuals aging with and into disability. The research and development briefly presented here illustrates the use of a range of interaction devices, programming engines, and emerging approaches for addressing the sensorimotor and cognitive challenges that adults with disabilities face throughout the lifespan. Such NIDRR-funded research aims to support the theory-informed development of a range of VR applications focused on improving upper/lower extremity functions, remediating cognitive impairments, and for providing technology to promote access to home-based exercise for persons aging with and into disability. These novel VR exercise and rehabilitation strategies are now providing evidence that indicate improvements in function when used by these groups.

516

A. Rizzo et al. / VR Applications for Addressing the Needs of Those Aging with Disability

References [1] A.W. Heinemann, State of the science of postacute rehabilitation: setting a research agenda and developing an evidence base for practice and public policy. Rehabil Nurs 33,2 (2008), 82-7. [2] C.I. Vincent, I. Deaudelin, L. Robichaud, J. Rousseau, C. Viscogliosi, L.R. Talbot, & J. Desrosiers, Rehabilitation needs for older adults with stroke living at home: perceptions of four populations. BMC Geriatric 7 (2007), 20. [3] R.T. Galvin, T. Cusack & E. Stokes, A randomised controlled trial evaluating family mediated exercise (FAME) therapy following stroke." BMC Neurol 8 (2008), 22. [4] M.K. Holden, Virtual Environments for Motor Rehabilitation: Review, Cyberpsy & Behav. 8,3 (2005), 187-211. [5] T. Parsons & A.A. Rizzo, Affective Outcomes of Virtual Reality Exposure Therapy for Anxiety and Specific Phobias: A Meta-Analysis, Jour. of Behav. Therapy & Exper. Psychiatry 39 (2008), 250-261. [6] G. Riva, Virtual Reality in Psychotherapy: Review, CyberPsy. and Behavior 8, 3 (2005), 220-230. [7] F.D. Rose, B.M. Brooks & A.A. Rizzo, Virtual Reality in Brain Damage Rehabilitation: Review, CyberPsychology and Behavior 8, 3 (2005), 241-262. [8] Rizzo AA, Buckwalter JG, van der Zaag C. Virtual environment applications for neuropsychological assessment and rehabilitation. In Stanney, K, (Ed.), Handbook of Virtual Environments. New York, NY: L.A. Earlbaum; 2002, 1027-64. [9] Kantak SS, Sullivan KJ, Fisher BE, Knowlton BJ, Winstein CJ. Neural substrates of motor memory consolidation depend on practice structure. Nat Neurosci. 2010 Aug 13(8):923-5. [10] Liu C, Latham N. Progressive resistance strength training for improving physical function in older adults. Cochrane Database of Systematic Reviews 2009(3). [11] Bandura A. Health promotion by social cognitive means. Health Educ Behav. 2004 Apr;31(2):143-64. [12] Bodenheimer T, Lorig K, Holman H, Grumbach K. Patient self-management of chronic disease in primary care. JAMA. 2002 Nov 20;288(19):2469-75. [13] Hart T, Evans J. Self-regulation and goal theories in brain injury rehabilitation. J Head Trauma Rehabil. 2006 Mar-Apr;21(2):142-55. [14] Siegert RJ, Taylor WJ. Theoretical aspects of goal-setting and motivation in rehabilitation. Disability and Rehabilitation. 2004 Jan 7;26(1):1-8. [15] Winstein C, Wolf S. Task-oriented training to promote upper extremity recovery. In: Stein J, RL H, Macko R, Winstein C, Zorowitz R, eds. Stroke Recovery & Rehabilitation. New York: Demos Medical 2008: 267-90. [16] Wolf S, Winstein C. Intensive physical therapeutic approaches to stroke recovery. In: Cramer S, Nudo R, eds. Brain Repair After Stroke: Cambridge University Press 2010. [17] Kwakkel G, Kollen BJ, van der Grond J, Prevo AJ. Probability of regaining dexterity in the flaccid upper limb: impact of severity of paresis and time since onset in acute stroke. Stroke.2003;34(9):2181-6. [18] Krebs HI, Volpe BT, Ferraro M, Fasoli S, Palazzolo J, Rohrer B, et al. Robot-aided neurorehabilitation: from evidence-based to science-based rehabilitation. Topics in Stroke Rehabil. 2002;8(4):54-70. [19] Kahn LE, Lum PS, Rymer WZ, Reinkensmeyer DJ. Robot-assisted movement training for the strokeimpaired arm: Does it matter what the robot does? J Rehabil Res Dev. 2006;43(5):619-30. [20] Merians AS, Poizner HP, Boian R, Burdea G, Adamovich SV, Sensorimotor training in a virtual reality environment: does it improve functional recovery post-stroke? Neural Rehabil Neur Repair,2006:20 (2). [21] Jenkins WM, Merzenich MM. Reorganization of neocortical rep- resentations after brain injury: a neurophysiological model of the bases of recovery from stroke. Prog Brain Res 1987;71:249-66. [22] Nudo RJ, Wise BM, SiFuentes F, Milliken GW. Neural substrates for the effects of rehabilitative training motor recovery after ischemic infarct. Science 1996; 272:1791-4. [23] Patton JL, Dawe G, Scharver C, Muss-Ivaldi FA, Kenyon R. 2006. Robotics and virtual reality: A perfect marriage for motor control research and rehabilitation. Assistive Technology 18:181-95. [24] Rozario S, Housman S, Kovic M, Kenyon R, Patton J. 2009. Therapist-mediated post-stroke rehabilitation using haptic/graphic error augmentation. IEEE Engin.in Med.& Bio. Conf. Minn., MN [25] Dvorkin A, Zollman F, Beck K, Larson E, Patton J. 2009. A Virtual Environment-Based Paradigm for Improving Attention in TBI. IEEE Intern. Conf. on Rehabilitation Robotics (ICORR). Kyoto, Japan. [26] Dvorkin A, Ramaiya M, Zollman F, Larson E, Pacini S, et al. 2010. A virtual environment-based paradigm for improving attention in severe TBI. Cog. Neurosci. Soc. meeting. Montreal, Canada. [27] Zhang, S., Banerjee, P. P., Luciano, C.: Virtual Exercise Environment for Promoting Active Lifestyle for people with Lower Body Disabilities. Proc. IEEE International Conference on Networking, Sensing and Control, Chicago, 2010, pp. 80-84. [28] Banerjee, P. P., Zhang, S., Luciano, C. and Rizzi, S.: Remote Exercise and Game Architecture Language (REGAL), Proc. Rectech 2nd State of the Science Conference, Arlington, VA, 2010. 53-56.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-517

517

The Validation of an Instrumented Simulator for the Assessment of Performance and Outcome of Knot Tying Skill: A Pilot Study David ROJASa,c, Sayra CRISTANCHOb,1, Claudia RUEDAa, Lawrence GRIERSONc Alex MONCLOUa, and Adam DUBROWSKIc,d a Faculty of Electronics Engineering, Pontificia Bolivariana University, Bucaramanga, Colombia. b Department of Surgery and Centre for Education Research & Innovation, Schulich School of Medicine & Dentistry, University of Western Ontario. London, Canada. c The Hospital for Sick Children Learning Institute. Toronto, Ontario Canada. d Department of Pediatrics, Faculty of Medicine, and The Wilson Centre, University of Toronto. Toronto, Canada.

Abstract. The construct validity of a surgical bench-top simulator with built-in computer acquired assessments was examined. It features two parallel elastic tubes instrumented with flexion sensors that simulate the walls of a wound. Participants from three groups (9 novices, 7 intermediates, 9 experts) performed 10 two-handed, double square knots. The peak tensions at the initiation of the first knot, the completion of the first knot and the completion of the second knot, as well as measures of movement economy indicated technical performance. Product quality was indicated by knot stability defined as the amount of slippage of the knot under the tension. There were significant differences between experts and novices for peak tension on first knot (p=.03), movement economy (p=.02), and knot stability (p=.002). The results support the construct validity of these objective measures. Keywords. Simulation, computer-based assessment, technical skills

Introduction The objective assessment of surgical technical performances and the resulting products are critical for ensuring that standards of practice are met as well as for augmenting the learning process. Methods of objective assessment have been classified broadly as either expert-based [1,3-4] or computer-based [2,7-11]. The expert-based methods are criticized for their objectivity and feasibility [1]. Additionally, because these methods rely on expert presence, they are expensive for formative evaluations [6]. Computerbased assessments rely on the acquisition of information about technical performance such as specific motor processes [10], movement patterns and related information about hand motion efficiency [7], as well as outcomes [2,13]. However, assessments of 1

Corresponding Author.

518

D. Rojas et al. / The Validation of an Instrumented Simulator

the processes leading to the skillful execution of technical surgical skills and the quality of their products are typically performed independently. In this paper we present an integrated alternative that allows for the computerbased assessment to measure both the process and the quality of final products associated with knot-tying skills. Specifically, we examined the construct validity of the variables associated with a newly developed computer-based surgical skills assessment device. We hypothesized that the analyses of the variables derived from the device can discern between novice, intermediate and experts. Thus the construct validity of this device will be supported. Of secondary interest to the study is to establish baseline characteristics of the behaviours of the three groups of performers in order to conduct further, more powerful validation studies.

1. Methods 1.1. Participants The novices (n=9) were undergraduate students from the Universidad Pontificia Bolivariana (Bucaramanga, Colombia). The intermediates (n=7) were senior medical students from Hospital Universitario Santander (Bucaramanga, Colombia). The experts (n=9) were senior surgical residents from Hospital Universitario Santander and practicing surgeons from Toronto General Hospital (Toronto, Ontario, Canada), Sunnybrook Hospital (Toronto, Ontario, Canada) and The Hospital for Sick Children (Toronto, Ontario, Canada). All volunteers provided informed consent in accordance with the guidelines set out by The University of Toronto Research Ethics Board and the 1964 Declaration of Helsinki. 1.2. Apparatus and Procedure A custom bench-top suturing simulator was developed (Fig. 1). It was instrumented with sensors to measure the movement parameters related to motor processes, overall movement patterns, economy of hand motions and the quality of the final products. Specifically, two parallel polyethylene tubes were mounted with two flexion sensors (Flex Sensor, SpectraSymbol, Salt Lake City, Utah), which measured the variation in the distance between the tubes. The participants were asked to initiate the knot tying skill with sutures (SOLFSILK 2.0, TYCO Healthcare, Pointe-Claire, Canada) in a predetermined position, and use the two-handed double square technique to approximate the two tubes as close to each other as possible. All sensor data were transformed and recorded as Voltages (V) and processed by computer (HP Pavilion dv2550, Intel Pentium 4, 2.8 GHz) via LabView software (8.5, Texas, EEUU). All data were filtered through an 8Hz low-pass Butterworth filter (Matlab, R2007b, California, EEUU) prior to analysis.

D. Rojas et al. / The Validation of an Instrumented Simulator

519

Figure 1. Experimental apparatus and steps in the technique. The first pull, defined as the moment when the initial tension on the apparatus was applied to perform the first knot (a), the first knot (b) and the last knot (c), defined as the peak tensions during the completion of the first and last knots.

1.3. Variables of Interest We examined three distinct knot-tying phases: the first pull (the moment when the initial tension on the apparatus was applied to perform the first knot), the first knot (the peak tension during the completion of the first knot), and the final knot (the peak tension during the completion of the second knot). [10,11] Movement economy was quantified by measuring the number of minor peaks in the signal using a custom algorithm. A minor peak in the signal was considered a variation in the current position of the tube of more than its own diameter (5 mm) with respect to its immediately previous position. These variables are referred to as measures of technical performance. The distance between the tubes at two-seconds following the knot’s completion was used to measure the quality of the final product (i.e., knot stability). This product quality measure is related to the amount of slippage of the knot under tension imposed by the natural elastic properties of the tubes. 1.4. Analysis Each dependent measure was subjected to a one-way analysis of variance with group experience as the only factor (Expert, Intermediate, Novice). All effects significant at p < .05 were further analyzed using Tukey’s Honestly Significant Difference post hoc methodology.

2. Results 2.1. The First Pull The peak tension on the initial pulling of the sutures revealed no significant group effect (grand mean = 0.93, standard deviation = 0.07 Volts (V)). The analysis of time to the initial pulling of the sutures did not reveal any significant effect (grand mean = 0.41, standard deviation = 0.04 seconds (s)).

520

D. Rojas et al. / The Validation of an Instrumented Simulator

2.2. The First Knot The peak tension of the first knot revealed a significant main effect for group, F (2, 22) = 3.78, p = .038. Post hoc analysis indicated that the first knots of experts were tighter than those of novices. The intermediate group’s knots were centered between and not significantly different than other groups’ (Figure 2a). The analysis of the time to the first knot yielded no significant group effects (grand mean = 1.03, standard deviation = 0.13 s).

Figure 2. The means and standard deviations for the number of dependent variable collected from the instrumented simulator plotted as a function of experience (Expert, Intermediate, Novice), a) The mean of peak tension (expressed in Voltages (V)) of the first knot, b) the mean movement economy measure (expressed as a number of peaks performed), c) the mean of peak tension (expressed in V) of the knot stability.

2.3. The Final Knot The peak tension of the sutures at the final knots revealed no significant differences between the groups (grand mean = 2.20, standard deviation = 0.12 V); however, the experts’ knots trended towards being conventionally tighter than the novices’ (F (2, 22) = 3.17, p = .063). The analysis of time to the final knot did not reveal any significant differences (grand mean = 2.87, standard deviation = 0.55 s).

D. Rojas et al. / The Validation of an Instrumented Simulator

521

2.4. Movement Economy The movement economy revealed a significant group effect, F (2, 22) = 8.15, p = .002. Post hoc analysis of this effect indicated that the intermediate group moved the tubes more while performing the knot tying skill than the novice or expert groups (Figure 2b). 2.5. Knot Stability (i.e., slippage of the knot under tension imposed by the natural elastic properties of the tube) The knots stability revealed a significant group main effect, F (2, 22) = 4.91, p = .017 Post hoc comparisons indicated that the experts’ knots remained closer together than the novices’ (Figure 2c). The intermediate’s knots were centered between and were not significantly different than the other two groups.

3. Discussion The results offer preliminary evidence that supports construct validity of the device insofar that it was capable of differentiating the technical performance and the quality of the final product of performers of varying levels of experience. The unique feature of this device is that it captures data about the technical performance as well as the quality of the final product or outcomes at the same time. Specifically, we have demonstrated that experts apply lower tensile forces than the novices during placement of the surgical knots. Furthermore, that the intermediate trainees used the same amount of tension as the experts on the first knot and slightly, yet not significantly less, in the second knot may indicate that these parameters are acquired and optimized quickly in the learning process. An interesting finding was that the immediate group performed the skills with more movements than the novices and experts. One possible explanation for this finding is that the intermediates perform the skill with certain comfort and explore variations in practice in order to learn, while the novices perform more consistently because the current technique is the only one they know. On the contrary the experts perform consistently as they have automated the technique [15]. The results are also in agreement with previous reports investigating the validity of methods for monitoring technical performances [1,7-9,12]. However, our data reveal low sensitivity to differentiating levels of trainees. This is in partial agreement with previous findings reported by Datta and colleagues [7]. Finally, the measures of the final outcomes in this study also support construct validity. That is, the knot stability measure differed significantly between the experts and the novice trainees. This is in line with previous reports [13,14]. Poorly constructed knots are unstable and slip under tension exposing the tissues, and potentially leading to increased rates of infections. However, here, the knot stability measures show low sensitivity where the quality of the knots produced by the intermediate group of trainees was not statistically different from those of the experts and the novices. This low sensitivity may be a shortcoming of the sensors.

522

D. Rojas et al. / The Validation of an Instrumented Simulator

3.1. Implications for Simulation-Based Teaching The ability to quantify the technical performance and the quality of the final products may enable surgical skills educators to provide the trainees with in-depth feedback about their skills performance. Providing such feedback is in accordance with the benefits described by many theoretical models of motor skill acquisition. [15-17]. One important characteristic of these models is that the acquisition of a motor skill progresses through distinct stages, which may require different feedback strategies. Our data suggest that different feedback about different aspects of the overall performance may be necessary for different levels of trainees. That is, it is apparent that the motor process parameters related to peak tension applied during knot tying technique were well optimized by the intermediate trainees, and therefore they will most likely not benefit from augmented feedback about this aspect of their performance. On the contrary the novice trainees may need augmented feedback to optimize this parameter. On the contrary, both novices and intermediates showed poor movement economy and therefore both groups would probably benefit from feedback about movement patterns. Likewise, augmented feedback about the quality of knots may be highly beneficial to both the novice and intermediate groups in order to reach the learning goals. 3.2. Study Limitations A significant limitation of the current study is the small number of participants in each of the experimental groups. This could affect the variability in the data and therefore affect our ability to detect group differences where they truly exist; or, conversely, detect group differences where they do not exist. For example, this lack of statistical power could explain why most variables, except for the economy of motion, were not able to discriminate between the expert and intermediate groups. This is especially significant if considering using this device for the assessment of resident proficiency before proceeding to the clinical arena. In addition, to increase the statistical power, we are currently improving technological aspects of the device, such as the sensor sensitivity and the algorithms, to better determine the variables of interests.

Acknowledgments The authors would like to acknowledge the contributions of the Natural Sciences and Engineering Council of Canada for providing funding for this project. We would also like to acknowledge contributions of BISEMIC group of undergraduate engineering students from the Universidad Pontificia Bolivariana, Bucaramanga, Colombia who built the first prototype of the assessment device.

References [1] Brydges R, Sidhu R, Park J, Dubrowski A. Construct validity of computer-assisted assessment: quantification of movement processes during a vascular anastomosis on a live porcine model. Am J Surg. 2007;193(4):523-9.

D. Rojas et al. / The Validation of an Instrumented Simulator

523

[2] Martin JA, Reznick RK, Rothman A, Tamblyn RM, Regehr G. Who should rate candidates in an objective structured clinical examination? Acad Med. 1996;71(2):170-5. [3] Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skill via an innovative "bench station" examination. Am J Surg. 1997;173(3):226-30. [4] Reznick RK, MacRae H. Teaching surgical skills –changes in the wind. Nw Eng J Med. 2006;355:2664-2669. [5] Szalay D, MacRae H, Regehr G, Reznick R. Using operative outcome to assess technical skill. Am J Surg. 2000;180(3):234-237. [6] Wanzel KR, Ward M, Reznick RK. Teaching the surgical craft: from selection to certification. Curr Prob Surg. 2002;39:573–660. [7] Datta V, Chang A, Mackay S, Darzi A. The relationship between motion analysis and surgical technical assessments. Am J Surg. 2002;184(1):70-3. [8] Bann SD, Khan MS, Darzi AW. Measurement of surgical dexterity using motion analysis of simple bench tasks. World J Surg. 2003;27(4):390-4. [9] Moorthy, Munz, Dosis, Bello, Darzi. Motion analysis in the training and assessment of minimally invasive surgery. Minim Invas Therap Alli Tech. 2003;12(3):137-42. [10] Dubrowski A, Sidhu R, Park J, Carnahan H. Quantification of motion characteristics and forces applied to tissues during suturing. Am J Surg. 2005;190(1):131-136. [11] Dubrowski A, Larmer JC, Leming JK, Brydges R, Carnahan H, Park J. Quantification of process measures in laparoscopic suturing. Surg Endosc. 2006;20:1862–6. [12] Brydges R, Classen R, Larmer J, Xeroulis G, Dubrowski A. Computer-assisted assessment of onehanded knot tying skills performed within various contexts: a construct validity study. Am J Surg. 2006;192(1):109-13. [13] Hanna GB, Frank TG, Cuschieri A. Objective assessment of endoscopic knot quality. Am J Surg. 1997;174(4): 410-413. [14] Leming K, Dorman K, Brydges R, Carnahan H, Dubrowski A. Tensiometry as a measure of improvement in knot quality in undergraduate medical students. Ad Health Sci Ed. 2006;12(1):331-344. [15] Fitts PM, Posner MI. Human Performance. Belmont (CA): Books Cole; 1967. [16] Gentile AM. A working model of skill acquisition with application to teaching. Quest Monog. 1972;17:3–23. [17] Gentile AM. Skill acquisition: Action, movement, and neuromotor processes. In: Carr JH, Shepherd RB, editors. Movement Science: Foundations for Physical Therapy. Rockville (MD): Aspen; 2000. P. 87111.

524

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-524

Manual Accuracy in Comparison with a Miniature Master Slave Device- Preclinical Evaluation for Ear Surgery RUNGE, A.d ;HOFER, M. a,d;DITTRICH, E. a; NEUMUTH, T. a; HAASE, R. e; STRAUSS, M. c; DIETZ, A. a,d; LÜTH, T. c; STRAUSS, G. a,d a BMBF-Innovation Center Computer Assisted Surgery ICCAS, University of Leipzig b MIMED, Department of Micro Technology and Medical Device Engineering, Prof. Dr. Tim C. Lueth, Technische Universität München c IMETUM, Central Institute for Medical Engineering, Technische Universität München d University Hospital, ENT Department / Plastic Surgery, University of Leipzig e Phacon GmbH, Leipzig

Abstract. Manual accuracy in microsurgery is reduced by tremor and limited access. A surgical approach through the middle ear also puts delicate structures at risk, while the surgeon is often working at an unergonomic position. At this point a micromanipulator could have a positive influence. A system was developed to measure “working accuracy”, time and precision during manipulation in the middle ear. 10 ENT- surgeons simulated a perforation of the stapedial footplate on a modified 3D print of a human skull in a mock OR. Each trial was repeated more than 200 times aiming manually and using a micro-manipulator. Data of over 4000 measurements was tested and graphically processed. Work strain was evaluated with a questionnaire. Accuracy for manual and micromanipulator perforation revealed a small difference. Learning curves showed a stronger decrease both in deviation and time when the micromanipulator was used. Also a lower work strain was apparent. The micromanipulator has the potential as an aiding device in ear surgery. Keywords. Accuracy, Precision, Patient Model, Micromanipulator, Learning Curve, Stapedotomy, Master Slave

Introduction Surgery of the middle or inner ear requires accuracy at a sub-millimeter level and must therefore be classified as micro-surgery. A high level of surgical accuracy and precision is required to gain optimal postoperative results. However, there are several factors bearing a risk of inaccuracy: the working area is limited to a radius of approximately 1 cm.The movement range of the surgeon’s hand is further reduced by the use of microsurgical instruments and few possible ways of preparation (mainly anterograde). Ergonomics -the interaction between humans and another complete system, in this case the surgical setup- in an otologic intervention are rather poor: the high shoulder of the patient is an impediment, since it makes direct ergonomic access impossible and the surgeon has to perform with fully extended arms (Fig.1). Under these circumstances physiological tremor (an involuntary oscillatory movement

A. Runge et al. / Manual Accuracy in Comparison with a Miniature Master Slave Device

525

coherent in all human motion) puts another negative influence on microsurgical accuracy and precision. Mürbe et al. proved that its ampitude correlates directly with the muscular tension the weight of the instrument as well as physical exertion. [1] A resulting deviation might cause damage of delicate structures and disturb the regular surgical procedure (such as fixing a protheses or implant properly)[2] Of course the surgical experience, commonly illustrated as a learning curve, also has to be regarded [3]. One can find several studies on physiological tremor and its impact on accuracy [4,5,6,7]. In ear surgery, however, the exactness of manual pointing accuracy with a surgical instrument has not yet been evaluated and in addition has not yet been compared to that of a micro-manipulator. It eliminates increased physiological tremor and could improve ergonomics as it is remote- controlled.[8] It is the goal of this study 1. to create a suitable and realistic model in order to measure positioning accuracy of a surgical instrument regularly used in middle ear surgery. 2. to determine positioning accuracy in manual performance as compared to that of the micromanipulator. The results shall be visualized as learning curves and serve as a basis for an illustration of an individual surgical signature. 3. to determine the time-span of execution for manual performance as compared to that of the micro-manipulator. The results shall be displayed as learning curves. 4. to determine a possible reduction of physical work strain by using the micromanipulator.

1. Materials and Methods Surgical task. In this study, the perforation of the stapedial footplate with a perforator of 0.6 mm in diameter (Karl Storz, Germany) was simulated in a strongly simplified and thus easily repeatable manner. This procedure is a step of a stapedotomy, a routine intervention in the event of otosclerosis. Simulation Model and Set Up. The simulation took place in vitro using a 3D Rapid Prototyping print as a phantom.The module included the bony external hearing canal and the tympanic cavity, the latter being openly accessible. The boundary to the inner ear was replaced by a glass plate. The outline of the stapes footplate (surface area: 3.2mm²) [5] was schematically represented on it and provided with a central target point. In place of the inner ear, a miniature camera was installed and connected to an image processing application, through which any movement in the middle ear could be registered.The phantom was positioned in a clinically realistic environment. The stapedial footplate was viewed through a microscope (Zeiss, OPMI 1, Oberkochen, Germany). An upper-body model simulated the “high shoulder” of the patient. Performance. 10 otosurgeons were assigned to guide the perforator from a defined starting point to the center of the stapedial footplate via a transmeatal approach. The test-surgeons were asked to execute the procedure as precisely as possible and at an efficient pace with more than 200 repetitions. Additionally, the same assignment was carried out with the aid of a miniature masterslave device (TUM Manipulator, Prof. Tim Lueth, Technische Universitaet Muenchen). The manipulator is a surgical assistant device with bimodular structure. It contains a motor, through which an instrument-conveying carriage can be manipulated in three different degrees of freedom (x-, y- and z-directions) (Fig. 3). A movement clearance of about 10mm at each axis is possible. An operating console serves as the

526

A. Runge et al. / Manual Accuracy in Comparison with a Miniature Master Slave Device

interface with two joysticks (Fig. 2 and Fig. 3), whereby the movements of the surgeon’s hand are scaled and translated to the motor via cable. Also postural tremor is not transferred to the instrument’s tip. A large movement of the hand is translated into a small motion of the device (scaling 3.5:1). Rotatable clamps enable regulation of the miniature master-slave device in a relaxed seating position during microscopic viewing (Fig. 2). Within the groups data was divided into experts (n=5, specialists in otosurgery) and novices (n=5, residency in ENT) and again time and accuracy values were compared. Automatic evaluation. With the aid of image-processing software (Phacon, Leipzig, Germany) in combination with workflow-analysis software (ICCAS Surgical Workflow Editor, ICCAS, Leipzig, Germany), the time-span of the experiment, positioning deviation and a movement signature were automatically registered. Departure from the starting point and arriving at the end-point were detected through contact readings,all of them being set off by a touch with the tip of the perforator. The release of the starting point within the external hearing canal immediately started time measurement. The trial was concluded automatically as soon as the surgeon touched the glass plate firmly. Statistical methods. For analysis of the collected data, Microsoft Office Excel 2007 and SPSS 17.0 were used. Plain descriptive analysis provided a general overview. The Wilcoxon test was used to investigate significant differences for the manual and micromanipulator group regarding time and accuracy. Also, differences between experts and novices were considered. In order to determine the learning effect of each test-person, the mean values of the first and last 50 pointing attempts for both manual and manipulator assisted test series were placed in a percentage ratio. Graphic displays such as scatter plots, learning and plateau curves were generated for visual feedback on surgical precision and improvement of measuring results. Assessment of the stress level. After execution of the manual and manipulator assisted measurements, the surgeons had to evaluate perceived physical work-strain (stress levels) using a ratio scale from the “NASA- TLX score for evaluation of physical work strain “[9,10].

Figure 1 Unergonomic position

Figure 2 Ergonomic position

A. Runge et al. / Manual Accuracy in Comparison with a Miniature Master Slave Device

Figure 3 Set Up with model skull, PC with software and micromanipulator

2. Results Table 1 Overall results Distance Time Manual Manual [mm] [s]

Distance Time Micromanipulator Micromanipulator [mm] [s]

Mean value

.22

.65

.29

3.69

Median

.20

1.28

.26

2.59

.10

1.87

.19

4.96

Variance

.01

3.49

.04

24.64

Maximum

1.22

31.38

1.73

88.70

N

2206

2206

2286

2286

Standard

deviation

Table 2 Accuracy and time: Experts vs. novices; all values refer to the mean value of each group Micromanipulator

Manual Experts Accuracy [mm] Time [s]

.21 ± .86

.24± .14

1.14 ± 1.97

2.773 ± 3.66

Novices Accuracy [mm] Time [s]

.20 ± .11

.28 ± .23

1.45 ± 1.76

2.48 ± 5.92

527

528

A. Runge et al. / Manual Accuracy in Comparison with a Miniature Master Slave Device

Table 3 Improvement on accuracy and time depending on manual and micro-manipulator preparation, mean values Avg. 1st 50 Avg. last 50 Avg. 1st 50 Micro- Avg. last 50 Micromanual trials manual trials manipulator trials manipulator trials Accuracy [mm] Relative improvement

Time [s] Relative improvement

.26

.21

.38

21,2%

1.83

1.76 3,9%

.26 32,4%

5.58

2.71 51,5%

Overall. (Table 1) A total of 4492 measurements were carried out. The measuring values did not refer to a Gaussian distribution.The mean accuracy measured during manual performances was 0.22 mm and the application of the micromanipulator lead to deviations smaller than 0.29 mm on average. For manual perforation, 0.65 s were required, and using the micromanipulator, each pointing attempt took 3.69 s on average. Only few values exceed the majority of the measurements considerably explaining the difference between mean values and the median in measurements of accuracy and time. Experts vs. Novices. (Table 2) Comparing mean values of accuracy in manual performance (.21 mm experts vs. .20 mm novices), no significant difference between the two groups were found. It took the more experienced test persons less time to reach the target (1.14 s experts vs. 1.45 s novices). Looking at the micromanipulator performance, the novice group did not meet with the accuracy level of the expert group (.24 mm experts vs. .28 mm novices), but pointed significantly faster (2.77 s experts vs. 2.48 s novices). Learning curves. Learning curves were generated to display results of each test person. These graphs regard accuracy achieved and time required for both manual and manipulator aided pointing (Fig. 4 and Fig. 5). A total of 40 learning curves could thus be generated. (For reasons of capacity, there are only two examples provided at this point.) Comparison of the percentage ratios shows an improvement by 32.4% on average in manipulator aided accuracy while the mean manual accuracy was increased by 21.2% (Table 3). The time required for a targeting attempt at the end of the manipulator-assisted series of measurements was on average 51.5% shorter than in the beginning, the duration for manual simulation decreased by 3.9%. Movement signatures. The measured values around the target point were arranged in scatter plots, again regarding manual performance (Fig. 6) and application of the micro-manipulator separately (Fig. 7). A total of 20 individual movement signatures were thus recorded. In 7/10 cases, a higher density of the measuring points is recognisable when the micromanipulator was applied, indicating a higher precision. (For reasons of capacity, there are only two examples provided at this point.) Stress level. The overall physical exhaustion when using the micro-manipulator reached a score of 8.7 (max. score 21, with 0 showing no exertion and 21 indicating high exerction) compared to 12.4 for the manual pointing.

529

(mm)

Deviation from ref. point

A. Runge et al. / Manual Accuracy in Comparison with a Miniature Master Slave Device

No. of trials

Time

Figure 4 learning curve, comparing manual (dotted line) and master slave (solid line) accuracy

No. of trials

Deviation from ref. point on y-axis (Pixel)

(Pixel)

Deviation from ref. point on y-axis

Figure 5 learning curve, comparing time needed for manual (dotted line) and master slave (solid line) aided pointing

Deviation from ref. point on x-axis (Pixel)

Figure 6 Scatter plot manual accuracy

Deviation from ref. point on x-axis (Pixel)

Figure 7 scatter plot master slave aided accuracy

530

A. Runge et al. / Manual Accuracy in Comparison with a Miniature Master Slave Device

3. Conclusion 1.

It was possible to develop a suitable and realistic phantom for measuring positioning accuracy of a perforator. Thus automatic measuring of the manual accuracy in comparison to that of the micromanipulator was possible. 2. It was possible to display each series of accuracy measurements as a learning curve, showing interindividual differences. Evaluation of the average first and last fifty results showed a clearly more steeply descending course when the micromanipulator was used. Individual movement signatures were visualized. 3. For the time length of pointing attempts, inter-individual differences were evidenced for all 10 test surgeons displayed in the learning curves. Regarding time for master slave supported pointing attempts, the learning curves showed an even stronger decrease. 4. The NASA-TLX score was lower with deployment of the micromanipulator which indicates improved ergonomics. Applying the micromanipulator revealed no improvement in accuracy as compared to that of pure manual performance; here the accuracy was even slightly higher. This might be due to rather poor manipulator assisted accuracy especially in the beginning of the measuring trials. However, the steep learning curves show a strong learning effect in decreasing values as test persons became more familiar with the device. The manipulator used in this study is an approach to provide the surgeon with a compact, remote controlled instrument without limiting his manual performance range. Repeated usage as well as clinical evaluation may eventually show the manipulator’s potential as an assisting device in middle ear surgery or approach to the cochlea as in compensating for poor ergonomics or as an additional steady holding function (e.g. the positioning of a prothesis) without prolonging the standard surgical procedure.

References [1]

Muerbe D, Huettenbrink KB, Zahnert T et al. Tremor in otosurgery: influence of physical strain on hand steadiness. Otol Neurotol 2001; 22:672-7. [2] Babighian GG, Albu S. Failures in stapedotomy for otosclerosis. Otolaryngol Head Neck Surg 2009; 141:395-400. [3] Su, E;Win TL; Ang, WT; Lim TC; Teo, CL; Burdet, E. Micromanipulation accuracy in pointing and tracing investigated with a contact-free measurement system Conference proc: Annual international Conference of the IEEE Engineering in Medicine and Biology Society; 2009 : 3960-3 [4] Wade P, Gresty MA, Findley LJ. A normative study of postural tremor of the hand. Arch Neurol 1982; 39:358-62. [5] Duval C, Jones J. Assessment of the amplitude of oscillations associated with high-frequency components of physiological tremor: impact of loading and signal differentiation. Exp Brain Res 2005; 163:261-6. [6] Riviere CN, Khosla PK. Accuracy in positioning of handheld instruments. Amsterdam: Microsurgical and Robotic Interventions I, 18th Annual International Conference of the IEEE Engineering in Medicine and Biology Society; 1996: 211-213. [7] Calzetti S, Baratti M, Gresty M et al. Frequency/amplitude characteristics of postural tremor of the hands in a population of patients with bilateral essential tremor: implications for the classification and mechanism of essential tremor. J Neurol Neurosurg Psychiatry 1987; 50:561-7. [8] Maier T, Strauss G, Dietz A et al. First clinical use of a new micromanipulator for the middle ear surgery. Laryngorhinootologie 2008; 87:620-2. [9] Samel A, Wegmann HM, Vejvoda M et al. Two-crew operations: stress and fatigue during longhaul night flights. Aviat Space Environ Med 1997; 68:679-87. [10] Hart SA, Staveland LE. Human Mental Workload. Amsterdam: North Holland Press; 1988

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-531

531

Are Commercially Available Simulators Durable Enough for Classroom Use? Jonathan C. SALUD, BS, Katherine BLOSSFIELD IANNITELLI, MD, Lawrence H. SALUD, MS and Carla M. PUGH, MD, PhD1 Northwestern University Feinberg School of Medicine, Department of Surgery, 251 East Huron Street, Galter 3-150, Chicago, IL 60611 USA

Abstract. Our efforts show that commercially available simulators can be modified to affect realism and durability. Keywords. Simulators, simulation development, simulator maintenance

Introduction According to the American Cancer Society performing the digital rectal exam (DRE) is a highly effective way to detect prostate cancer [1]. Mannequin-based simulators allow medical students to practice the DRE in safe, controlled, environments [2-3]. However, the durability and reliability of commercial products for classroom use has not been established.

1. Methods There are many commercially available diagnostic prostate simulators. For classroom use and research purposes we chose a simulator with the capacity for interchangeable prostate pathologies, but there were many features that could be improved. We performed a DRE on our chosen simulator and compared it to performing the exam on real patient. The prostate gland of the simulator was not placed in the commonly accepted location and the rectum did not feel realistic. In addition, the model could only be placed in the left lateral decubitus position, Figure 1 (a). US practitioners most commonly perform the DRE with the patient in prone position [4-6].

1 Corresponding Author: Carla M Pugh, MD, PhD, Northwestern University Feinberg School of Medicine, Department of Surgery, 251 East Huron Street, Galter 3-150, Chicago, IL 60611 USA; E-mail: [email protected]

532

J.C. Salud et al. / Are Commercially Available Simulators Durable Enough for Classroom Use?

Figure 1. (a) Image of the diagnostic prostate simulator from Limb and Things. Notice that it is in the left decubitus position. (b) Image of the repositioned diagnostic prostate simulator. Notice that the model is supported by a different piece of fiberglass, as well as a cork wedge.

The model also lacked durability. After undergoing 200 student examinations, it was common for the ano-rectal junction to tear, Figure 2. It is unclear whether this was a problem with the structural design or a limitation of the material that was used.

Figure 2. (a) Image of the ano-rectal junction torn from extensive use in class. (b) Image of an intact anorectal junction.

In an effort to make the mannequin feel realistic, we modified the interior of the prostate simulator using customized foam pieces to provide support and imitate the feel of a rectal cavity. We also used commercially available artificial flesh to mimic the texture and elasticity of the rectal cavity. Our team measured the amount of stress that could be applied to three different materials: Vixskin, Cyberskin, and Fakeskin, to determine which would be most appropriate for our DRE simulator.

2. Results To allow for flexibility in positioning, we replaced the original frame with a fiberglass base that allows for prone position, Figure 1 (b). Next we modified the model’s interior with foam, allowing us to stabilize and correct the prostate gland position. Although we placed the prostate in a more accurate location, we maintained the manufacturer’s intent for interchangeable prostates. After securing the new prostate position with custom foam, we improved the model’s realism, by lining the perineum with memory foam. The foam helped imitate the fleshy feel of the ischiorectal spaces. Lastly, we addressed rectal wall feel and durability. Our tests found Fakeskin/ Dermasol 300 to have favorable elastic properties. Test results showed σ = 0.445MPa for Fakeskin/ Dermasol 300, σ = 0.447MPa for Cyberskin, and σ = 0.25MPa for Vixskin. Although Cyberskin had an advantage of σ = .002, we felt that Fakeskin/ Dermasol 300 was the best fit for our simulator because it is also robust, non-reactive, warm to touch, and moldable. The next problem was the constant tearing of the perineum’s anus. In an effort to strengthen the model we stitched different materials to the rectum base. First, we

J.C. Salud et al. / Are Commercially Available Simulators Durable Enough for Classroom Use?

533

sutured a non-lubricated condom to the models base. This worked, but success was short lived because the condom ripped after some use. Next we stitched plastic strips from a zip-lock bag to the base, which also ripped. We now use Tegaderm, folded in half, secured together by two adhesive undersides to support the base. This works better than the previous alternatives. This mechanism provides more durability to the ano-rectal junction and is a conduit for the newly developed Fakeskin/ Dermasol 300 rectum. The modified model was tested by faculty at Northwestern University’s Feinberg School of Medicine, and was approved for classroom use. Although the model is not perfect, faculty agree that it is more realistic than many commercial products and is outstanding in giving students the chance to practice clinical skills. The model now endures 400 examinations, double the original, before the perineum rips, and must be replaced. We may confirm that it is not only a material problem, but a structural one as well.

3. Conclusion This new model, a heavily modified version of the original, is an effective learning tool. Although it does not feel exactly like a human rectum, it is an enormous improvement on commercially available products and effective teaches medical students how to perform the male DRE. Research shows that our models significantly lower anxiety for first year medical students while learning the digital rectal exam. Our modified mannequin supports prior research, which maintains that companies who produce mannequins for simulation-based learning should have a closer relationship with the professors that use them [7]. A stronger relationship will address problems regarding realism and durability prior to manufacturing and distribution, leading to more reliable simulators, better-prepared medical students, and better healthcare. We have reported some of the above findings to the manufacturer and are happy to note they have incorporated a few of the changes into their product.

4. Acknowledgements The model used in this research was the Diagnostic Prostate Simulator (PART #60364) from Limbs & Things USA. We would like to thank Leslie Gittings, Nurul Hamzah, Sofiah Syed, and Latha Subramaniam for their research on artificial skin. We would also like to thank Jessica Haring, and Abby Kaye, for editing this article.

References [1]

[2] [3]

How is prostate cancer found? American Cancer Society. (2010). < http://ww2.cancer.org/docroot/CRI/content/CRI_2_2_3X_How_is_prostate_cancer_found_36.asp?rnav =cri >. N.J. Maran, and R.J. Glavin, Low- to high-fidelity simulation - a continuum of medical education?, Medical Education 37 (2003), 22-28. C.M. Pugh, K.M Blossfield-Iannitelli, D.M. Rooney, and L.H. Salud, Use of mannequin-based simulation to decrease student anxiety prior to interacting with male teaching associates, Teaching and learning in medicine (In press).

534 [4] [5] [6] [7]

J.C. Salud et al. / Are Commercially Available Simulators Durable Enough for Classroom Use? J.Y. Gillenwater, Adult and pediatric urology. Lippincott Williams & Wilkins, Philadelphia, 2002. M.F. Campbell, and P.C. Walsh, Campbell’s urology, W.B. Saunders, Philadelphia, 1998 J.L Willms, H. Schneiderman, and P.S. Algranati, Phisical diagnosis: Bedside evaluation of diagnosis and function, Williams & Wilkins, Baltimore, 1994 J.B. Cooper, and V.R. Taqueti, A brief history of the development of mannequin simulators for clinical education and training, Quality and safety in healthcare 13 (2004), i11-i14

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-535

535

Toward a Simulation and Assessment Method for the Practice of Camera-Guided Rigid Bronchoscopy Lawrence H. SALUD1, Alec R. PENICHE, Jonathan C. SALUD, Alberto L. DE HOYOS, and Carla M. PUGH Northwestern University, Department of Surgery, Feinberg School of Medicine

Abstract. We have developed a way to measure performance during a cameraguided rigid bronchoscopy using manikin-based simulation. In an effort to measure contact pressures within the airway during a rigid bronchoscopy, we instrumented pressure sensors in a commercially available bronchoscopy task trainer. Participants were divided into two groups based on self-reported levels of expertise: novice (none to minimal experience in rigid bronchoscopy) and experts (moderate to extensive experience). There was no significant difference between experts and novices in the time taken to complete the rigid bronchoscopy. However, novices touched a greater number of areas than experts, showing that novices induce a higher number of unnecessary soft-tissue contact compared to experts. Moreover, our results show that experts exert significantly less soft tissue pressure compared to novices. Keywords. Bronchoscopy Simulation, Surgical Simulation, Sensors, Learning Technologies, Simulation Support Systems, Evaluation/methodology, Humancentered computing

Introduction Rigid bronchoscopy requires exceptional technical skill to avoid causing trauma to the upper airway and to the bronchi. In addition, clinicians must avoid damaging the teeth and soft tissues, or lacerating the larynx or the bronchial mucosa [1]. Performing the proper direct insertion technique demands a considerable amount of expertise [2]. In a statement released by the European Respiratory Society, a trainee should have performed at least 20 supervised rigid bronchoscopy procedures before attempting it alone. Pulmonary fellowship training programs in the United States currently require at least 50 bronchoscopies for pulmonary trainees to achieve competency in flexible bronchoscopy [3]. While the most dangerous complications associated with rigid bronchoscopy are those related to the use of general anesthesia, the traditional apprenticeship model is not the best way to train this skill as most patients in need of the procedure are in distress and facing life or death circumstances [2]. The development of novel, manikin-based simulators that can capture and present performance metrics has provided an opportunity to define and validate assessment 1

Corresponding Author: Research Engineer, Northwestern Center for Advanced Surgical Education, Northwestern University, 303 E. Chicago Avenue Chicago, IL 60611; E-mail: [email protected]

536

L.H. Salud et al. / Toward a Simulation and Assessment Method

measures in greater depth, and to further refine training needs. To date, however, effective performance criteria for minimally invasive procedures, such as the cameraguided rigid bronchoscopy, has not been defined.

1. Objective In this article, we look at a sensor-enabled simulator that captures rigid bronchoscopy performance from novice and expert clinicians. Our goal is to determine if electronically captured bronchoscopic maneuvers can be analyzed to reveal insightful differences between expert and novice performance. By combining a commercially available bronchoscopy-teaching manikin with force-monitoring sensors, we aim to identify potentially critical assessment criteria for establishing competency in rigid bronchoscopy.

2. Methods 2.1. Participants The participants in this study were attendees at the 44th Annual Meeting of the Society of Thoracic Surgeons, held in Fort Lauderdale, FL. This meeting provides a forum for over 2,000 experts from around the world to discuss and present laboratory and clinical research concerning care of patients in need of heart, lung, esophageal, and other surgery for the chest. The conference is for individuals involved clinically or scientifically in thoracic and cardiovascular surgery. Clinician data were collected over a three-day period during which 38 clinicians visited a booth stationed in the exhibit hall and volunteered to perform a simulated rigid bronchoscopy with an endoscopic camera. The institutional review board approved this study. 2.2. The Rigid Bronchoscopy Simulator The rigid bronchoscopy simulator, as shown in Figure 1, is a newly developed task trainer that is instrumented internally with electronic sensors that can monitor and report forces on contact. A commercially available bronchoscopy model with normal anatomy of the mediastinum is typically used as a task trainer for various introductory procedures. For our study, we instrumented this model with sensors to detect possible abrasions from a rigid bronchoscope. Sensor locations were determined based on feedback from a preliminary pilot study in which both experts and novices performed the rigid bronchoscopy on the simulator. The six sensors were placed along the mucosal walls of the airway and esophageal, specifically, the tongue, the hypopharynx, the vocal cords, the upper trachea-esophageal junction, and the anterior trachea, Figure 2. The sensors are placed on the outside luminal structures to allow for smooth passage of the rigid bronchoscope. Despite their locations, the sensors are sensitive enough to pick up abrasion forces or contact pressures caused by bronchoscope maneuvers.

L.H. Salud et al. / Toward a Simulation and Assessment Method

537

Figure 1. Clinicians perform a rigid bronchoscopy on a sensor-instrumented simulator.

Figure 2. Force-monitoring sensors are placed in six locations of the airway and esophageal, specifically, the tongue, hypopharynx, vocal cords, the upper trachea-esophageal junction, the anterior trachea, and the esophageal.

The sensor-enabled device allows learners and instructors to visualize in real-time how much contact force is being applied on surrounding tissue while advancing and withdrawing the rigid bronchoscope. In the assessment mode, the computer screen is turned away from the user. While the bronchoscopy is being performed, individual performance data are collected. Sensor inputs are sampled at a rate of 30 Hz, and the outputs are captured and stored in named data files for off-line analysis. The device used in this study is a prototype and is not yet commercially available. The computer was used in assessment mode in this study.

2.3. Experimental Protocol Before conducting a camera-guided rigid bronchoscopy, each participant completed a demographics survey indicating years in practice, gender, and specialty. They were also asked to rate their experience in performing a rigid bronchoscopy on a 4-point Likert scale. The level of expertise was then divided into two groups: novice (none to minimal) and experienced (moderate to extensive). Using our sensor-enabled task trainer, electronic data were collected from the simulator as the participants performed a complete rigid bronchoscopy including, advancing the rigid bronchoscope through

538

L.H. Salud et al. / Toward a Simulation and Assessment Method

the airway, inspecting the carina and withdrawing the bronchoscope. Each participant was told in advance that there was a 90 percent occlusion of the airway with the intent to set time sensitive conditions. 2.4. Data Analysis In our previous work, we developed algorithms to extract key variables from our sensors to assess palpation skills [4]. Applying the same parameter extraction technique for rigid bronchoscopy, the simulator data were analyzed by using MATLAB version 7.10 data mining software (The Mathworks, Inc., Natick, MA, USA) to extract the following variables: (1) time to perform a complete rigid bronchoscopy, (2) maximum wall abrasion forces, and (3) number of areas (sensors) contacted during the procedure (an indirect measure of economy of motion and efficiency). Data analysis was performed by using SPSS version 18 (SPSS, Inc, Chicago, IL). Chi-square and analysis of variance were used to compare clinician performance between experts and novices. A p-value less than .05 was considered to be significant.

3. Results The participant group (N=38) included fourteen experts and twenty-four novices. The participants were primarily from thoracic and cardiothoracic specialties with an average of fifteen years in practice. All participants performed the procedure with the aid of the endoscopic camera and light source for visual guidance. We hypothesized that level of expertise would influence the time taken to complete the procedure and that experts would perform the task in a significantly shorter time than novices. Contrary to our expectations, results showed there were no significant differences between experts and novices in the time taken to complete the rigid bronchoscopy (45 seconds). However, the number of areas touched was higher on average for the novices (3.37 versus 2.93), although the difference was not significant. Figures 3 shows two distinct performance characteristics of one expert participant and one novice participant who performed rigid bronchoscopies on the same simulator. Each line graph, with its unique shade of grey, represents a specific anatomical location where a force-monitoring sensor is placed. The six areas are shown in the keys on the right of each line graph. The vertical axis denotes pressure units where 1 unit is equivalent to about 1.25 in pounds per square inch. The horizontal axis depicts time in seconds. In essence, the line graphs show that novices touch more area with greater pressures compared to experts. As shown in Table 1, there were no significant differences between the groups in maximum contact pressures from the bronchoscope at the tongue (experts: 2.18, novices: 3.71, p=.096), hypopharynx (experts: 6.67, novices: 6.64, p=.764), vocal cords (experts: .58, novices: .22, p= .319), anterior trachea (experts: .26, novices:.25, p=.964), or esophagus (experts: 2.04, novices: 2.51, p=.598). However, when comparing expert and novice maximum pressures on the trachea-esophageal junction, novice performers exerted significantly higher pressures with the bronchoscope at this junction than experts (experts: 1.71, novices: 3.31, p=.003). The sensor was located on the anterior portion of the esophageal wall, about five centimeters from the junction joining the trachea and the esophagus. Contact with this sensor is associated with a higher chance of an esophageal intubation.

539

L.H. Salud et al. / Toward a Simulation and Assessment Method

Figure 3. The novice waveform exhibits significantly more rigid bronchoscope manipulation activity about the tongue and the upper trachea-esophageal (TE) junction where the latter forces are repeated from the 10 to about 26-second mark. Entry into the airway may have occurred soon after the 16-second mark where there is a one-pressure unit abrasion at the vocal cords during this time. In contrast, close inspection of the expert reveals only two instances of a less than 3 pressure units of force applied on the tongue. Two seconds later, about one half a pressure unit is exerted about the upper TE junction. Both line graphs show an initial 3.5 pressure units on the hypopharynx. This is caused by the weight of the simulated patient’s head during bronchoscopy procedure.

Table 1. Data analysis results comparing experts and novice performers of the rigid bronchoscopy procedure. *The p-value comparing maximum pressures at the upper trachea-esophageal (TE) junction is less than .05. Performance Variables Time (seconds) Number of Areas Touched Maximum Pressure – Tongue Maximum Pressure – Hypopharynx Maximum Pressure – Vocal Cords Maximum Pressure – Upper TE Junction* Maximum Pressure – Ant. Trachea Maximum Pressure – Esophageal

Experts (N=14) 45.3 2.93 2.18 6.77 .58 1.71 .26 2.04

SD 19.4 1.07 2.26 1.38 1.33 1.31 .60 2.59

Novice (N=24) 45.9 3.37 3.71 6.64 .22 3.31 .25 2.51

SD 22.16 .92 2.87 1.31 .85 1.61 1.11 2.67

p .934 .184 .096 .764 .319 .003 .964 .598

540

L.H. Salud et al. / Toward a Simulation and Assessment Method

4. Discussion Comparisons between expert and novice performances reveal visually apparent differences during the rigid bronchoscopy procedure. The novice waveform exhibits significantly more rigid bronchoscope manipulation activity about the tongue and the upper trachea-esophageal (TE) junction where the latter forces are repeated from the ten to about the twenty six-second mark. Entry into the airway may have occurred soon after the sixteen-second mark where there is a pressure unit of abrasion force at the vocal cords during this time. In contrast, close inspection of the expert performances reveals only two instances of less than three pressure units of force applied to the tongue. Two seconds later, about one half a pressure unit is exerted about the upper TE junction. From this point forward, no contact pressures are reported. Both line graphs show an initial 3.5 pressure units on the hypopharynx. This is caused by the weight of the simulator head and its movement during the rigid bronchoscopy procedure. A correct rigid brocchoscopy entails finding the epiglottis and gently lifting it with the bronchoscope to visualize the vocal cords. The bronchoscope is then advanced through the cords and positioned in the airway [5]. Because vision through the bronchoscope is limited through its small beveled opening, it was difficult in our study for a novice to find the epiglottis. The novice users who could not find the epiglottis tended to advance the bronchoscope into the esophagus. These novices would realize that they were not in the airway. We observed that this realization occurred at varying depths within the esophagus. As such, novices would stop and reverse direction, repeatedly performing the lifting maneuver while slowly withdrawing the bronchoscope. This movement caused the bronchoscope to make contact with the sensor located at the trachea-esophageal junction. Some novices completely withdrew and reinserted the bronchoscope. A few novices were able to detect the epiglottis but proceeded to advance the instrument into the esophagus. One participant communicated that it was difficult to visualize the vocal chords and requested assistance. It is feasible to quantify differences between expert and novice performers of the rigid bronchoscopy procedure through the use of simulation that can capture electronics-based, sensor performance. Our pilot study suggests that bronchoscope abrasion forces applied to specific areas of the chest cavity can be quantified and that experts and novices may differ significantly. The fact that a significant difference in contact forces between expert and novice participants may be found at the upper trachea-esophageal junction suggests that novices may be finding difficulty setting themselves up for executing the critical step of visualizing the vocal cords, hence preventing a successful intubation. Task completion times did not differ significantly between experts and novices. The experts in our study were more accustomed to using a direct line of sight through the bronchoscope and revealed that they found the camera-guided technique a little more difficult.

5. Limitations This pilot study was intended to test the simulator’s ability to familiarize untrained learners with the basics of rigid bronchoscopy and investigate whether distinct novice performance characteristics are discoverable in relation to their expert guides.

L.H. Salud et al. / Toward a Simulation and Assessment Method

541

Additional sensor variables need to be defined in order to extract all possible measures of performance.

6. Conclusion Training for complex procedural skills is the main goal of residency training programs. The technical mastery of procedures is best achieved outside of the operating room. After completing training, a resident in the operating room should be more equipped to focus on higher-order, cognitive tasks such as surgical decision-making and situational awareness. By providing a validated, data-driven feedback mechanism for reporting learner performance against objective standards, a simulator may provide the ideal platform for practice opportunity. This kind of expert guidance—a specialized type of learning construction at the level of the anatomy—could pave the necessary pathway to skills transfer of the rigid bronchoscopy procedure. Validated outcomes from our proposed method could be used to establish critical assessment criteria for credentialing experts in rigid bronchoscopy.

7. Acknowledgement We wish to thank Abby Kaye her assistance in editing the manuscript. The data collection venue was supported by Karl Storz, Inc. We thank them for this support.

References [1] Kvale, P.A. and A.C. Mehta, Training bronchoscopists for the new era. Clin Chest Med, 2001. 22(2): p. 365-72, ix. [2] Ayers, M.L. and J.F. Beamis, Jr., Rigid bronchoscopy in the twenty-first century. Clin Chest Med, 2001. 22(2): p. 355-64. [3] Torrington, K.G., Bronchoscopy training and competency: how many are enough? Chest, 2000. 118(3): p. 572-3. [4] C.M. Pugh and P. Youngblood. “Development and Validation of Assessment Measures for a Newly Developed Physical Examination Simulator,” J Am Med Inform Assoc, vol. 9, no. 5, pp. 448-460, 2002. [5] Ernst, A., G.A. Silvestri, and D. Johnstone, Interventional pulmonary procedures: Guidelines from the American College of Chest Physicians. Chest, 2003. 123(5): p. 1693-717.

542

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-542

Use of Sensor Technology to Explore the Science of Touch Lawrence H. SALUD1. MS and Carla M. PUGH, MD, PhD Northwestern University, Department of Surgery, Feinberg School of Medicine Abstract. Two, world-renown researchers in the science of touch (Klatzky and Lederman) have shown that there are a set of reproducible and subconscious maneuvers that humans use to explore objects. Force measuring sensors may be used to electronically identify and quantify these maneuvers. Two sensored silicone breast models were configured to represent two different clinical presentations. One-hundred clinicians attending a local breast cancer meeting performed clinical breast examinations on the models, and their performance was captured using sensor-based data acquisition technology. We have found that Klatzy and Lederman’s previously defined touch maneuvers are used during the clinical breast examination and can be identified and quantified for the first time using sensor technology. Keywords. Haptics, Medical Simulation, Sensors, Learning Technologies, Simulation Support Systems, Evaluation/methodology, Human-centered computing

1. Introduction Extensive research by Klatzky and Lederman has shown that there is a set of reproducible and subconscious maneuvers that humans use to manually explore objects [1, 2]. The authors have established links between desired knowledge about objects and hand movements during haptic object exploration. Named “exploratory procedures (EPs)”, the hand movements are stereotyped movement patterns having certain characteristics that are invariant and highly typical when exploring various properties of an object, such as temperature and hardness. The EPs are comprised of lateral motion, pressure, static, unsupported holding, and enclosure, Figure 1(a). The lateral motion EP is used to explore an object that has texture. The pressure EP, which is defined as applying a normal force to an object, is used to explore hardness. The static EP is used to discover an object’s temperature and occurs when the hand drapes over the object’s surface, maximizing skin contact without an obvious effort. The unsupported holding EP occurs when the object is lifted away from any supporting surface and maintained in the hand without any effort to mold the hand to the object. This maneuver is associated with assessing an objects weight. The enclosure EP, which is used to judge an object’s global shape and size, involves molding the hand to object contours. Finally, the contour following EP is used to encode precise shape information and generally takes the form of traversing along edges of an object with the fingertips. 1

Corresponding Author: Research Engineer, Northwestern Center for Advanced Surgical Education, Northwestern University, 303 E. Chicago Avenue Chicago, IL 60611; E-mail: [email protected]

L.H. Salud and C.M. Pugh / Use of Sensor Technology to Explore the Science of Touch

543

Sensor-enabled task trainers and virtual reality devices have been designed to study palpation techniques and have been used to discover important clinical performance characteristics [3, 4]. For example, Pugh et al. have shown that using force sensors embedded in task trainers make it possible to capture and analyze handson performance [5]. In a 2006 publication, eleven sensors were embedded in a simulated breast model to capture clinician performance during clinical breast examinations [6]. Sensor measurements were analyzed to quantify an individual’s clinical breast examination (CBE) time, the number of sensors touched, the maximum palpation pressure and the mean palpation frequency. Furthermore, palpation was measured from a single sensor by plotting touch forces over time. Figures 1 (b) and (c) reveal differences between “tapping” and “rubbing” – two terms used by Pugh et al. to describe the palpation maneuvers being observed. Differences between two classes of tapping and rubbing are shown as measurable. One class is tapping or rubbing in a single area of the sensor. The other class is tapping or rubbing in multiple areas of the sensor. It is now clear that these maneuvers may be closely related to Klatzky and Lederman’s EPs.

Figure 1. Klatzky and Lederman have shown that there are six reproducible and subconscious maneuvers that humans use to detect objects (a). Force measuring sensors may be used to electronically identify these movements. Forces plotted over time illustrate differences between tapping (b) and rubbing (c).

In addition, there is an emerging body of work in medicine and engineering using technology to better understand touch [7-10]. Remote sensing and manipulation in teleoperated robotics, haptic-based mobile devices, and human-computer interaction have been the predominant focus areas in this body of work. While a by-product of many of the technologies developed may provide mechanisms to measure EPs, no one has yet to establish an association between Klatzky and Lederman’s video recorded observations and electronically measured palpation techniques.

544

L.H. Salud and C.M. Pugh / Use of Sensor Technology to Explore the Science of Touch

2. Theoretical Framework and Hypothesis We hypothesize that electronic capture of hands-on clinical skills in performing theduring a CBE may be used to identify Klatzky and Lederman’s EPs. By embedding sensors on two simulated breast models, we expect to map at least two of the EPs, lateral motion and pressure, to our sensor measurements. Additionally, we hypothesize that contour following and static pressure are observed when clinicians are palpating a mass or abnormality. The remaining two EPs are not applicable to the breast examination as only the fingertips are used. Moreover, we propose that the presence of “tapping” and/or “rubbing” during a CBE could be characteristic of at least two of the six EPs, namely lateral motion and pressure. By finding single and multiple area tapping and rubbing, we suggest a starting point into electronically deciphering the pressure EP (tapping) and lateral motion EP (rubbing).

3. Methods and Materials 3.1. Participants The participants in this study were 102 attendees at the 7th Annual Lynn Sage Breast Cancer Symposium, held in Chicago, IL. The Lynn Sage Breast Cancer Symposium provides a forum for discussing and presenting laboratory and clinical research concerning the care of patients with breast cancer. The conference is for individuals involved clinically or scientifically in diagnostic and therapeutic radiology, oncology, surgery, gynecology, family practice, and genetics. Each year, this conference brings together over 1,200 experts from around the world to exchange ideas and present research in an effort to further the knowledge about breast cancer within the medical community. After obtaining institutional review board approval, clinician data were collected over a 4-day period. One hundred two clinicians visited a booth stationed in the exhibit hall and volunteered to perform a full CBE on each of 2 silicone breast simulators. 3.2. The Breast Examination Simulators The breast examination simulator is a newly developed task trainer that is instrumented internally with several electronic sensors. The breast models that are part of the simulator can be reconfigured to represent various clinical presentations. Two different presentations were used for this study. Simulator A consisted of a dense breast with a firm 2-cm mass in the upper inner quadrant (UIQ), Figure 2 (a). Simulator B was a dense, premenopausal breast with thickened breast tissue in the upper outer quadrant (UOQ) and the inframammary ridge area, Figure 2 (b). Sensor inputs are sampled at a rate of 30 Hz, and the outputs are captured and stored in named data files for off-line analysis. Figure 2 (c) shows a diagram of sensor placements for the breast models. One of the eleven sensors in Simulator A is located in the UIQ just below the firm mass. While the examination is being performed, individual performance data are collected. The simulators used in this study are prototypes and are not yet commercially available.

L.H. Salud and C.M. Pugh / Use of Sensor Technology to Explore the Science of Touch

545

Figure 2. Simulator A contains a 2 cm firm mass (a) at the upper inner quadrant (UIQ) while Simulator B is configured as a dense breast with thickening (b). Both simulators are instrumented with eleven force measuring sensors on a solid planar support platform. Sensor locations are shown in (c).

3.3. Experimental Protocol Before examining the two simulators, clinicians completed a background survey indicating specialty. Clinicians were asked to perform complete CBEs and were told this was an annual examination with no complaints. Each of the 102 clinicians evaluated 2 simulators for a total of 204 examinations. Electronic data were collected directly from each simulator as the participants performed simulated CBEs. After examining each simulator, participants were asked to document their clinical findings on an assessment form. Specifically, the participants were asked to describe probable diagnoses of any discovered pathologies. Force measurements of “tapping” and “rubbing” from a single sensor was plotted over time and graphed. Further, a representative sample of one participant’s CBE was graphed for both the firm mass sensor of Simulator A and a similarly located sensor on Simulator B. Qualitative, visual interpretations were performed on the graphs to search for the presence of pressure and lateral motion EPs using the “tapping” and “rubbing” waveforms of Figures 1 (a) and (c) as comparative guidance. 3.4. Data Analysis Surveys were coded and analyzed using descriptive and comparative inferential statistics. The simulator data were analyzed using MATLAB 5.1 data mining software (The Mathworks, Inc., Natick, MA, USA) to extract the following variables: (1) time to perform a complete examination, (2) frequency each anatomic area was palpated, (3) maximum pressure used when palpating each area, and (4) number of areas (sensors) palpated during the examination (an indirect measure of thoroughness). Data analysis was performed by using SPSS version 10 (SPSS, Inc, Chicago, IL). A t-test was used to compare clinician performance on the 2 clinical presentations. A p-value less than .05 was considered to be significant.

4. Results The maximum pressure was higher for Simulator A than it was for Simulator B, Table 1. In contrast, the number of sensors touched and the mean palpation frequency were significantly higher for Simulator B than for Simulator A.

546

L.H. Salud and C.M. Pugh / Use of Sensor Technology to Explore the Science of Touch

As shown in Figure 3, multiple area tapping was used mostly on the sensor just below the firm mass in Simulator A. Klatzky and Lederman’s work show that this is the preferred technique (the pressure EP) to assess hardness, as in a solitary breast mass. This correlates with the clinical presentation of Simulator A. The same participant performed only multiple area rubbing on Simulator B. Klatzky and Lederman’s work show that this is the preferred technique (the lateral motion EP) to assess texture, as in thickened breast tissue. This correlates with the clinical presentation of Simulator B.

Figure 3. A participant’s CBE palpation pressure is recorded electronically, and one of the eleven sensors of both simulators A and B is plotted as shown. The sensor is located at the UIQ on both simulators, and under the mass of simulator A as shown in Figure 2(a) and (c). After 12 seconds, the participant applies multiple area tapping with increasing pressure on simulator A. For simulator B the participant performs what looks to be rubbing at about the 4 and 23 second mark.

Table 1. Mean Clinician (N=102) Breast Exam Performance Characteristics between Simulator A and B Performance Variable

Simulator A (Dense breast 2cm firm mass)

Simulator B (Dense breast w/ thickening)

Time (seconds)

40.5

42.28

No. of sensors touched

4.54*

6.16

5.35

5.12

17.03*

24.01

87%

68%

Maximum pressure

1

Mean palpation frequency

2

Percent correct 1 – 1 PU (Pressure Unit) is about 1.25psi 2 – measure for palpations 30 times/second * – p < .01 between simulators

5. Discussion The results suggest that when comparing Simulator A to Simulator B, the exploratory procedures appear to correlate with the clinical findings of the breast model. When there is a solitary, firm nodule (hardness objects property), the maximum pressure is

L.H. Salud and C.M. Pugh / Use of Sensor Technology to Explore the Science of Touch

547

higher and correlates with use of the pressure EP to detect hardness. Not only with the pressure higher, but it was used in multiple areas within a small radius, which may hint to contour following to detect the size of the mass. For Simulator B where there was a denser breast model with thickened breast tissue (texture object property) more rubbing occurs as evidenced by comparatively higher frequencies and higher numbers of sensors touched. This correlates with the use of the lateral motion EP to detect texture.

6. Future Work The data in this study was gathered using force sensors that are limited to reporting direct, normal forces. We aim to develop and use shear force sensors to help us delineate not only between pressure and lateral motion, but also between sub-classes of motion such as circular versus static contact. Moreover, we aim to develop the ability to delineate between experts and novice performers such that we may be able to understand how clinical exam maneuvers effect diagnostic accuracy at this more granular and electronically measured level.

7. Conclusion Pressures are higher in the presence of a firm mass while wide area lateral palpation movements correlate more closely with a breast model with texture. This is consistent with Klatzky and Lederman’s findings. Moreover, by graphically interpreting single and multiple area tapping and rubbing, we have proposed a window into quantitatively defining the pressure and lateral motion EPs.

8. Acknowledgement We wish to thank Abby Kaye and Jonathan Salud for their assistance in editing the manuscript. We also wish to thank Dr. Eric Volkman, Dr. Timothy Rankin and Alec Peniche for assisting in the collection and preparation of the dataset. This research was supported by the Augusta Webster Educational Innovation Grants Program at Northwestern University’s Feinberg School of Medicine. Figure 1 (a) was revised from Lederman, S.J., & Klatzky, R.L. (1987). Hand movements: A window into haptic object recognition. Cognitive Psychology, 19(3), 342-368.

References [1] Lederman, S.J. and R.L. Klatzky, Hand movements: a window into haptic object recognition. Cogn Psychol, 1987. 19(3): p. 342-68. [2] Klatzky, R.L. and S.J. Lederman, Stages of manual exploration in haptic object identification. Percept Psychophys, 1992. 52(6): p. 661-70. [3] Pugh, C.M., et al., Use of a mechanical simulator to assess pelvic examination skills. JAMA, 2001. 286(9): p. 1021-3. [4] Forrest, N., S. Baillie, and H.Z. Tan, Haptic Stiffness Identification by Veterinarians and Novices: A Comparison. World Haptics 2009: Third Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Proceedings, 2009: p. 646-651, 656.

548

L.H. Salud and C.M. Pugh / Use of Sensor Technology to Explore the Science of Touch

[5] Pugh, C.M. and P. Youngblood, Development and validation of assessment measures for a newly developed physical examination simulator. J Am Med Inform Assoc, 2002. 9(5): p. 448-60. [6] Pugh, C.M., et al., A simulation-based assessment of clinical breast examination technique: do patient and clinician factors affect clinician approach? Am J Surg, 2008. 195(6): p. 874-80. [7] Feller, R.L., et al., The effect of force feedback on remote palpation. 2004 Ieee International Conference on Robotics and Automation, Vols 1- 5, Proceedings, 2004: p. 782-788, 5306. [8] Okamura, A.M., Haptic feedback in robot-assisted minimally invasive surgery. Curr Opin Urol, 2009. 19(1): p. 102-7. [9] Mahvash, M., et al., Force-feedback surgical teleoperator: Controller design and palpation experiments. Symposium on Haptics Interfaces for Virtual Environment and Teleoperator Systems 2008, Proceedings, 2008: p. 465-471, 480. [10] Gerling, G.J., et al., The Design and Evaluation of a Computerized and Physical Simulator for Training Clinical Prostate Exams. Ieee Transactions on Systems Man and Cybernetics Part a-Systems and Humans, 2009. 39(2): p. 388-403.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-549

549

Real-Time “X-Ray Vision” for Healthcare Simulation: An Interactive Projective Overlay System to Enhance Intubation Training and Other Procedural Training Joseph T. SAMOSKYa,b,c,1, Emma BAILLARGEONb, Russell BREGMANb,c, Andrew BROWNb, Amy CHAYAb, Leah ENDERSb, Douglas A. NELSONb,c, Evan ROBINSONc, Alison L. SUKITSb and Robert A. WEAVERb,c a Department of Anesthesiology, University of Pittsburgh b Department of Bioengineering, University of Pittsburgh c Simulation and Medical Technology R&D Center, University of Pittsburgh

Abstract. We have developed a prototype of a real-time, interactive projective overlay (IPO) system that creates augmented reality display of a medical procedure directly on the surface of a full-body mannequin human simulator. These images approximate the appearance of both anatomic structures and instrument activity occurring within the body. The key innovation of the current work is sensing the position and motion of an actual device (such as an endotracheal tube) inserted into the mannequin and using the sensed position to control projected video images portraying the internal appearance of the same devices and relevant anatomic structures. The images are projected in correct registration onto the surface of the simulated body. As an initial practical prototype to test this technique we have developed a system permitting real-time visualization of the intra-airway position of an endotracheal tube during simulated intubation training. Keywords. Projective augmented-reality display, visualization, non-contact position sensing, intubation training, human-computer interaction, user interfaces.

Introduction A basic but significant limitation of training on real patients is that human bodies are mainly visually opaque. For many procedures—endotracheal tube insertion, Foley catheter placement, bronchoscopy—there may be advantages if a trainee could see what is happening inside the body as he or she manipulates a tool or device outside the body. This could enhance the trainee’s development of accurate and memorable cognitive and spatial models of the internal consequences of their external actions. We can accomplish this in simulation-based training environments by means of augmented reality techniques: via computer-generated visual displays we can make seen what is normally hidden from view in the real world of clinical medicine. We developed this approach in a prototype augmented-reality display of endotracheal tube (ET) position. 1 Corresponding author: Joseph T. Samosky, Director, Simulation and Medical Technology R&D Center, University of Pittsburgh, 230 McKee Place, Suite 401, Pittsburgh, PA 15213; E-mail: [email protected].

550

J.T. Samosky et al. / Real-Time "X-Ray Vision" for Healthcare Simulation

Figure 2 Neodymium disk magnet fixed transversely in lumen near tip of endotracheal tube.

Figure 1 A linear array of Hall effect sensors is affixed to the trachea and right mainstem bronchus of the fullbody simulator. A single Hall effect sensor detects esophageal intubation.

Figure 3 One frame from digital video of ET tube being inserted into coronally hemisected goat trachea with chromakeyed background converted to black.

Our approach aims to avoid the potential encumbrance of head-mounted displays or mirrors interposed between user and workspace, and differs from prior work in projecting anatomy onto a mannequin (such as [1, 2]) in that the IPO system senses the real-time position of a medical device inserted into or manipulated within the simulated body and the projected video images include representations of the device correlated with the position and motion of the actual device. We employ a sensed physical variable to control the playback position of a digital video to generate dynamic simulated images of the 1-dimensional motion of a device, tool or object.

Methods and Materials The IPO system augments a standard full-body human simulator (SimMan®, Laerdal Medical AS) with: (1) a DLP projector (Dell M209X) mounted above it on an adjustable support arm, and (2) a non-contact position sensing system (Figures 1 and 2) that measures the depth of insertion of an endotracheal tube. Control software written in LabVIEW (National Instruments) uses the measured position of the ET tube to select the playback frame of a pre-recorded digital video (Figure 3) that presents a top-down view of an endotracheal tube being inserted into a partially cut-away trachea. For the prototype, a hemisected goat trachea was filmed against a green chroma-keyed background. The background was then post-processed to black. A non-contact method for measuring ET tube depth was desired to enable free manipulation of the ET tube. We designed an array of 7 Hall effect sensors, affixed to the mannequin’s trachea, to sense the position of a small neodymium disk magnet fixed in the lumen near the tip of a standard ET tube. A custom algorithm converts the sensor outputs to a measurement of linear position. An additional sensor attached to the esophagus detects the clinically important error of esophageal intubation. Software controls playback of the digital video via an ActiveX interface to Windows Media Player. Playback position is calibrated and synchronized to the measured tip position of the real ET tube. User control of system functions is performed via a Wiimote.

J.T. Samosky et al. / Real-Time "X-Ray Vision" for Healthcare Simulation

551

Figure 4 Real-time augmented reality display of endotracheal tube position within trachea. An overhead DLP projector projects image correlated to actual tube position onto neck and thorax of simulator. Trainee performance metrics such as intubation depth and time are projected for objective assessment and feedback. System functions are controlled by a Wiimote, employed as a simple, compact wireless control interface.

Results The system successfully creates the illusion of a “see through” view through the anterior chest wall during simulated intubation training (Figure 4). The non-contact intubation depth sensing system exhibited an average absolute accuracy of 1.1 mm +/1.9 mm (s.d.). The position of the projected ET tube matched the actual ET tube position to within 2 mm (average across 13 depths from 14 to 26 cm). The system also accurately detected esophageal intubation, alerting the user via an auditory alarm.

Discussion Using the IPO system trainees can clearly and immediately see important errors such as mainstem bronchus intubation in a way that would otherwise be obscured. We hypothesize that such augmented visual feedback can enhance the development of memorable mental models of procedures, and that proximate feedback on errors is superior to delayed debriefing in developing accurate psychomotor skills efficiently. Extensions of the technique to other procedures are in development (including Foley catheterization and pulmonary artery catheterization). Experiments with clinical trainees are planned to examine the ultimate training effectiveness of the system. References [1] [2]

Kondo D, and Kijima R. Poster: Free Form Projection Display: Virtual Image Located Inside Real Object. IEEE Symposium on 3D User Interfaces. Reno, Nevada. March 2008. Kondo D, Kijima R, and Takahashi Y. Dynamic Anatomical Model for Medical Education using Free Form Projection Display. Proc. 13th International Conference on Virtual Systems and Multimedia. Brisbane, Australia. September 2007: 142-149.

552

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-552

Toward a Comprehensive Hybrid Physical-Virtual Reality Simulator of Peripheral Anesthesia with Ultrasound and Neurostimulator Guidance Joseph T. SAMOSKYa,b,d,1, Pete ALLENb, Steve BORONYAKb, Barton BRANSTETTERc, Steven HEINb, Mark JUHASb, Douglas A. NELSONb,d, Steven OREBAUGHa, Rohan PINTOb, Adam SMELKOb, Mitch THOMPSONb and Robert A. WEAVERb,d a Department of Anesthesiology, University of Pittsburgh b Department of Bioengineering, University of Pittsburgh c Department of Radiology, University of Pittsburgh d Simulation and Medical Technology R&D Center, University of Pittsburgh

Abstract. We are developing a simulator of peripheral nerve block utilizing a mixed-reality approach: the combination of a physical model, an MRI-derived virtual model, mechatronics and spatial tracking. Our design uses tangible (physical) interfaces to simulate surface anatomy, haptic feedback during needle insertion, mechatronic display of muscle twitch corresponding to the specific nerve stimulated, and visual and haptic feedback for the injection syringe. The twitch response is calculated incorporating the sensed output of a real neurostimulator. The virtual model is isomorphic with the physical model and is derived from segmented MRI data. This model provides the subsurface anatomy and, combined with electromagnetic tracking of a sham ultrasound probe and a standard nerve block needle, supports simulated ultrasound display and measurement of needle location and proximity to nerves and vessels. The needle tracking and virtual model also support objective performance metrics of needle targeting technique. Keywords. Peripheral nerve block, mixed reality, hybrid reality, virtual reality, human computer interaction, 3D segmentation, MRI, stereolithography

Introduction A variety of approaches have been pursued to simulate the multiple and varied aspects of peripheral nerve block procedures, including physical models, models with embedded conductive “nerves” [1], computer-simulated ultrasound [2], VR systems with commercial haptic interfaces [3], and novel haptic devices to simulate needle force profiles during insertion [4]. A fundamental design decision for simulation systems is the choice of physical or virtual models. We believe each has merits and limitations for perceptual display and 1 Corresponding author: Joseph T. Samosky, Director, Simulation and Medical Technology R&D Center, University of Pittsburgh, 230 McKee Place, Suite 401, Pittsburgh, PA 15213; E-mail: [email protected].

J.T. Samosky et al. / Toward a Comprehensive Hybrid Physical-Virtual Reality Simulator

553

interaction. In this work we are investigating a hybrid approach to emulate the salient aspects of peripheral anesthesia. Our initial prototype focuses on development of a training system for brachial plexus blockade.

Materials & Methods Isomorphic Physical and Virtual Models. A high-resolution 3D MRI scan was obtained of a subject’s arm while encased in a silicone mold and fiberglass support shell. The mold was used to create a silicone model of the skin of the arm and axilla. Nerves, blood vessels and other relevant anatomy were segmented from the MRI scan (using MIMICS, Materialise NV) to create a virtual model that was registered with the physical model. The virtual model was also used to fabricate physical models of fascial planes via stereolithography. These components of the physical model provide the characteristic “pop” felt during needle passage through the fascia. Needle and US Probe Tracking. A 3D electromagnetic tracking system (Aurora, NDI Inc.) was employed to measure the pose of a standard block needle (with attached miniature 5 DOF sensor) and sham ultrasound probe (with a 6 DOF sensor). Anesthetic Syringe Simulator. A subsystem consisting of a flow sensor, solenoid valves and fluid reservoirs enable the ability to draw back blood simulant if the needle tip is determined to be within a (virtual) blood vessel, indicating the need to reposition the needle prior to injection. Fluid resistance can be controlled during injection and plunger pullback depending on the sensed location of the needle tip with respect to structures in the virtual model such as blood vessels, soft tissue or nerve fascicles. Simulated Ultrasound (US). We explored initial capability to track the pose of the US probe, specify an imaging plane through the virtual model and display an approximation to an US image via reformatting of 3D model data. Twitch Display. A set of mechatronic actuators and a cable-drive system was incorporated into the physical arm model to provide visual feedback of muscle twitch to the user when any of the four principal nerves of the brachial plexus are stimulated. Neurostimulator Interface. Any standard clinical neurostimulator can be used with the system. An electronic interface senses the electrical output of the neurostimulator. The measured rate and intensity of the current pulses are then combined with needle-tonerve distance data to compute twitch output quality and intensity.

Results A translucent cylindrical silicone phantom and corresponding virtual model were constructed to test the system (Figure 1 and Figure 1 inset). We verified physicalvirtual registration and that needle tip contact with soft tissue, nerve or vessels was correctly identified. The syringe simulator provided blood simulant during pull-back when the needle tip contacted a virtual vessel and varied flow resistance depending on tip location. Metrics computed on needle tip trajectories included total path length and instantaneous and average velocity and acceleration. The full extremity mechatronic model (Figure 2) displayed four- and two-finger twitch, elbow extension and flexion, and wrist adduction as well as graded response to intensity. Figure 3 shows the corresponding isomorphic virtual model derived from the 3D MRI data, with segmented skin, fascia, brachial artery and vein, and nerves of the brachial plexus.

554

J.T. Samosky et al. / Toward a Comprehensive Hybrid Physical-Virtual Reality Simulator

Sham ultrasound probe and block needle, each with tracking sensor Cylindrical phantom “limb” model

Monitor displaying instructor interface Monitor displaying learner interface Syringe simulation subsystem

Aurora electromagnetic field generator Anesthetic syringe

Figure 1 Peripheral nerve block simulator test system: tracking, display and syringe. Inset: Detail of test phantom with tracked simulated US probe and tracked block needle.

Physical arm model fabricated via lifecasting and stereolithography Standard commercial neurostimulator Actuators and driver electronics

Figure 2 Mechatronic arm with twitch actuators and neurostimulator interface.

Figure 3 MRI-derived virtual model isomorphic to physical model, with segmentation of the brachial artery and vein, and the nerves of brachial plexus.

Discussion We are currently evaluating the fidelity of needle insertion haptics and further developing the US simulation algorithms. We are also exploring augmented visual feedback to the trainee and automated proficiency metrics based on tracked needle trajectories, needle targeting accuracy, and sensed injection rate and volume. References [1] [2] [3] [4]

Eastwood, CB and Moore, DL. A Simple, Near-Ideal Simulation Model for Ultrasound-Guided Regional Anesthesia. SPA/AAP Pediatric Anesthesiology 2010 Winter Meeting (poster P106). Zhu Y, Magee D, Ratnalingam R, and Kessel D (2007). A Training System for Ultrasound-Guided Needle Insertion Procedures. MICCAI 2007: 566-574 Ullrich S, Frommen T, Rossaint R, and Kuhlen T (2009). Virtual Reality-based Regional Anaesthesia Simulator for Axillary Nerve Block. Medicine Meets Virtual Reality 17, 2009: 392-394 Lim YJ, Valdivia P, Chang C, and Tardella N (2008). MR Fluid Haptic System for Regional Anesthesia Training Simulation System. Medicine Meets Virtual Reality 16, 2008: 248-253.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-555

555

A Fixed Point Proximity Method for Extended Contact Manipulation of Deformable Bodies with Pivoted Tools in Multimodal Virtual Environments Ganesh SANKARANARAYANAN a, Zhonghua LU b and Suvranu DE a,1 a Department of Mechanical, Aerospace and Nuclear Engineering, Rensselaer Polytechnic Institute, USA b Intelligent Manufacturing and Control Institute, Wuhan University of Technology, China

Abstract. In the real world, tools used for manipulation are pivoted with specialized tips for specific functions including grasping and cutting. Manipulating deformable virtual objects with them involves maintaining extended contact, which is difficult due to the variations in applied force. Our method consists in selecting a fixed set of points on the jaws of a pivoted tool, and placing them either equidistant or according to the geometry of the tool. Vertex and triangle proximities are calculated for each of the interacting deformable objects for collision detection. This method was successfully tested in a surgical simulation scenario where a deformable omental fat model was grasped and retracted while maintaining full contact with the pivoted tool tip at all times. .Keywords. Collision detection, Haptics, Virtual environments

Introduction Haptic interaction of deformable bodies is a difficult problem because of the computational time requirements for collision detection and deformation calculations. For realistic contact and response it is often desired to model the tool as a line (“ray”) at the expense of computation time for collision. In real world interactions the tools are made for special purposes and they often have pivoted jaws for grasping. Such tools often have extended contact along their jaws due to squeezing force applied to the tool handles for tighter contact. Interaction of such tools can still be modeled as a line segment for rigid contacts whereas the same is not true for deformable objects. The major problem in such interactions is the extended contact of the tool with the model, and the collision detection and response between the pivoted jaws and the deforming objects. Collision detection is a well known problem in the field of interactive computer graphics and haptics. In collision detection, interpenetration of source and target 1

Suvranu De, Associate Professor, Department of Mechanical, Aerospace and Nuclear Engineering, Rensselaer, Polytechnic Institute, Troy, NY, USA. Email: [email protected]

556

G. Sankaranarayanan et al. / A Fixed Point Proximity Method

objects is constantly checked and reported back for subsequent action. Various types of collision checking methods exist and we refer to surveys [1] and [2] for more details regarding collision detection. In [3] a ray-based haptic rendering method was introduced in which the haptic probe was modeled as a line and collision with convex objects was performed using hierarchical bounding boxes. In [4] a fast and efficient line based collision detection method known as dynamic point was introduced for haptic based applications. For deformable objects, modeling of tool tissue interactions is modeled using a variety of techniques. We refer to [5] for a detailed review in this area. In this paper, we introduce a fixed point proximity (closest distance) algorithm specifically for the interactions between a pivoted tool and triangular meshes of deformable objects in haptic enabled virtual environments. This method is based on a fixed set of carefully sampled points along the pivoted tool jaws whose proximity to the nearby triangular meshes are constantly updated in real-time for haptic interactions. The vertex and triangular neighborhood information in a mesh and the spatial and temporal coherence is used to update the proximities, which gives a near constant time complexity for arbitrary mesh sizes for a fixed set of samples points. We show the effectiveness of this method in a surgical simulation scenario as a case study.

1. Materials and Methods The fixed point proximity method consists of a preprocessing step and an online computation step during which the proximities are updated at near constant time. The preprocessing step consists of selection of fixed points on the pivoted tool tip for collision detection and subsequent application of constraints to the deformable model and haptic response to the user. The selection criteria include the shape of the pivoted tool jaws and the mesh density of the interacting deformable objects. For coarser meshes, fewer points would be sufficient. Once the points are chosen, an initial distance check is performed from each of the points in the tool jaw to the deformable meshes to compute proximity and updated in a database. Two types of proximities can be computed for each mesh – a proximity to the closest triangle surface PijT (Figure 1a) and a proximity to the closest vertex PijV (Figure 1b). For coarser mesh, the triangle proximity would provide a better approximation of the interaction than the vertex proximity and for denser meshes; vertex proximity would itself be enough for realistic interactions. The vertex proximity can also be used for non triangular meshes (example surface voxels of volumetric models). For each of the meshes in the scene, the vertex and triangular neighborhoods are computed and stored in a database. It is reasonable to assume [4, 6] that at haptic sampling rates, the vertex and triangle proximities vary only minimally between two successive frames. Therefore, the precomputed triangle and vertex neighborhood information is used to update the proximities by checking the nearest neighbors at runtime. Collision is detected when the distance between the proximity and the fixed points are less than a predetermined small tolerance value. Once collision is reported, constraints are applied to the deformable mesh at the vertices of the contact triangle or at the contact vertex when using vertex proximities. Reaction force is then applied to the user based on the computed force acting at the contact vertex or the triangle of the model.

G. Sankaranarayanan et al. / A Fixed Point Proximity Method

(a)

557

(b)

Figure 1. Pivoted tool jaws with four fixed points for each jaw. (a) Triangle proximity and triangle neighbors (b) Vertex proximity and vertex neighbors.

2. Results The fixed point proximity method was applied to a surgical simulation scenario where a surgical grasper was used to interact with an omental fat in the upper region. Both grasping and scooping motion was achieved successfully with our method. The testing scenario consisted of organs of the upper peritoneal region created for a Laparoscopic Adjustable Gastric Banding (LAGB) simulator. In addition to the organ geometry, volumetric models of omental fat were created and simulated using position based dynamics [7]. The implementation is robust and utililizes graphical processing units (GPUs) for computational efficiency. The PHANToM Omni haptic interface device was used for force feedback. The buttons in the Omni controlled the opening/closing of the tool jaws. Triangle proximity was used for collision detection while interacting with the fat. There were a total of 1581 triangles faces in the mesh. A total of four evenly spaced fixed points were attached to each of the jaws of the surgical tool (Figure 2a). Whenever the proximity distance was less than the tolerance value, contact was established between the fixed point and the closest triangle. The opening/closing of the jaws provided two states for simulation. In the closing state, squeezing and manipulation of the fat tissue was made possible by directly applying position constraints to the tetrahedral vertex nearest to the contacted surface triangle. Subsequently, the position and orientation changes in the rest length of the interacting tetrahedra were computed and the corresponding force vector was applied to the haptic device. In the opening state of the tool, the position constraints applied to the fat were removed allowing them to return to their original configuration. Figure 2b shows a snapshot from the simulator where the fat was scooped by the surgical grasper. The fixed point based proximity method enabled the fat to be scooped by the tool in a realistic manner, similar to what would be observed in a real surgical video. In figure 2b, one can observe the bulging of the fat as it is squeezed by the grasper tool.

558

G. Sankaranarayanan et al. / A Fixed Point Proximity Method

(a)

(b)

Figure 2. (a) Surgical grasper tool (rendered in wireframe) with four equally spaced points for the pivoted tip and the corresponding proximity lines. (b) Grasping and pulling up the omental fat using a surgical grasper.

Figure 3 shows a plot of collision detection time for various mesh sizes and three sets of fixed points half of which were on each jaw. The scene consisted of three meshes, a liver, stomach and omental fat. Only the fat mesh was tessellated with increasing number of triangles. The total number of triangles after tessellation was 19404, 26656, 45472 and 67182 respectively. The timing plot clearly shows that for a fixed set of points, the fixed point proximity method has a constant time of computational complexity. During the timing trials, the vertex and triangle neighborhood information along with initial distance check was computed and it ranged from a minimum of 24 to a maximum of 40 seconds, which is quite low for collision detection preprocessing times.

Figure 3. Computation time for collision detection for three different sets of fixed points with various number of triangles in the scene.

3. Conclusions In this work we proposed a new method called the fixed point proximity to specifically address the problem of extended contact manipulation of deformable objects with

G. Sankaranarayanan et al. / A Fixed Point Proximity Method

559

pivoted tools. By using vertex and triangle neighborhood information, the collision detection between the points on the pivoted tool jaws and the model was computed at haptic rates. Position constraints were used as the collision response and applied directly to the tetrahedral nodes nearest to the contact surface of the tool to the deformable model. We also showed a successful implementation of this method in a realistic surgical simulation scenario that involved manipulation of highly deformable fat tissues. Since our method requires local neighborhood information for fast updates of the proximities, any change in topology would require some computational time to update the neighborhood structure. We plan to use more efficient methods for updating the neighborhood structure to enable cutting and other changes to topology during the interactions. We also plan to test this method on more complex pivoted tool structure which cannot be approximated by straight line.

Acknowledgements The authors gratefully acknowledge the support of this NIH/NIBIB through grant # R01EB005807.

References [1] [2]

[3]

[4]

[5] [6]

[7]

P. Jimnez, F. Thomas, and C. Torras. 3D Collision Detection: A Survey.Computers and Graphics, 25(2):269–285, Apr 2001. M. Teschner, S. Kimmerle, B. Heidelberger, G. Zachmann, L. Raghupathi, A. Fuhrmann, M.-P. Cani, F. Faure, N. Magnenat-Thalmann, W. Strasser, and P. Volino. Collision detection for deformable objects. Computer Graphics forum, 24(1):61–81, mar 2005. C.-H. Ho, C. Basdogan, and M. A. Srinivasan. Ray-based haptic rendering:Force and torque interactions between a line probe and 3d objects in virtual environments. I. J. Robotic Res., 19(7):668– 683, 2000. Maciel A, De S. An efficient dynamic point TM algorithm for line-based collision detection in real time virtual environments involving haptic. In Computer Animation and Virtual Worlds, volume 19(2), pages 151-163. Misra, S., Ramesh K. T., Okamura A. M., "Modeling of Tool-Tissue Interactions for Computer-Based Surgical Simulation: A Literature Review" Presence 17(5), 463-491, 2008. J. D. Cohen, M. C. Lin, D. Manocha, and M. K. Ponamgi. ICOLLIDE: An interactive and exact collision detection system for large-scale environments. In Proceedings of the 1995 Symposium on Interactive 3D Graphics, pages 189–196, 1995. M. Müller, B. Heidelberger, M. Hennix, and J. Ratcliff, Position Based Dynamics, 3rd Workshop in Virtual Reality Interaction and Physical Simulation, VRIPHYS, 2006.

560

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-560

Collision and Containment Detection between Biomechanically Based Eye Muscle Volumes a

Graciela SANTANA SOSAa and Thomas KALTOFEN a,1 Research Unit Medical-Informatics, RISC Software GmbH, Hagenberg, Austria

Abstract. Collision and containment detection between three-dimensional objects is a common requirement in simulation systems. However, few solutions exist when exclusively working with deformable bodies. In our ophthalmologic diagnostic software system, the extraocular eye muscles are represented by surface models, which have been reconstructed from magnetic resonance images. Those models are projected onto the muscle paths calculated by the system’s biomechanical model. Due to this projection collisions occur. For their detection, three approaches have been implemented, which we present in this paper: one based on image-space techniques using OpenGL, one based on the Bullet physics library and one using an optimized space-array data structure together with software rendering. Finally, an outlook on a possible response to the detected collisions is given. Keywords. Collision detection, containment detection, ophthalmology, biomechanical model, deformable surface models, 3D muscle, software system

Introduction The main goal of the work described in this paper is the implementation of a collision and containment detection engine for SEE++, an ophthalmologic diagnostic tool developed within the research project SEE-KID (Software Engineering Environment for Knowledge-based Interactive eye motility Diagnostics, www.see-kid.at) [1]. The SEE++ software system implements a biomechanical model of the human eye, with the possibilities of simulating pathologies in the field of strabismus and performing a corrective surgery with a "virtual patient" so the results and effects of this surgery can be approximated. In SEE++, the path of each extraocular eye muscle is defined by several geometrical properties: the muscle's insertion on the surface of the globe (eye ball), the point of tangency, which is the last area where the muscle touches the globe, the pulley and the anatomical origin in the back of the orbit. The three-dimensional (3D) location data of insertions, pulleys and anatomical origins of all muscles are taken from averaged statistical data measured by Volkmann [2] and Miller [3]. Since the insertion is fixed to the globe and the anatomical origin is fixed to the bony orbit, eye muscles cannot move freely. Moreover, the pulley, an anatomical structure, which stabilizes the 1

Corresponding Author: Thomas Kaltofen, RISC Software GmbH, Research Unit Medical-Informatics, Softwarepark 35, 4232 Hagenberg, Austria; E-mail: [email protected].

G. Santana Sosa and T. Kaltofen / Collision and Containment Detection

561

muscle path in the area behind the globe, additionally restricts a muscle's freedom of movement. Apart from these properties, the path of a muscle also depends on the biomechanical model's influence on the behavior of the muscle in different gaze positions. Several approaches exist for modeling the muscle paths, especially regarding point of tangency and pulley, as described in [4]. SEE++ uses the approach of a static as well as an active pulley model based on the active pulley hypothesis [5]. For modeling the 3D volumes of the eye muscles, each muscle was reconstructed individually from coronal magnetic resonance (MR) images [6] for specific gaze positions. The resulting 3D surface models were projected onto the previously described muscle paths. For gaze positions, where no 3D models were reconstructed, the available models were interpolated. Due to that projection and interpolation, our problem is that 3D muscle volumes intersect, creating the need for a collision detection module. The main task of such a module is to detect collisions and containment of all eye muscles in all gaze positions in order to provide, in a subsequent step, a proper response to the intersections between the eye muscles. The response to the detected collisions in form of a deformation of the muscle volumes is not covered in this paper. For our problem, the definition of collision detection is more a definition of containment, since we have to find out which area of a muscle is contained by another one. Detection of containment with surface models is usually more difficult than just detecting collisions since algorithms that only perform surface intersection tests do not support checking for containment. In our case, the muscles are deformable bodies colliding already in their initial state. As indicated before, a muscle's shape, force distribution and path are solely defined by the biomechanical model which means that the surfaces of the muscle volumes change in every gaze position as determined by the biomechanical model, which prevents for example the usage of continuous collision detection. Another major requirement is that the detection of containment is computed almost in real time, because one of the future goals is building a response, which influences not only the muscle volumes but the biomechanical model itself. Three different solutions were implemented and tested within the SEE++ system. Details about each solution, including our current implementation, which uses a new approach based on existing techniques, as well as performance benchmarks are presented in this paper. The benchmarks were performed on an Intel Core i7 @ 2.66GHz with 6 GB DDR3 RAM and an ATI Radeon HD 4850 with 512 MB GDDR3 RAM.

Methods and Materials Many previous works address the need for a collision detection engine and propose a solution for rigid [7] [8] and for deformable (soft) bodies [9]. One characteristic of our problem is that there are no objects in motion, but the shape of the objects changes in every gaze position. Therefore, the situation is similar to a discrete collision detection (in contrast to a continuous collision detection [7]), with the difference that no information about the first contact time can be retrieved. The reason is the previously mentioned projection of the muscle volumes onto the muscle paths and the resulting colliding state even in primary position when both eyes are looking straight ahead. Given our need for a fast collision and containment detection, we started implementing a module using common image-space techniques based on GPU (Graphics Processing Unit) computation using OpenGL. The main idea behind this image-based approach is doing a rendering of the scene and analyze whether the vertices forming a muscle are

562

G. Santana Sosa and T. Kaltofen / Collision and Containment Detection

contained in another volume or not by checking their visibility. The result of this image-space approach is a list of all the vertices contained in another volume for all the volumes in the scene, in other words: a volume containment instead of a surface intersection. One drawback of this approach is that the whole scene is treated in pairs of objects while looking for possible containment, which means that all volumes in the scene have to be checked against all other volumes. Obviously, this slows down the collision detection process if the number of objects is increased. The different steps of the algorithm are as follows: 1.

For each object in the scene, an Oriented Bounding Box (OBB) structure [10] is built, which completely contains it. 2. The Separating Axis Theorem (SAT) test [10] is carried out between all boxes of all objects, except between boxes belonging to the same object (no self collision supported). 3. The OBBs that were found colliding in step 2 and all the vertices they contain are marked as possibly contained. 4. A pair of objects (in our case, a pair of muscles) is selected from the scene for the collision detection. 5. The surface of one of the muscles is rendered into the color frame buffer using the background color. This makes the muscle invisible in the color frame buffer, but not in the depth buffer (with depth testing enabled). 6. A color value is calculated for each of the vertices in the other muscle, which allows to uniquely identify each vertex in the color frame buffer. 7. All those vertices are rendered with different colors in the color frame buffer with depth testing enabled. 8. The information of the rendering is retrieved by reading the color frame buffer and getting all the pixel data as an array. 9. All the pixels in the array are analyzed. If an RGB (red, green, blue) color value different from the background color is found, the color represents the ID of the corresponding vertex. Since the vertex is visible, it is marked as not being contained. 10. The invisible vertices have to be further analyzed, because they can either be inside the other volume or behind it. 11. Since containment of only one muscle was detected so far (the muscle which vertices have been rendered with different colors), steps 5 to 10 have to be repeated with the other muscle's vertices being rendered with different colors. 12. Steps 5 to 11 are reiterated with the next pair of muscles. To solve the problem stated in step 10, an Axis Aligned Bounding Box (AABB) containing the two selected muscles was calculated for defining the OpenGL viewing frustum for rendering. The viewing frustum was then rotated several times and steps 5 to 11 were carried out again for each pair of muscles so more information about the scene could be retrieved. The timings for the collision and containment detection of six muscles in primary position comprising 44.500 triangles defined by 30.000 vertices with different resolutions of the color frame buffer can be seen in Table 1. However, with this approach there were still vertices wrongly detected as being contained, because even with very high resolutions of the color frame buffer vertices lying extremely close to each other were always drawn to the same location making one of them invisible.

G. Santana Sosa and T. Kaltofen / Collision and Containment Detection

563

Table 1. Timings for the collision and containment detection with the image-space approach (44.500 triangles, 30.000 vertices). Resolution (pixels) 100 x 100 200 x 200 250 x 250

Depth Buffer (bits) 24 24 24

Time (milliseconds) 766 1837 2682

Therefore, the vertices never visible during rendering were tested with the standard OpenGL picking test [11]. For ensuring if the vertices were really contained or not, the quads of the other muscle were determined that were closest to the vertex the picking test was performed with. The normal vectors of those quads were then used to determine whether the vertex was inside or outside the other volume. The main drawback of the described algorithm is that, for every picking test, parts of the other volume have to be rendered. The result is a very time consuming computation, even if done on the GPU, especially when the rendering steps do not detect enough visible vertices (a large amount of vertices remains for the picking test), which makes the approach not suitable for fast collision and containment detection. While working on our image-space approach, a version of the physics library Bullet [12] supporting collision and containment detection between deformable objects was released (version 2.75). Thus, the collision detection for deformable bodies was analyzed, because it supported concave objects (one of our requirements) due to the convex decomposition it does by using clustering and then applying the GilbertJohnson-Keerthi distance (GJK) algorithm [13]. In order to achieve the required precision, Bullet had to be configured to use one cluster per triangle [12] and since the muscle paths were defined by the biomechanical model as previously explained, those clusters had to be updated every time the paths changed. The time needed for collision and containment detection with Bullet (including the updating of the clusters) for six muscles in primary position comprising 44.500 triangles defined by 30.000 vertices is 134 milliseconds. Although Bullet performs way better than our image-space approach regarding speed, results were still not satisfying. Based on the ideas of our image-space approach and Bullet, we tried to combine the advantages of both to develop a new approach. One of the main disadvantages of the image-space approach was that the depth buffer in OpenGL only stored information about the object closest to the near clipping plane [11] and no information about the depth values of objects behind it. This also was the reason why the scene had to be rendered from different angles and why our algorithm could only be applied to a pair of muscles at a time. In order to overcome these limitations, a data structure similar to a combination of OpenGL's color frame buffer and depth buffer was defined, which allowed storing the depth information of all objects in the scene at the same time with just one rendering step. However, such a structure could not be used when rendering with OpenGL and therefore, we decided to switch from rendering on the GPU to software rendering [14]. Our data structure can be seen as a 3D array representing the world space (space-array), where the information stored in every cell is not a color or depth value (like in OpenGL) but a reference to a complex data structure (PositionInfo). This structure allows, on the one hand, to keep track of more than one object per array cell (position in space) and, on the other hand, to store more information about the primitive (quad, triangle) which the rendered vertex belongs to, such as the normal vector of the primitive. By directly storing the normal vectors of all rendered vertices in the cells of the array, it is now easy to determine which vertices are contained in other

564

G. Santana Sosa and T. Kaltofen / Collision and Containment Detection

volumes by simply carrying out a fast picking test similar to the OpenGL picking test (like explained before). In order to be able to properly detect containment by doing such a test, one requirement is that all analyzed volumes must have a closed surface. The workflow is the following: 1. 2. 3. 4.

5.

6.

7.

For each object in the scene, an OBB structure is built, which completely contains it. The SAT test is carried out between all boxes of all objects, except between boxes belonging to the same object (no self collision supported). With the OBB structure, an AABB covering all muscles is built and used for the projection of the real vertices of the muscles into the 3D array. The surface of the objects, defined by quads, is triangulated. Every triangle, which is contained in one of the OBBs detected to be colliding in step 2, is rendered by software rendering. The position of the rendered vertices in space correlates with their actual location in the 3D array. In every position of the 3D array in which a vertex was rendered, a reference to the PositionInfo data structure is stored. Moreover, the data structure is filled with information about the rendered vertex such as the normal vector of the primitive the vertex belongs to. While performing the rendering, it may happen that objects intersect and therefore, are rendered into the same position in space (in the 3D array). Whenever that happens, we have detected a collision between two primitives. In order to detect vertices being contained in another volume, the depth test is carried out for all relevant vertices along the Z-axis of the 3D array. The vertices suspected to be contained are tested the same way as explained in the image-space approach's picking test (standard OpenGL picking test) but without any additional rendering involved.

After carrying out all described steps, collisions and containment of all objects in the scene have been detected and no further processing is required. The timings for the collision and containment detection of six muscles in primary position comprising 44.500 triangles defined by 30.000 vertices with different resolutions of the 3D array can be seen in Table 2. Benchmarks in other eye positions with more contained vertices than in primary position (where 3,7 % of the vertices are detected as being contained) only show logarithmic growth of the calculation time.

Results We have implemented a new collision and containment detection algorithm by combining existing techniques, which can be seen as a mixture of voxeling, 3D rendering and spatial hashing [15]. The main advantage of the space-array approach presented in this paper is that after just one rendering into a special 3D array, all information needed for the detection of collisions and containment is available. This results in a very fast collision detection for convex and concave objects with no previous decomposition. Figure 1 shows the detected contained vertices after applying the space-array approach to a left eye with all six muscles. One parameter that has to be chosen with care is the resolution of the 3D array into which the rendering is done. This parameter has a direct influence on the balance between accuracy and performance.

G. Santana Sosa and T. Kaltofen / Collision and Containment Detection

565

Table 2. Timings for the collision and containment detection with the space-array approach (44.500 triangles, 30.000 vertices). Resolution (array cells) 100 x 100 200 x 200 250 x 250

Depth Buffer (array cells) 50 100 100

Time (milliseconds) 28 41 45

The algorithm of the space-array approach has proven to be very fast and stable and it fulfills all the requirements presented before. Figure 2 shows a direct comparison between the timings of the different approaches with the data taken from Table 1 and Table 2. In case of Bullet, the accuracy of the collision and containment detection is only comparable when having one triangle per cluster. Therefore, equal timing for Bullet is shown in case of all three resolutions in Figure 2. Although even the lowest resolution chosen for the performance benchmarks still provides full accuracy with our volumes, it is recommended to increase the resolution of the 3D array if the vertices of any volume in the scene are not equally distributed throughout the volume. Due to the way the data structure is built, the space-array approach currently has a limitation of 64 objects (muscles) per collision and containment detection and 1.666.425 vertices per object. These limitations can be overcome by reorganizing the way each vertex is identified in the 3D array.

Conclusion Our new approach for collision and containment detection presented in this paper uses a combination of existing technologies and is particularly suitable when exclusively dealing with deformable bodies. The algorithm fulfills all our previously described requirements and its main advantage, compared to other approaches, is that there is no need for creating a hierarchical structure for the objects in the scene. Consequently, there is no need for preprocessing the objects and no time consuming updates of the hierarchical structure are required.

Figure 1. Top view (A) and front view (B) of a left eye with the collision areas marked.

566

G. Santana Sosa and T. Kaltofen / Collision and Containment Detection

Figure 2. Time comparison between the three approaches in logarithmic scale.

The future work will concentrate on the implementation of a response to the detected collisions. The response will deform the surface of the muscle volumes depending on the amount and location of interpenetration, which can be easily determined since all contained vertices are known. Moreover, in a subsequent step the incorporation of the response into the biomechanical model itself is also planned, meaning that the muscle paths calculated by the model will be influenced by the collision detection and vice versa.

References [1]

[2] [3] [4] [5] [6]

[7] [8] [9]

[10] [11] [12] [13] [14] [15]

Buchberger M, Kaltofen T, Priglinger S, Hörantner R. Construction and application of an objectoriented computer model for simulating ocular positioning defects. Spektrum Augenheilkd, 17(4):151157, 2003. Volkmann AW. Zur Mechanik der Augenmuskeln. Berichte Sächsische Gesellschaft der Wissenschaften, Mathematisch-physikalische Klasse, 1869. Clark RA, Miller JM, Demer JL. Threedimensional location of human rectus pulleys by path inflections in secondary gaze positions. Invest Ophthalmol Vis Sci, 41(12):3787-97, 2000. Miller JM. Understanding and misunderstanding extraocular muscle pulleys. J Vis, 7(11):10-15, 2007. Kono R, Clark RA, Demer JL. Active pulleys: magnetic resonance imaging of rectus muscle paths in tertiary gazes. Invest Ophthalmol Vis Sci, 43(7):2179-88, 2002. Buchberger M, Kaltofen T. Ophthalmologic diagnostic tool using MR images for biomechanicallybased muscle volume deformation. In Proceedings of SPIE, Vol. 5032, eds. Sonka M, Fitzpatrick M, pages 60-71, 2003. Redon S, Kheddar A, Coquillart S. Fast Continuous Collision Detection between Rigid Bodies. Proc. of Eurographics (Computer Graphics Forum), 21(3):279-288, 2002. García-Alonso A, Serrano N, Flaquer J. Solving the Collision Detection Problem. IEEE Comput Graph Appl, 14(3):36-43, 1994. Teschner M, Kimmerle S, Heidelberger B, Zachmann G, Raghupathi L, Fuhrmann A, Cani MP, Faure F, Magnenat-Thalmann N, Strasser W, Volino P. Collision Detection for Deformable Objects. Eurographics State-of-the-Art Report (EG-STAR), Eurographics Association, pages 119-139. 2004. Gottschalk S, Lin MC, Manocha D. OBBTree: A Hierarchical Structure for Rapid Interference Detection. Comput Graphics, 30:171-180, 1996. Woo M, Neider J, Davis T, Shreiner D. OpenGL Programming Guide, Version 1.2. Addison-Wesley Longman Publishing, Boston, USA, 1999. Bullet Physics Library. http://www.bulletphysics.org. Gilbert EG, Johnson DW, Keerthi SS. A fast procedure for computing the distance between objects in three-dimensional space. IEEE J Robotic Autom, 4(2):193-203, 1988. Kaufman A, Shimony E. 3D scan-conversion algorithms for voxel-based graphics. Proceedings of the 1986 workshop on Interactive 3D graphics, pages 45-75, ACM, New York, USA, 1987. Turk G. Interactive Collision Detection for Molecular Graphics. Technical report, Chapel Hill, USA, 1990.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-567

567

Visualization of 3D Volumetric Lung Dynamics for Real-Time External Beam Lung Radiotherapy Anand P SANTHANAMa , Harini NEELAKKANTANb, Yugang MINc, Nicolene PAPPd, Akash BHARGAVAd, Kevin ERHARTe, Xiang LONGe, Rebecca MITCHELLf, Eduardo DIVOe, Alain KASSABe, Olusegun ILEGBUSIe, Bari H RUDDYf, Jannick P. ROLLANDg, Sanford L. MEEKSb and Patrick A. KUPELIANa a Department of Radiation Oncology, University of California, Los Angeles 200 UCLA Medical Plaza, Suite B265 Los Angeles, CA 90095 b Department of Radiation Oncology, M.D. Anderson Cancer Center Orlando 1400 S. Orange Ave Orlando FL 32806 c School of Computer Science, University of Central Florida d College of Optics/FPCE, University of Central Florida e Mechanical, Material and Aerospace Engineering, University of Central Florida f Health and Public Affairs, University of Central Florida 4000 Central Florida Blvd Orlando FL 32826 g Institute of Optics, University of Rochester 275 Hutchison Rd. Rochester, NY 14627-0186

Abstract. This paper reports on the usage of physics-based 3D volumetric lung dynamic models for visualizing and monitoring the radiation dose deposited on the lung of a human subject during lung radiotherapy. The dynamic model of each subject is computed from a 4D Computed Tomography (4DCT) imaging acquired before the treatment. The 3D lung deformation and the radiation dose deposited are computed using Graphics Processing Units (GPU). Additionally, using the dynamic lung model, the airflow inside the lungs during the treatment is also investigated. Results show the radiation dose deposited on the lung tumor as well as the surrounding tissues, the combination of which is patient-specific and varies from one treatment fraction to another. Keywords. Lung Radiotherapy, 3D Lung Dynamics, Computational Fluid Dynamics.

1. Introduction Medical simulation and visualization is a critical component in planning procedural interventions and predicting patient outcomes [1,2]. One of the key application domains is the visualization of lung radiotherapy for patients with Non-Small Cell Lung Cancer (NSCLC) [3]. Lung anatomy moves during breathing, which can reduce the amount of radiation dose deposited on the lung during radiotherapy. Such dose reduction lowers the overall treatment efficacy. The ability to predict the lung motion during radiotherapy

568

A.P. Santhanam et al. / Visualization of 3D Volumetric Lung Dynamics

coupled with the ability to calculate the radiation dose in real-time facilitates monitoring and visualizing the actual lung radiation delivery and investigating treatment optimizations that account for the lung motion. The success of such medical simulations is evidenced by the fact that over a third of all medical schools in the United States augment their teaching curricula using patient simulators [4]. Physics-based surface lung models have been previously shown to be effective for monitoring the radiation dose delivered on the tumor [3]. Such physics base models address the issue of predicting the lung tumor motion during the radiotherapy treatment. Results show variations in the dose delivered to the tumor when the tumor motion is taken into account. In this paper, we describe a research effort that focuses on developing methods for creating patient-specific physics-based volumetric lung models. Such volumetric lung models will account for the radiation delivered not only to the tumor but also to the surrounding lung tissues. We hypothesize that GPU-based deformation methods and dose computations are necessary to adequately visualize, monitor, model, and characterize the treatment efficacy of the lung radiotherapy for a given NSCLC patient. The reported results summarize the volumetric model development and the treatment simulation and its real-time nature. Variations observed for a patient from one treatment fraction to another and also from the actual planned treatment are discussed.

2. Proposed Method for 3D Lung Deformation In this section we discuss the methodology adopted for the dynamic simulation of 3D volumetric lungs deformation. 2.1 Patient Data Acquisition The patient imaging data used for the proposed framework includes the 4D Computed Tomography (4DCT) imaging during the diagnostic stage. Additionally, the flowvolume breathing signal is also collected during the imaging stage. The patient’s lung compliance is measured using Impulse Oscillometry (IOS) measurements. IOS is a relatively new noninvasive method for measurement of respiratory impedance, (i.e. airway resistance and reactance) at different oscillation frequencies. Oscillometry uses external forcing signals, which can be mono- or multi-frequency, and applied either continuously or in a time-discrete manner [5]. For the current study, the forced oscillations are performed using a MasterScreenIOS device [6]. A volume-controlled loudspeaker generates a multi-frequency impulse

(a)

(b)

(c)

(d)

Figure 1. The YM estimated for each lung voxel at 100 % ((a) and (b)) and 30% ((c) and (d)) inhalation is shown, for left and right lungs 4DCT dataset.

A.P. Santhanam et al. / Visualization of 3D Volumetric Lung Dynamics

569

signal lasting 45 milliseconds. Each impulse produces a volume shift consisting of approximately 40 ml either in the inspiratory or the expiratory direction. The power spectrum of the impulse covers a range from above 0 to 100 Hz. The loudspeaker unit is coupled with a tube of 35 cm length and 3 cm diameter to the measuring head based on a Y-connector. The Y-connector is terminated with a mesh screen resistor of 0.1 kPa/(l/s). On the front side, a Lilly-type Pneumotachograph [6] provides differential pressure to sense flow [5]. The patient is instructed to sit upright during tidal volume breathing with a solid lip seal around the mouthpiece flange. Cheeks are held with light pressure using the patient’s hands during the breathing task. The measured impedance is represented for the range of frequencies, which represents the internal airway levels. 2.2 Estimating the YM Value of Each Lung Voxel The YM value of each voxel is estimated for known values of airflow and the volumetric lung displacement estimated from the 4DCT lung dataset using a modified optical flow approach [7]. A Hyper-spherical Harmonic (HSH) transformation is employed to compute the deformation operator. The HSH coordinated transformation method facilitates accounting for the heterogeneity of the deformation operator using a finite number of frequency coefficients. Spirometry measurements are used to provide values for the airflow inside the lung. Using a 3D optical flow-based method, the 3D volumetric displacement of the left and right lungs, which represents the local anatomy and the deformation of a human subject, was estimated from the 4DCT dataset. Results from an implementation of the method show the estimation of the deformation operator for the left and the right lungs of a human subject with NSCLC [8]. Figure 1 represents the color-coded representation of the YM values for the left and right lungs at 30% and 100% tidal inhalation. The ranges of values are represented by 0-400 Pa (black), 400800 Pa (red), 800-1200 Pa (yellow), 1200-1600 Pa (green) and 1600-2000 Pa (white). It can be seen that differences in the elasticity can be observed in different regions of the lung as well as from one sub-anatomy to another. 2.3 Tracheobronchial Fluid Dynamics Simulation The lungs are modeled as poro-elastic media, where the flow field satisfies Darcy’s law and the elasticity field is solved using the non-homogeneous Navier’s equation. The tissue properties, i.e., the porosity, permeability, and shear modulus, are considered non-homogenous as they vary throughout the lung as illustrated by the elasticity. In simulating the fluid flow inside the lungs, two different approaches are considered. The two approaches cross-verify each other’s simulation. In the first approach, a meshless modeling technique is investigated in order to reduce the dependence of the solution accuracy on the discretization of the lung model itself. For stability of the solution, radial basis functions are coupled with moving least squares [9]. For the second approach, the 3D lung anatomy was discretized into finite elements for computation using commercial software ADINA. The FEM model considered for the airflow within the lung to occur through a poro-elastic medium and the structural dynamics were resolved using a flow-structure interaction model. The tissue properties in this case are estimated from the patient’s 4DCT as discussed in section 2.2. The spatial deformation is predicted over a complete breathing cycle. The volumetric lung deformation obtained using this simulation is then employed for refining the YM value estimated for each lung voxel.

570

A.P. Santhanam et al. / Visualization of 3D Volumetric Lung Dynamics

2.4 3D Volumetric Lung Dynamics Simulation A GPU is used for computing the 3D anatomy deformation using the 3D lung anatomy taken during the diagnostic stage and the YM parameter associated with each voxel (estimated in section 2.2) coupled with the spirometry signal that gives the air volume of the lung. A 3D convolution is performed between the YM elasticity distribution inside the lung and the applied force computed using the spirometry air volume and the airway compliance estimated from the IOS measurements. The result of this convolution provides the displacement for each voxel during the breathing. The 3Dlung anatomy is deformed as follows: The deformed 3D lung volume is first initialized in such a way that voxel positions which represent the lung in the un-deformed lung volume are set to the Hounsfield number representing the air. This initialization is done in order to make sure that the voxel, which will not have the deformed lung anatomy, will be filled with air. For each voxel in the un-deformed 3D lung anatomy, the position of the voxel during the deformation is first computed. The Hounsfield number is then copied into that location [10]. Results show that using GTX 480, the volumetric lung dynamics is achieved at the rate of once every 60 milliseconds. 2.5 3D GPU Based Dose Calculation The 3D dose convolution is performed using a 3D separable dose convolution approach. In this method, a 3D dose convolution is split into row-wise, column-wise and planewise 1D convolutions. A 10 cm3 lung data has been taken from a CT dataset with 128 slices and a 3D data with 1283 voxels is created. The 1D convolutions are computed for each voxel using its 127 neighbors along the row, column, and plane respectively. For real-time purposes, we employed a shared-memory based data access for performing the convolutions. Specifically, the 3DCT data representing the patient anatomy, the displacement vector associated with each voxel, and the voxels representing the 3D dose accumulated on each image voxel are copied into the shared memory of the GPU. The row and column 1D convolutions for each voxel initiate bulk data transfer between the GPU processor and the shared memory thereby increasing the memory bandwidth usage. However, the hardware architecture of the GPU does not directly allow initiating bulk data transfer from the processor. Thus in the proposed method, we employ an optimization where a 3D matrix transpose is performed after the row and column 1D convolution. Such a matrix transpose re-arranges the voxels from neighboring planes to be placed next each other thereby facilitating bulk data transfer. An initial flux of 6 MV photons is assumed and a 3D dose is computed for each lung voxel. To calculate first scattering dose components, the 3D dose is convolved with a kernel of size 33. The kernel is computed using the image voxels surrounding each point to be convolved [3,10,11]. Results show that the dose calculation is achieved at the fastest rate of 130 milliseconds using the Nvidia GTX 480 graphics card. 3D GPU based volume visualization is employed for visualizing the 3D lung volume together with the 3D lung radiation dose accumulated on it. The method is

A.P. Santhanam et al. / Visualization of 3D Volumetric Lung Dynamics

571

implemented as follows. The intensity of each voxel is used to determine the opaqueness of each voxel. The 3D lung anatomy is represented as a grey scale image, while the 3D dose is represented in a set of colors (blue 0-85% dose, green 85-90% dose, red 90-95% dose and white 95%-100% dose).

2.6 OpenCV Based Spirometry Tracking Clinical spirometry systems that track the patient’s flow volume perform tracking at 20-30 Hz. For our experiment, we use PowerLab 6.0 for measuring the spirometry in real-time. However to incorporate the flow volume signal into the proposed framework, we developed an OpenCV- based spirometry-tracking system. An external HD camera is mounted and calibrated to track the spirometry display screen. The tracking system tracked the flow volume signal displayed on the spirometry equipment and communicates with the proposed framework using inter-process communication techniques. The advantage of using such a spirometry tracking is that the radiotherapymonitoring framework is ensured to work synchronously with the spirometry signal. Based on the computational capability of the simulation system, the flow volume signal is obtained from the spirometry tracking system and the volumetric lung deformation is performed. Additionally, the spirometry tracking enables real-time data acquisition by keeping the spirometry interface independent of the deformation model framework. The interface works as follows: The spirometry display system displays the flow signal as a graph with the air volume along the vertical axis and the time along the horizontal axis. We ensure that the signal displayed by the spirometry has a unique color in the screen. The user sets the breathing time lag, which is the time lag after which the patient’s breathing is reflected in the visualization. This is done by appropriately setting the monitoring vertical axis in the HD camera image. Once the patient starts breathing, the HD camera image is acquired in real-time and the unique color representing the flow signal is tracked along the vertical axis representing the breathing time lag. 2.7 The Integrated Framework We now describe the integrated framework for visualizing the radiation therapy delivery. For a given patient, a 4DCT image of the lung is acquired during the diagnostic stage as discussed in section 2.1. The 3D deformable model is developed by estimating the YM of each lung voxel as discussed in section 2.2. The YM estimation is then refined using the simulation of the tracheobronchial airflow modeling and the overall lung deformation. Finally, during the treatment fraction, the patient breathing is acquired in real-time using the spirometry tracking system as discussed in section 2.6. The 3D volumetric lung is deformed as discussed in section 2.4 and the dose deposited in the volumetric lung is visualized as discussed in section 2.5.

572

A.P. Santhanam et al. / Visualization of 3D Volumetric Lung Dynamics

Figure 2. The real-time radiation dose delivered for a patient lung with (a) no motion, (b) sinusoidal tumor motion, (c) breathing curve obtained on day 1, (d) day 2, (e) day 3, and (f) day 4.

(a) (b) (c) Figure 3. The DVH of the patient for the different treatment fractions with sinusoidal breathing volume considered. The difference among the DVH is significant.

(a) (b) (c) Figure 4. The DVH of the patient for the different treatment fractions with patient-specific breathing volume considered. The difference among the DVH is significant.

3. Results We now present the patient results obtained using the proposed framework. Figure 2 represents the dose calculated on the 3D deforming volumetric lung for a NSCLC patients. It can be seen that at each fraction the dose deposited on the tumor and the surrounding tissues vary from one another. Figure 2a represents the 3D dose deposited on the volumetric lung without taking into account the lung motion. When the volumetric lung motion is taken into account using sinusoidal breathing (Figure 2b), it can be seen that the radiation dose deposited on the lung varies from Figure 3a, which

A.P. Santhanam et al. / Visualization of 3D Volumetric Lung Dynamics

573

shows the changes in the treatment efficacy from the actual treatment plan. When the actual patient breathing is considered, Figure 2c-f shows the variations in the radiation dose delivered during every treatment fraction. The difference in the dose accumulation between the sinusoidal breathing and subject-specific breathing can be observed. Such variations can be correlated with the treatment outcomes to further optimize the treatment plan. Figure 3-4 shows the Dose Volume Histogram (DVH) for three different fractions and using two different breathing variations of the same patient. When the lung motion is modeled using a sinusoidal breathing volume (Figure 3), the change in the dose delivered to the tumor changes from one day to another. Such changes in the DVH are attributed to the changes in the breathing lung model developed from each of the 4DCT. When the lung motion is modeled using the patientspecific breathing collected using spirometry (Figure 4), the DVHs significantly varied both from one day to another and from using sinusoidal motion analysis. Such variations further quantify the need for accurately accounting for the patient lung volume changes and the need for real-time adaptive lung radiotherapy, where the treatment is monitored and modified in real-time. To conclude, the usage of physicsbased lung deformation and real-time dose calculation can be used to monitor the subject specific dose delivery and make critical changes to the treatment plan.

4. Acknowledgement This work is funded by the James & Esther King Foundation.

5. References Nye, L.S., The minds' eye, Biochemistry and Molecular Biology Education. 32 (2) 123-131, (2004). Robb, R.A., Three-dimensional visualization in medicine and Biology, Handbook of medical Imaging: Processing and Analysis, I.N. Bankman, Editor Academic Press: San Diego,CA, 2000. [3] Santhanam, A.P., T. Willoughby, I. Kaya, A. Shah, S.L. Meeks, J.P.Rolland, and P. Kupelian, A Display Framework for Visualizing Real-time 3D Lung Tumor Radiotherapy, IEEE Journal of Display Technology “Special issue on Medical Displays” 4 (4) 473-482, (2008). [4] Good, M.I., Patient simulation for training basic and advanced clinical skills, Medical Education. 37 14, (2003). [5] Gerhard, K., Thorsten, B, Ute, R.,Alwin, G. Hans-Juergen, S. and Klaus, P, Measurement of respiratory impedance by Impulse Oscillometry – effects of endotracheal tubes, Research in Experimental Medicine, 200 17-26, (2000). [6] H.J. Smith, H.J., Reinhold, P. and Goldman, M.D., Forced oscillation technique and impulse oscillometry, European Respiratory Monographs, 31 72–105, (2005). [7] Min, Y., N. Papp, A. Shah, S. Meeks, and A.P. Santhanam. 2010. 4D-CT lung registration using anatomy based multi-level multi-resolution optical flow analysis and thin plate splines. Physics in Medicine and Biology (in review). [8] Santhanam, A., Y. Min, S. Mudur, E. Divo, A. Kassab, B. H. Ruddy, J. Rolland and P. Kupelian, An inverse hyper-spherical harmonics-based formulation for reconstructing 3D volumetric lung deformations. Comptes Rendus Mechanique 338 (7-8) 461-473, (2010). [9] V. Huayamave, A. Vidal, A. Kassab, E. Divo, A. Santhanam, and P. Kupelian, A meshless approach to solve the fluid poro-elastic interaction problem between the tracheo-bronchial tree and the lungs, Int. Conf. on Computational Methods for Coupled Problems in Science and Engineering, Ischia Island, Italy, June 8-10, 2009. [10] Min, Y., A. Santhanam, A., Y. Min, H. Neelakkantan, B.H. Ruddy, S. Meeks, P. Kupelian, A GPU based framework for modeling real-time 3D lung tumor conformal dosimetry with subject-specific lung tumor motion, Physics in Medicine and Biology 55 5137, (2010). [11] Santhanam, A.P., T. Willoughby, S.L.Meeks, and P. Kupelian, Modeling simulation and visualization of 3D lung conformal dosimetry, Physics in Medicine and Biology 54 6165-6180, (2009). [1] [2]

574

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-574

Laser Surgery Simulation Platform: Toward Full-Procedure Training and Rehearsal for Benign Prostatic Hyperplasia (BPH) Therapy Yunhe SHEN a,b,1, Vamsi KONCHADA a,c, Nan ZHANG a,b, Saurabh JAIN a,c, Xiangmin ZHOU a,d, Daniel BURKE a,b, Carson WONG e, Culley CARSON f, Claus ROEHRBORN g and Robert SWEET a,b a Center for Research in Education and Simulation Technologies, Univ. of Minnesota b Department of Urologic Surgery, University of Minnesota c Department of Computer Science and Engineering, University of Minnesota d Department of Mechanical Engineering, University of Minnesota e Department of Urology, University of Oklahoma Health Sciences Center f Department of Surgery, University of North Carolina g Department of Urology, University of Texas Southwestern Medical Center at Dallas

Abstract. Recently, photo-selective vaporization of the prostate (PVP) has been a popular alternative to the standard electrocautery - transurethral resection of prostate (TURP). Here we introduce a new training system for practicing the laser therapy by using a virtual reality (VR) simulator. To interactively and realistically simulate PVP on a virtual organ with an order of a quarter million elements, a few novel and practical solutions have been applied to handle the challenges in modeling tissue ablation, contact/collision and deformation; endoscopic instruments tracking, haptic rendering and a web/database curriculum management module are integrated into the system. Over 40 urologists and surgical experts have been invited nationally and participated in the system verification. Keywords. BPH, PVP, surgery simulation, virtual reality.

Introduction Benign prostatic hyperplasia [1] or prostate enlargement commonly occurs to men as an aging problem, with a direct symptom of lower urinary tract blockage or occlusion caused by the enlargement, as shown in Figure 1(a). Since 1990’s, a series of minimally invasive therapies and transurethral endoscopic surgeries have been routinely applied to BPH patients in clinics. These treatments generally deliver external energies in terms of electrical current (in TURP), radiofrequency or microwave, thermal current or laser beam (in PVP) into prostate for tissue resection or ablation.

1

Corresponding Author: Yunhe Shen, University of Minnesota, 420 Delaware St SE, Mayo Building A584, Minneapolis, MN 55455; E-mail: [email protected].

575

Y. Shen et al. / Laser Surgery Simulation Platform

Photo-selective vaporization of the prostate is an endoscopic laser surgery. In PVP, laser energy is delivered to a patient’s prostate gland through a fiberoptic glass/probe inside a cystoscope, as shown in Figure 1(b) and 1(c). After inserting the scope through urethra into the prostate or bladder filled with irrigation fluid, a urologist can translate and rotate the scope and the probe to precisely aim the laser beam at certain local treatment area, where a vaporization mechanism can be developed by sweeping the high energy laser beam along the local tissue surface while maintaining appropriate distance between the probe and the tissue. By repeating this operation pattern inside the prostate, most part of the prostate tissue is gradually vaporized and the urinary tract is recovered. Digital video is captured inside the prostate by a cystoscopic camera and displayed in the operating room as visual clue to the surgeon during this operation, as shown in Figure 2(a). The laser used in PVP usually seals bleeding vessels as it ablates the prostate; occasionally as few bleeders do occur, the laser power can be lowered in order to coagulate the bleeders.

1.8mm

(a)

(b)

(c)

Figure 1. (a) BPH [15]. (b) PVP therapy. (c) Cystoscope tip, endoscopic lenses, and laser fiber [8].

The PVP example we choose to simulate is a GreenLight™ laser surgery system [2], which uses a side-firing fiber/probe emitting a 532 nm wavelength lithium triborate (LBO) laser with maximum power of 120W in HPS version and 180W in XPS version. A few technologies and algorithms in the related fields are worth mentioning for the research and development of the real time PVP simulator. Visualization methods such as isosurface extraction by marching tetrahedrons [3] or marching cubes [4], the level set method [5-6], and various physically based modeling approaches are reviewed for deriving the most suitable solutions for this medial VR application. Experimental data of laser-tissue properties [7] provide quantitative estimation in calibrating tissue vaporization rates. The objective of this work is to design and build a VR simulator and provide the most consistent, standardized, modular training curriculum to develop and improve users' skills in performing PVP Therapy, and to achieve rapid clinical proficiency leading to excellent clinical outcomes.

1. Requirements Analysis and Training Modules Design 1.1. Interdisciplinary Research We have formed up an interdisciplinary team nationally including a clinical advisory panel consisting of four key opinion leaders for BPH who collaborate with our engineering group throughout the research and development process from requirements

576

Y. Shen et al. / Laser Surgery Simulation Platform

analysis to training curricula development. More than 40 board-certified urologists participated in the verification testing of the training platform at the American Urological Association conference, San Francisco, 2010. The GreenLight™ simulator clinical advisory panel defines desired outcomes, learning objectives and desired metrics which led to the development of desired exercises. These exercises are divided into a full-procedure laser operation on 6 variations of virtual BPH models, as well as several subtasks specifically designed by task deconstruction for part-task training. 1.2. System Framework Design This simulation framework consists of the following major components: • Anatomical models and variations • Volumetric ablation and visualization • Collision detection and contact modeling • Tissue and fiber deformation • Graphical rendering and special effects - bleeding, vaporization, etc. • Motion tracking and haptic rendering • Web/database module for integrated learning management • Subtasks, quizzes and tutoring The main framework and core VR modules are implemented with C++ and objectoriented programming (OOP). Multithreading is used to allow several computationintensive modules run in parallel, whereas hardware acceleration or extensive threading on GPU/CPU cores can be a potential reinforcement for the computation needs in future versions. 1.3. Anatomical Models and Variations We model the anatomy of prostate, urethra and bladder with surface or volumetric meshes generated from medical imaging data, and refine these mesh models in graphical design software, which is also used to model surface textures, materials as well as surgical instruments involved in PVP. Our virtual BPH prostate models cover a mass/volume range from 30 to 95 grams or cm3. The 6 key common clinical variation prostate cases defined by the clinical advisory panel and represented in the curriculum consist of (1) small normal, (2) large median lobe, (3) high median bar, (4) small fibrous, (5) tri-lobar enlargement, and (6) prominent apex. Figure 1(c) shows a close view of the cystoscope tip structure, from which we realize that although a prostate is not considered a large organ, its mesh model does require us to spend a large amount of elements to ensure its resolution and quality, because the scale of this laser-tissue ablation is precisely controlled in an order of a few millimeters at a given time during PVP treatment, and then cystoscopy magnifies it to a full screen size; tissue ablation causes geometrical or topological modifications applicable to the entire prostate volume in the treatment, which requires the modeling to main the level-of-details accordingly.

Y. Shen et al. / Laser Surgery Simulation Platform

577

2. Ablation and Visualization Isosurfaces of the ablation are derived from the initial tetrahedral mesh of the virtual prostate model by implicit surface construction. A volumetric ablation method using standard constructive solid geometry (CSG) algorithm is improved with our volumecontrolled CSG approach [10] to maintain the tissue ablation or vaporization rates within the reasonable range estimated by the study in [12] according to the in vitro bovine experimental data [7]. In addition, we have advanced the CSG concept further to a new ablation algorithm, which generates realistic visual effects such as tissue melting or shrinking in PVP treatment, and is applicable to relatively lower resolution models. By using this method, we are able to reduce the tetrahedral elements of a prostate model from more than a million [10] to a quarter million, thus the computation load in deformation and collision handling are substantially relieved.

3. Collision Detection and Contact Modeling A fast and reliable solution for collision or contact handling is critical to the success of PVP simulation, because a cysctosope is tightly and constantly surrounded by largely deformed or stretched prostate tissue; in most time during the treatment, percentage of contact area is also high – both the scope and fiber are deeply operated posterior to the verumontanum of the prostate to avoid damages to the urinary sphincter or the apex zone. “Popping through” or penetration problem was a challenge to an earlier prototype, in a couple of cases when users tried to create a narrow channel by ablation and push the rigid scope against the soft tissue all the way to the prostate boundary. Several analytical bounding shapes are applied to the cystoscope and the fiber probe models, to correct or constrain the movements of those elements that belong to a deformable prostate model but are detected inside or on the surface of a rigid PVP instrument model in the virtual surgery environment. This fast processing is able to address the contact or collision problems at interactive rates. This module also calculates a few PVP metrics such as sweep speed and treatment distance, which is defined as the estimated distance from an emitting point of a laser fiber to the intersection spot where a laser beam meets the surface of a prostate. It detects weather certain predefined anatomical structures such as external urinary sphincter, verumontanum and prostate capsule have been damaged in the treatment, in that case, quantitative results can be calculated and logged in the learning management database.

4. Tissue and Fiber Deformation Deforming the prostate model is another challenging problem to solve for this application. In PVP, a laser beam is arbitrarily manipulated within the volume of a prostate and constantly modifies tissue geometry and topology. Simulating this procedure requires that the deformable model is applicable to and synchronized with the updated prostate ablation model per simulation frame. Concerns in computation size and algorithm stability require a swift and robust deformation model. Starting from our earlier volumetric approach [11], we simplify this module to such a degree that its

578

Y. Shen et al. / Laser Surgery Simulation Platform

computation complexity is reduce to minimum yet it handles complicated shapes and presents realistic visual response to users input at local treatment area. The fiber probe is thin and flexible; it may be deflected by or penetrates into soft tissue as being pushed against the inner surface of a prostate. It’s important to enable this fiber motion in PVP simulation, or it would feel quite different from the real operation. We simplify this phenomenon to rigid transformation of the fiber without spending processing power in solving fiber glass deformation.

5. Graphical Rendering and Special Effects Particle systems and texture animations are applied to render bubbles, laser beam, and bleeding in an environment filled with irrigation fluid. Ogre [9] is integrated in the current system for its graphics/GPU rendering features.

6. Motion Tracking and Haptic Rendering Rigid body of cystoscope has 6 Degrees of freedom (DoF) motion; scope camera has 1 DoF independent rotation; fiber has 2 independent DoF (translation in and along the cystoscope plus axial rotation). Real GreenLight™ foot pedals have been connected to the system for 3 control signals - vaporization, coagulation and standby. Some peripheral devices such as motion tracking sensors and hardware assemblies are designed and manufactured by working with two industrial partners. A low-cost Novint Falcon™ [13] force rendering device provides haptic feedback through the cystoscope model, which is a 3-D scan and remodeling of an actual cystoscope. In haptic rendering, resultant Ft is directly calculated from the original deformation or collision force Fd; here we add a center-line enhancement Fc to Ft: (1) Where wd and wc are two scalar values determined by an attenuation function. Our study shows that haptic feedback is important in this VR application. Without being assisted by the haptic guidance existing in real PVP practices, users could lever the cystoscope out of the narrow space or the deformation limits inside the prostate model.

7. Web and Database for Integrated Learning Management A web-based interface and a SQL database connection module have been proposed in our early design phase and now integrated with the real time simulation system. This learning management platform stores and maintains training curriculum including performance metrics, scores and medical knowledge base together with system configurations in the SQL database server. As a powerful multi-institutional validation study tool, this management platform organizes all participants and curricula data at global, local site and local group levels. A web-based interface provides remote users

579

Y. Shen et al. / Laser Surgery Simulation Platform

online access to the database server. This unique PVP training platform design can be further extended in the next version to include uploading function allowing certain patient-specific features be built into customized training sessions.

8. Subtasks, Quizzes and Tutoring Similar to the previous task deconstruction approach [14], we divide the skill set in the PVP procedure into several subtasks training laser sweep speed, sweep distance, coagulation and anatomy identification. Other BPH knowledge base or quizzes can also be added in the simulation platform as individual training modules. In a tutoring mode, users will have instant guidance or error warning feedback during self-learning of the PVP procedure on the simulator. In other exercise or skill assessment mode, performance metrics including surgical errors are also logged into the learning management database by the virtual tutor.

9. Results In Figure 2(a), real surgical video captures are shown in the upper row as comparison to the simulation results shown in the lower row. The left column shows the narrow urinary tract being dilated by a cystoscope tip which encapsulates the lenses but is not visible in the view; the middle column shows the scene of laser ablation and part of the fiber/probe extending along and ahead of the cystoscope case; the right column shows the reopening of the urinary track and the fiber optics with laser power off. VR simulation modules are successfully implemented for the simulated PVP procedure. Figure 2(b) shows the integrated system including the VR interface. All processing threads have achieved a refreshing rate above 30 frames per seconds on a mid-range Intel® Core™ i7-860 workstation with Nvidia GTS240 graphic card, as testing the PVP procedure over a set of prostate models with resolutions vary from 220k to 480k tetrahedral elements. This is mainly achieved by algorithm optimization and simplification.

(a)

(b)

Figure 2. (a) PVP surgical video captures and simulation results. (b) The PVP simulator.

580

Y. Shen et al. / Laser Surgery Simulation Platform

10. Verification and Validation In 2010 American Urology Association meeting at San Francisco, CA, over 40 urologists tested our prototype PVP BPH simulator. In the recent version we have demonstrated several newly-updated methods in the training system including the new ablation model, the haptic/force-feedback feature and full-motion-tracking interface for the endoscope operation, as well as the integrated learning platform to collect and manage performance data. Two prototype simulators have been running smoothly and sustained 5 day exercise without raising noticeable reliability issues. Encouraged by the valuable feedback from the tests, we are refining the system and developing the next version training platform as well as planning a formal multi-institutional validation study for this VR-based curriculum. Details of the new updates in the functional modules and the results of the validation study will appear in our future report.

Acknowledgement This research and development was supported in part by American Medical Systems®. Visualization facilities supported by Minnesota Supercomputer Institute (MSI) are acknowledged.

References [1] American Urological Association. Guideline on the management of benign prostatic hyperplasia, Chapter 1: Diagnosis and treatment recommendations. J. of Urology, 170(2), pages 530–537, 2003. [2] American Medical Systems. GreenLight system: http://www.greenlighthps.com/lasersystems.html [3] A. Guéziec and R. Hummel. Exploiting triangulated surface extraction using tetrahedral decomposition. IEEE Trans. Visualization and Computer Graphics, 1(4), pages 328-342, 1995. [4] W. Lorensen and H. Cline. Marching cubes: A high resolution 3D surface construction algorithm. Computer Graphics, Vol. 21(4), pages 163-169, July 1987 [5] S.Osher and J. Sethian. Fronts propagating with curvature-dependent speed: Algorithms based on Hamilton-Jacobi formulations, J. Comput. Phys. 79, pages 12–49, 1988. [6] J. Sethian. Level set methods and fast marching methods: Evolving interfaces in computational geometry, fluid mechanics, computer vision, and materials science. Cambridge University Press. 1999. [7] H. Kang, D. Jebens, R. Malek, et. al. Laser vaporization of bovine prostate: A quantitative comparison of potassium-titanyl-phosphate and lithium triborate lasers. J. Urology, 180, pages 2675-2680, 2008. [8] American Medical Systems. GreenLight Standardized Training, CD media, 2009. [9] Ogre rendering engine: http://www.ogre3d.org/ [10] N. Zhang, X. Zhou, Y. Shen and R. Sweet. Volumetric modeling in laser BPH therapy simulation. IEEE Visualization, special issue of IEEE Trans. on Visualization and Computer Graphics, to appear. 2010. [11] Y. Shen, X. Zhou, N. Zhang, K. Tamma, and R. Sweet. Realistic Soft Tissue Deformation Strategies for Real Time Surgery Simulation. Stud. Health Tech. Inform, 132, pages 457-459. 2008. [12] X. Zhou, N. Zhang Y. Shen, et. al. Phenomenologial model of laser-tissue interaction with application to benign prostatic hyperplasia (BPH) simulation. Submitted to MMVR’18. [13] Novint Falcon: http://home.novint.com/products/novint_falcon.php [14] K. Adiyat, R. Beddingfield, T. Holden, Y. Shen, T. Reihsen and R. Sweet, "Task deconstruction facilitates acquisition of TURP skills on a virtual reality trainer," J. Endourology, 23(4), pages 665-668, 2009. [15] National Cancer Institute, AV: CDR462221, 2004. http://visuals.nci.nih.gov/details.cfm?imageid=7137

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-581

581

3D Tracking of Surgical Instruments Using a Single Camera for Laparoscopic Surgery Simulation Sangkyun SHIN1, Youngjun KIM, Hyunsoo KWAK, Deukhee LEE and Sehyung PARK Intelligence and Interaction Center, Korea Institute of Science and Technology, Korea

Abstract. Most laparoscopic surgery simulation systems are expensive and complex. To overcome these problems, this study presents a novel threedimensional tracking method for laparoscopic surgical instruments that uses only a single camera and fiducial markers. The proposed method does not require any mechanical parts to measure the three-dimensional positions/orientations of surgical instruments and the opening angle of graspers. We implemented simple and cost-effective hardware using the proposed method and successfully combined it with virtual simulation software for laparoscopic surgery. Keywords. Medical simulation, laparoscopic surgery, computer vision, virtual reality.

Introduction The practice of medical simulation was developed to guarantee patient safety. Medical students or novice surgeons can be efficiently trained using these simulation systems. Compared with patient or animal training environments, medical simulation provides a typical and uniform training condition. One area of potential application is laparoscopic surgery. This type of surgery involves difficulties due to the narrow scope of vision, a lack of perspective, weak contact feelings, and the pivoting movement of the surgical instruments, all of which are in contrast to traditional invasive surgery. Hence, laparoscopic surgery requires systematic and continuous learning of the type provided by virtual surgery simulation. Laparoscopic surgery simulation has been improved by many investigators in many countries using advanced medical technology. However, most systems continue to be associated with problems that arise related to cost-effectiveness. Hence, most medical students scarcely have the opportunity to access training in this type of surgery. To tackle the problem, a 3D tracking technique of surgical instrumentation was recently introduced by means of applied computer vision technology [1-3]. Doignon [1, 2] proposed an algorithm that estimates the 3D pose of the surgical instrument via collinear markers on the instrument. This computer vision-based tracking system has a

1

Corresponding Author: Intelligence and Interaction Center, Korea Institute of Science and Technology(KIST); E-mail: [email protected]

582

S. Shin et al. / 3D Tracking of Surgical Instruments Using a Single Camera

simple and cost-effective structure, as it does not rely on mechanical parts to measure the positions and orientation of the surgical instruments. In the present paper, this earlier method was applied to estimate the 3D positions and orientations of laparoscopic surgical instruments. Furthermore, we expanded the previous method by proposing a novel method of measuring the angle of laparoscopic surgical graspers using a “single” camera. The proposed method is simple and costeffective. We successfully combined the proposed computer vision-based tracking module with a training simulation system for laparoscopic surgery.

1. Method To estimate the pose of the instrument and measure the opening angle of the graspers, a camera unit and four band-type fiducial markers were used.

Figure 1. Photos of the grasper tip (P0, P1, P2: static marker, P3: dynamic marker, upper: opened, lower: closed)

As shown in Figure 1, three markers were static markers; these were used to estimate the instrument’s pose. The other marker was a dynamic marker which measured the opening angle of the grasper. While a user manipulates the graspers, the dynamic marker moves according to the opening angle of the tip. The procedures for the 3D tracking of the laparoscopic surgical instrument are given below. 1. Sequential grey images are obtained from the camera. 2. The image coordinates of the markers are calculated via image processing by OpenCV [4]. 3. The 3D pose of the surgical instrument is estimated from three static markers using Haralick’s algorithm [3]. 4. The grasper’s opening angle is computed from the dynamic marker. 1.1. Hardware Setup To represent the laparoscopic surgery environment and create the simulator on a computer screen, a test case was produced. To simulate skin tissue, we attached rubber onto the top surface of the case. We also considered the movement of the surgical instruments and field of view of the

S. Shin et al. / 3D Tracking of Surgical Instruments Using a Single Camera

583

camera to determine the magnitude of the case. To maintain an even illumination level, we attached a light emitting diode (LED).

Figure 2. Hardware set-up of the computer vision-based laparoscopic surgical simulator

As shown in Figure 2, we used the surgical instrument and a trocar as the training equipment in practice. The surgical instrument has labeled markers which contain four white signs so as to be easily recognized on the black background. The marker located on the tip of the instrument was built to move vertically according to the angle of the gripers. 1.2. 3D Pose Estimation of Laparoscopic Surgical Instrument The image processing procedures for the 3D pose estimation of the laparoscopic surgical instrument are shown in Figure 3. First, Zhang’s single camera calibration method was utilized to obtain the camera’s intrinsic parameters [5]. While tracking the laparoscopic surgical instrument, sequential images are captured by the camera. The images are then binarized with a specific threshold value for the markers (Figure 4). From a binarized image, blobs corresponding to the markers are detected. The image coordinates of each marker are set as the center point of each blob. The center point is calculated to sub-pixel accuracy.

Figure 3. Image processing procedures

In previous work by Doignon, the 3D positions and orientations of the laparoscopic surgical instrument could be estimated using collinear markers [4-5]. With Haralick’s algorithm, if more than 3 points are lying on a common line and the

584

S. Shin et al. / 3D Tracking of Surgical Instruments Using a Single Camera

distances between n collinear points are given, the position vector t and the relative orientation vector r of a set of n collinear points can be recovered. The input parameters for the estimation are (1) the camera’s intrinsic parameters, (2) the actual distances between the three static fiducial markers, and (3) the markers’ image coordinates. The actual gaps between the markers are measured by a caliper. The markers’ image coordinates (u, v) are taken from the image pixel coordinates. From the input parameters, t and r are obtained as follows: Let P0 = t , P1 = t + λ1r , … , Pi = t + λi r (i=0, 1, 2, 3) be collinear points where λi is the distance between P0 and Pi. The position of point P0 is chosen as the origin, the perspective projection Qi for Pi with homogeneous coordinates (ui, vi, 1) is expressed as Eq. (1)

ªui º [0 0 1][t + λi r ]««vi »» = K c (t + λi r ) «¬ 1 »¼

(1)

where K c is a (3 × 3) upper diagonal matrix whose components are the camera parameters. From Eq. (1), Haralick proposed a homogeneous linear system with a univariate matrix K c = diag( f , f , 1) .

[Ar

ªr º At ]« » = 0 ¬t ¼

(2)

A is a matrix of (2n × 6). It can be solved with n ≥ 3 distinct points. This linear system is reformulated as a classical optimization problem as Eq. (3). Τ (3) min Ar r + At t subject to r r = 1 where Ar and At are two (2n × 3) matrices containing the camera parameters and values of λi . The solution for Eq. (3) is obtained via singular value decomposition of the following symmetric matrix E. (4) E = ArΤ ( I − At ( AtΤ At ) −1 AtΤ ) Ar The eigenvector corresponding to the smallest eigenvalue of E is the relative orientation vector r, and the position vector t is given by t = −( AtΤ At )−1 AtΤ Ar r .

Figure 4. Image processing (left up: original image, left down: binary image)

1.3. Opening Angle Measurement of the Laparoscopic Surgical Grasper The angle of the grasper is correlated with the distance between static marker (P0) and dynamic marker (P3), because the angle and distance are represented as a straight line. This distance is solved by Eq. (2). Eq. (2) is then reinterpreted into Eq. (5) by λ3 .

S. Shin et al. / 3D Tracking of Surgical Instruments Using a Single Camera

§ λ3 f ¨¨ © 0

0

− λ3u3

f

0

λ3 f

− λ3v3

0

f

§ rx · ¨ ¸ ¨ ry ¸ − u3 ·¨ rz ¸ ¸¨ ¸ = 0 − v3 ¸¹¨ t x ¸ ¨t ¸ ¨ y¸ ¨t ¸ © z¹

585

(5)

Here, λ3 is the distance between the static marker and the dynamic marker, ( rx , ry , rz ) is the r vector and ( t x , t y , t z ) is the t vector. As Eq. (5) = 0, we can conclude λ3 by Eq. (6) and Eq. (7)

λ3 = λ3 =

u3t z − ft x frx − u3rz v3t z − ft y fry − v3rz

(6) (7)

The solutions for Eq. (6) or Eq. (7) are obtained by the λ3 value of the distance between static marker (P0) and dynamic marker (P3) using the image coordinates (u3, v3), the t vector and the r vector. From section 2.2, the t vector and the r vector are calculated.

2. Results

To assess the accuracy of the proposed method, we conducted two tests. First, an accuracy test of the 3D position and orientation of the laparoscopic surgical instrument was performed with a precisely designed test device. Second, the relationship between the opening angle of the graspers and the actual distance of the markers was checked. Finally, the 3D tracking module of the laparoscopic surgical instrument was effectively combined with virtual simulation software for use in laparoscopic surgery. 2.1. 3D Coordinates of Surgical Instrument In section 2.2, the 3D coordinates of the laparoscopic surgical instrument were calculated. An accuracy test was conducted which compared the measured values with the theoretical values, as shown in Figure 5. Columns were installed at known lattice positions with intervals of 50 mm by 50 mm for the accuracy tests. After locating the laparoscopic surgical instrument at the accuracy test device’s lattice positions, the 3D coordinates were calculated by the method described in section 2.2. The 3D coordinates for all positions of the accuracy test device were also calculated. One test set consisted of a total of 10 tests, all of which were performed successively to determine the test position in an effort to measure the repeatability. The standard deviations obtained were approximately 1 mm for each position. The results of two test sets are listed in Table 1. The resulting average error of the accuracy tests was determined to be 2.0855 mm.

586

S. Shin et al. / 3D Tracking of Surgical Instruments Using a Single Camera

Table 1. Results of the accuracy test (unit mm) Position 1 X

Position 2

Y

Z

X

Y

Z

Avg.

-197.39

-59.52

-2.57

-247.62

-59.89

-3.06

SD

0.4794

0.6477

0.0483

1.6903

0.3871

0.0516

Figure 5. Accuracy test device

2.2. Opening Angle of the Laparoscopic Surgical Grasper The actual distance was measured between the static marker P0 and the dynamic marker P3 with a caliper. The measured range of the graspers’ opening angle was from 0o to 85o. The range of the distances between P0 and P3 is from 11.42 mm to 13.72 mm. The opening angle of the graspers is assumed to be proportional to the distance between P0 and P3. Eq. (8) is given by the theoretical values 2.3 (8) y= x + 11.42 85

where x is the opening angle of the graspers and y is the distance between P0 and P3. As described in section 2.3, the distance between the static marker (P0) and dynamic marker (P3) can be calculated with the image coordinates (u, v) using Haralick’s algorithm. The results are shown in Figure 6. The average error between the theoretical values and the experimental values is 0.2131 mm.

Figure 6. Comparison between the theoretical value and the experimental value for the opening angle of the grasper

2.3. Integration with Simulation Software We combined the proposed surgical instrument tracking module with virtual simulation software that is used for laparoscopic surgery (Figure 7). When a user manipulates the actual laparoscopic surgical instrument, the 3D pose and the opening state of the actual device is measured. A virtual model of the surgical instrument is then manipulated in a virtual simulation environment in real-time. A boundary element method (BEM) is implemented for the physical simulation of a deformable model of the liver [6]. A raytraced collision detection method is used to check for collisions between the surgical tool and the liver model [7]. A real-time interactive simulation was realized at approximately 25 fps, including the time necessary for 3D tracking of the instrument

S. Shin et al. / 3D Tracking of Surgical Instruments Using a Single Camera

587

and the physical simulation of the liver model. The bottleneck for the calculation was determined to be the frame speed of the CCD camera.

Figure 7. Laparoscopic surgery simulation system

3. Conclusion

A novel method for the 3D tracking of a laparoscopic surgical instrument using a single camera and fiducial markers is proposed here. Compared with general computer vision technology in which the resolution is close to 2.0 mm, the accuracy of the proposed method for tracking a laparoscopic surgical instrument was considerable. However, the accuracy required in the medical simulation field is higher than that currently available with general computer vision technology. In addition, the tracking of laparoscopic surgical instrument greatly depends on the lighting conditions. A number of issues were considered in an effort to improve the accuracy of the proposed method. These included (1) using a camera capable of high resolution and a high speed frame, (2) providing optimized lighting conditions with uniform brightness, and (3) increasing the number of fiducial markers.

Acknowledgments

This research was supported by the Ministry of Culture, Sports and Tourism (MCST) and by the Korea Creative Content Agency (KOCCA) of the Culture Technology (CT) Research & Development Program of 2010.

References [1] [2]

[3] [4] [5] [6] [7]

C. Doignon: An Introduction to Model-Based Pose Estimation and 3D Tracking Techniques, Scene Reconstruction, Pose Estimation and Tracking (2007), 530 C. Doignon, F. Nageotte, B. Maurin, and A. Krupa: Pose estimation and feature tracking for robot assisted surgery with medical imaging, Unifying Perspectives in Computational and Robot Vision (2008) Vol. 8, 79-101 R. M. Haralick and L. G. Shapiro: Computer and Robot Vision, Addison Wesley Publishing(1992) G. Bradski and A. Kaehler: Learning OpenCV computer vision with the OpenCV library, O’REILLY(2008) Z. Zhang: A Flexible New Technique for Camera Calibration , IEEE Transactions on Pattern Analysis and Machine Intelligence (2000), 1330-1334 D. L. James and D. K. Pai: A unified treatment of elastostatic contact simulation for real-time haptics, Haptics-e: The Electronic Journal of Haptics Research (2001), Vol 2 Y. Kim: Mesh-to-Mesh Collision Detection by Ray Tracing for Medical Simulation with Deformable Bodies, 2010 International Conference on CYBERWORLDS (2010)

588

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-588

Perceptual Metrics: Towards Better Methods for Assessing Realism in Laparoscopic Simulators Ravikiran B. SINGAPOGU*, Christopher C. PAGANO*, Timothy C. BURG* and Karen JKL BURG* * Haptic Interaction Lab, Clemson University, Clemson. SC. USA [email protected]

Abstract. This work proposes a novel class of metrics for assessing haptic realism in laparoscopic surgical simulators. Results from a proposed perceptual metric are presented and discussed. Keywords. haptics, perception, laparoscopic simulators, laparoscopic training

Introduction and Background The number of laparoscopic procedures performed in the United States has seen a continual increase in the last decade. Consequently, there is a need to devise training systems that enable faster and more efficient skills training for novices in laparoscopy [1]. Though several Virtual Reality (VR) trainers are currently available, they have not been widely adopted in surgical skills labs [2]. One of the main reasons for this is the lack of realism in VR trainers [3]. Though computer-based trainers feature realistic graphics, most trainers do not simulate the haptic “feeling” arising from tool-tissue interactions [4]. The few simulators that have sought to incorporate simulated haptics have produced only a slight benefit in task performance [5],[6]. For example, Salkini and coworkers demonstrated that the addition of haptic feedback in a specific laparoscopy simulator produced no significant performance benefits [7]. One suggested reason for this is inaccurate or unrealistic haptics. Methods for the assessment of “face validity”, the degree of realism of the simulator, are not well established in the current literature. Most studies reporting face validity for simulators have used a questionnaire-based approach. Subjects were asked to use a Likert-type scale to rate aspects of the simulators’ realism and “feel” [8]. This approach to measuring realism suffers from lack of objectivity and other biases. However, to design better simulators, better metrics for realism need to be designed and evaluated [9]. This work proposes a method to measure the haptic realism of VR simulators using “perceptual metrics.”

R.B. Singapogu et al. / Perceptual Metrics: Towards Better Methods

589

1. Materials and Methods Several studies have shown that humans are capable of accurately estimating length of unseen sticks by holding and wielding them [10],[11]. In this study, sticks of various lengths were rendered using a haptic device and subjects were asked to estimate their lengths based on feeling alone. Eight wooden rods which varied in the lengths and inertial properties were selected for this experiment (Table 1). The haptic interface device used in this experiment was the 5 degree-of-freedom Haptic Wand (Quanser Inc., Canada). Euclidean position and orientation of user’s motion is sensed and is used by the dynamic model of the stick. Force and torque are then calculated based on Newton-Euler laws for 6D motion. The software platform controlling the device consisted of MATLAB (v 7.1) with Real Time Workshop (v 2.1) and Wincon (v. 5.0). The experiment had two sessions: real sticks and virtual sticks. In the real sticks session subjects were given physical sticks that were occluded by a black curtain that eliminated visual feedback. Subjects were asked to wield the stick and estimate its length on a reporting scale. The reporting scale consisted of a sliding pointer, movable by the user to a position from 0-120 cm from the origin of the scale. No markings were visible on the user’s side; the other side had a centimeter scale and when the user estimated the stick length, the reading was noted. In the virtual sticks session, the same set of sticks were rendered by the haptic device and users were asked to wield the virtual stick to estimate length using the same reporting scale. The haptic device was occluded with a black curtain and was not visible to the user. Eight subjects participated in this experiment after providing informed consent. The participants were students between 18-25 years of age. Each user was randomly assigned to receive either the real or virtual session first. Within each session the eight sticks were given twice in a random order.

2. Results and Discussion After data was collected, correlation analysis was performed separately for each of the sessions. In both sessions, actual length was correlated with estimated length. Results of the eight subjects are shown in Table 1, all values are correlation coefficients. The mean value of correlation coefficient for the real sticks was 0.921, while for the virtual sticks it was 0.845. All correlation coefficients had a p-value of < 0.01. It was expected that the correlation coefficient for real sticks would be high (approximately .90) in keeping with previous results. The correlation coefficient of virtual sticks was expected to be lower than for real sticks. However, the closer the virtual correlation value is to the real value, the greater the haptic realism of the simulator. The high virtual value (0.845) in this experiment validates the realism of the haptic device and rendering algorithm.

3. Conclusions and Future Work Can a haptic device accurately render the feel of real surgical instruments and tooltissue interaction? How can the degree of realism of the simulator be accurately measured? This work points to a paradigm for measuring haptic realism using

590

R.B. Singapogu et al. / Perceptual Metrics: Towards Better Methods

“perceptual metrics.” In this study, the degree of realism of the virtual stick was measured by comparing it with real sticks using the perceptual metrics of perceived length. Face validity of haptic simulators can thus be measured using this paradigm, with other haptic perceptual metrics such as stiffness and texture estimation being used to measure other aspects of simulator realism. Table 1. left, correlation coefficients of 8 participants, right, rendered virtual “stick” properties Subject

1 2 3 4 5 6 7 8

Correlation Correlation Coefficient Coefficient Real Sticks Virtual Sticks 0.851* 0.934* 0.762* 0.884* 0.874* 0.903* 0.892* 0.964* 0.769* 0.949* 0.866* 0.837* 0.841* 0.970* 0.921* 0.936* * = p-value < 0.01

Stick

Length

Mass

Inertia

Density

Moment

1

0.50

0.0312

0.0026

0.0624

0.0078

2

0.57

0.0384

0.0042

0.0674

0.0109

3

0.69

0.0508

0.0081

0.0736

0.0175

4

0.80

0.0665

0.0142

0.0831

0.0266

5

0.85

0.0474

0.0114

0.0558

0.0201

6

0.90

0.0689

0.0186

0.0766

0.0310

7

0.95

0.0613

0.0185

0.0645

0.0291

8

1.00

0.0726

0.0242

0.0726

0.0363

References [1] [2]

[3]

[4] [5]

[6]

[7]

[8]

[9]

[10] [11]

K. Roberts, R. Bell, and A. Duffy, “Evolution of surgical skills training,” World Journal of Gastroenterology: WJG, vol. 12, May. 2006, pp. 3219-3224. A.S. Thijssen and M.P. Schijven, “Contemporary virtual reality laparoscopy simulators: quicksand or solid grounds for assessing surgical trainees?,” American Journal of Surgery, vol. 199, Apr. 2010, pp. 529-541. K. Gurusamy, R. Aggarwal, L. Palanivelu, and B.R. Davidson, “Systematic review of randomized controlled trials on the effectiveness of virtual reality training for laparoscopic surgery,” The British Journal of Surgery, vol. 95, Sep. 2008, pp. 1088-1097. S.M.B.I. Botden and J.J. Jakimowicz, “What is going on in augmented reality simulation in laparoscopic surgery?,” Surgical Endoscopy, vol. 23, Aug. 2009, pp. 1693-1700. L. Panait, E. Akkary, R.L. Bell, K.E. Roberts, S.J. Dudrick, and A.J. Duffy, “The Role of Haptic Feedback in Laparoscopic Simulation Training,” Journal of Surgical Research, vol. 156, Oct. 2009, pp. 312-316. P. Kanumuri, S. Ganai, E.M. Wohaibi, R.W. Bush, D.R. Grow, and N.E. Seymour, “Virtual reality and computer-enhanced training devices equally improve laparoscopic surgical skill in novices,” JSLS, Journal of the Society of Laparoendoscopic Surgeons, vol. 12, 2008, pp. 219–226. M.W. Salkini, C.R. Doarn, N. Kiehl, T.J. Broderick, J.F. Donovan, and K. Gaitonde, “The role of haptic feedback in laparoscopic training using the LapMentor II,” Journal of Endourology / Endourological Society, vol. 24, Jan. 2010, pp. 99-102. I.D. Ayodeji, M. Schijven, J. Jakimowicz, and J.W. Greve, “Face validation of the Simbionix LAP Mentor virtual reality training module and its applicability in the surgical curriculum,” Surgical Endoscopy, vol. 21, Sep. 2007, pp. 1641-1649. B.M.A. Schout, A.J.M. Hendrikx, F. Scheele, B.L.H. Bemelmans, and A.J.J.A. Scherpbier, “Validation and implementation of surgical simulators: a critical review of present, past, and future,” Surgical Endoscopy, vol. 24, Mar. 2010, pp. 536-546. M.T. Turvey, “Dynamic touch,” American Psychologist, vol. 51, Nov 1996, pp. 1134–1152. C.C Pagano, and P.A. Cabe, "Constancy in dynamic touch: Length perceived by dynamic touch is invariant over changes in media," Ecological Psychology, vol. 15, No. 1, 2003, pp. 1-18.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-591

591

Role of Haptic Feedback in a Basic Laparoscopic Task Requiring Hand-eye Coordination Ravikiran B. SINGAPOGU*, Christopher C. PAGANO*, Timothy C. BURG*, Karen JKL BURG* and Varun V. PRABHU* * Haptic Interaction Lab, Clemson University, Clemson. SC. USA [email protected]

Abstract. This work discusses the role and importance of haptic feedback and simulator training for simple laparoscopic tasks akin to the FLS peg-transfer task. Results from a study designed to examine haptics for this purpose are discussed. Keywords. Haptics, skills training, laparoscopic simulators, laparoscopic training

Introduction and Background The role and utility of haptic feedback in laparoscopic surgery is a topic of much debate in the current literature [1]. Recently, quantitative haptic information recorded during in vivo laparoscopy has been documented and demonstrates the presence of haptic (kinesthetic) feedback [2]. Further, these force values lie within a range that are perceivable by human operators [3]. The presence of haptics during surgery raises important questions for laparoscopic training. For example, what type of training will lead resident trainees to efficiently perceive and process haptic information during surgery? Also, what specific tissue properties are more readily perceived by haptic feedback? The Fundamentals of Laparoscopic Skills curriculum is used as the standard for laparoscopic skills training in U.S. medical schools [4]. The technical component of this program consists of five tasks ranging for basic hand-eye coordination to advanced force application and suturing. Previous studies have shown that haptic feedback is useful during force application tasks as well as in determining properties like tissue stiffness [5],[6]. However, the role of haptic feedback for learning hand-eye coordination laparoscopic skills is not well understood. This study investigated the role of haptic feedback in a FLS-based peg transfer-like task.

1. Materials and Methods For this study, virtual “blocks” of three colors were created with identical physical properties. The virtual environment was created using the Chai 3D library (www.chai3D.org). The physics of the environment was handled by Open Dynamics Engine (ODE) which contains collision detection and collision response algorithms.

592

R.B. Singapogu et al. / Role of Haptic Feedback in a Basic Laparoscopic Task

The virtual blocks were manipulated via a standard haptic interface, the Novint Falcon®. The low-level device control was done using the Chai 3D haptic library (Figure 1). The users’ goal was to stack the virtual blocks into sets of three according to their color. Users performed this stacking task with haptic feedback from the device and without haptic feedback. The task of stacking was chosen because it was used in previous studies for basic laparoscopic skill learning [7]. After users completed the virtual tasks, they performed a similar stacking task in the real world. A custom laparoscopic box trainer was built for this purpose using published specifications [8]. One standard laparoscope, inserted through the incision, was used to stack metal nuts of 1.7 cm diameter (Figure 1). Akin to the virtual task, the real task comprised of stacking nine nuts into groups of three according to their color. Participants of the experiment were first briefed about experiment’s objectives and randomly assigned to receive either the haptics or non-haptics virtual task first. The metric for assessing performance was time to completion measured in seconds. After completing both virtual tasks, subjects performed the real task of stacking metal nuts in the physical trainer. Time to complete the task was also used for performance assessment of the real task Ten subjects participated in this experiment after providing informed consent. The participants were students between 18-25 years of age. Recorded time data from all three sessions is shown in Table 1 Table 1. Time to complete stacking task in all three sessions Subject

No Haptics (seconds)

Haptics (seconds)

Real (seconds)

1

165

95

195

2

141

65

150

3

194

117

145

4

119

116

170

5

148

54

99

6

166

143

111

7

99

51

94

8

272

140

300

9

246

104

218

10

182

122

102

Table 1. Time to complete stacking task in all three sessions Figure 1. Physical laparoscopic trainer setup used for task, top virtual interface for haptic task

R.B. Singapogu et al. / Role of Haptic Feedback in a Basic Laparoscopic Task

593

2. Results and Discussion The hypotheses of the experiment are: (1) time to completion with haptics will be significantly shorter than without haptics and, (2) time scores from the haptic session will be more correlated to real task time scores than the non-haptic session scores. Statistical analysis was performed using Minitab (v 15.1). To investigate the first hypothesis a Mann-Whitney U-test was performed to compare the haptic and non-haptic scores. Results showed that scores were significantly different at a p-value of < 0.01. The median completion times were 110 and 165 seconds for the haptics and non haptics sessions, respectively. To investigate the second hypothesis, a correlation analysis was performed between the real scores and the haptics scores as well as real scores and the non-haptics scores. Results showed that non-haptic session scores were significantly correlated with real task scores (r=.747, p-value < 0.05) whereas haptic scores were not significantly correlated with real task scores (r=.432, p-value=.21). This result, contrary to the hypothesis, shows no correlation between haptic scores and real task scores.

3. Conclusions and Future Work The results of this study suggest that haptic feedback does not significantly affect task performance for basic hand-eye coordination tasks in laparoscopic training. This observation confirms earlier results from Chmarra and coworkers who suggested that haptic feedback was not necessary for basic laparoscopic tasks primarily involving hand-eye coordination skills. Consequently, when teaching these skills to residents, visual feedback is the primary sensory mode of learning and should be focused on accordingly.

References [1]

[2] [3] [4]

[5] [6]

[7] [8]

O. van der Meijden and M. Schijven, “The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: a current review,” Surgical Endoscopy, vol. 23, Jun. 2009, pp. 1180-1190. G. Picod, A.C. Jambon, D. Vinatier, and P. Dubois, “What can the operator actually feel when performing a laparoscopy?,” Surgical Endoscopy, vol. 19, Jan. 2005, pp. 95-100. C. Cao, M. Zhou, D. Jones, and S. Schwaitzberg, “Can Surgeons Think and Operate with Haptics at the Same Time?,” Journal of Gastrointestinal Surgery, vol. 11, Nov. 2007, pp. 1564-1569. G.M. Fried, L.S. Feldman, M.C. Vassiliou, S.A. Fraser, D. Stanbridge, G. Ghitulescu, and C.G. Andrew, “Proving the value of simulation in laparoscopic surgery,” Annals of Surgery, vol. 240, Sep. 2004, pp. 518-525; discussion 525-528. M. Chmarra, J. Dankelman, J. van den Dobbelsteen, and F. Jansen, “Force feedback and basic laparoscopic skills,” Surgical Endoscopy, vol. 22, Oct. 2008, pp. 2140-2148. P. Lamata, E.J. Gomez, F.L. Hernández, A. Oltra Pastor, F.M. Sanchez-Margallo, and F. Del Pozo Guerrero, “Understanding perceptual boundaries in laparoscopic surgery,” IEEE Transactions on BioMedical Engineering, vol. 55, Mar. 2008, pp. 866-873. S. Badurdeen, O. Abdul-Samad, G. Story, C. Wilson, S. Down, and A. Harris, “Nintendo Wii videogaming ability predicts laparoscopic skill,” Surgical Endoscopy, vol. 24, 2010, pp. 1824-1828. J.D. Beatty, “How to build an inexpensive laparoscopic webcam-based trainer.” BJU international, vol. 96, 2005, p. 679.

594

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-594

A Model for Flexible Tools Used in Minimally Invasive Medical Virtual Environments Francisco SOLERa, M. Victoria LUZONa, Serban R. POPb, Chris J. Hughesb, Nigel W. JOHNb,1 and Juan Carlos TORRESa a University of Granada, Spain b Bangor University, United Kingdom

Abstract. Within the limits of current technology, many applications of a virtual environment will trade-off accuracy for speed. This is not an acceptable compromise in a medical training application where both are essential. Efficient algorithms must therefore be developed. The purpose of this project is the development and validation of a novel physics-based real time tool manipulation model, which is easy to integrate into any medical virtual environment that requires support for the insertion of long flexible tools into complex geometries. This encompasses medical specialities such as vascular interventional radiology, endoscopy, and laparoscopy, where training, prototyping of new instruments/tools and mission rehearsal can all be facilitated by using an immersive medical virtual environment. Our model recognises and uses accurately patient specific data and adapts to the geometrical complexity of the vessel in real time. Keywords. Catheter, guide wire, endoscope, virtual environment, real time

Introduction With the increasing demand for computer based medical simulation, the requirement for realistic visualization has become even more important. Often it is not possible for the physics of every component within the simulation to be calculated correctly in realtime due to computational restraints [1, 3, 4]. Each component within the simulator can be categorized as either essential or supporting to the simulator, where essential components will require the use of the majority of the computational resources. Methods with low computational requirements are typically needed to realistically estimate any supporting components [3]. Varied “real patient” specific data can be acquired from medical scanners in a hospital as a series of two dimensional images. A stack of such images forms a volume of data representing the internal anatomy of the patient where each volume element (or voxel) represents an intensity value on a grid in three dimensional space. Typically, a time consuming segmentation step is then required to extract the 3D geometry of the anatomy of interest [1]. This step cannot be achieved fully automatically and is prone to error. Our approach will employ the medical scanner data in its raw format, however, without any other additional transformations and applies a novel computer generated 1

Corresponding Author: Nigel W. John, Bangor University, UK. E-mail; [email protected]

F. Soler et al. / A Model for Flexible Tools

595

insertion model for the tools used in minimally invasive procedures. This computer model is focused on speed, accuracy physical relevance and integrates successfully into a medical virtual environment. Currently there are very few voxel-based models for guide wire insertion into complex geometries of the human vascular system due to the high complexity and inherent computational demands. However, our model takes advantage of the physical aspects of the problem and limits the computations and the number of nodes describing the virtual tool to a minimum. Moreover, our model will be able to take into account motion effects due to pulsation, respiration or bio-fluid flow (blood, urine, plasma, etc.). Our simulator includes a specific model of bio-fluids flow that is integrated directly and in real-time [2]. The influence of blood flow is an essential component for many procedures in interventional radiology simulators, such as the injection of contrast medium whilst using fluoroscopy. However, in current simulations, blood flow generally plays a supporting role and can contribute to poor face validity. Although the flow does not necessarily need to be accurately computed, it does have to behave realistically and in real-time. The complexity of blood flow represents a challenge for conventional methods of simulation, even at a macroscopic scale (such as flow in arteries, veins). We have used a new computer simulated model to visualize blood flow in arteries using a flocking algorithm approach [7]. In this model each layer of fluid behaves as a flock of blood particles (or boids), interconnected by the parameters that govern the flow dynamics. The method can be successfully used to represent blood flow in the vascular structure and combined with complex haptic interactions of guidewires and catheters in interventional radiology. The blood particle boids react and adjust their position in real-time, hence at each guidewire movement they will recognize the obstacle and the change in the domain and will remodel and reposition in real-time. Accuracy is preserved and the result will be a clear image of the blood velocity field or the tracer concentration field.

1. Methods The research groups at Bangor and Granada are collaborating to develop a voxel-based tool insertion model suitable for catheters and guidewires. This virtual tool is represented by “stretches” (i.e. the amount of the tool inserted in a specific time frame or the length of tool inserted in a “push sequence”); each stretch involves three nodes, two nodes placed at the extremes and the third placed between them and called generically the “control node” (Figure 1). The position of the control node is essential for the tool bending (amount and direction) at the next stretch (if necessary, otherwise no control node is considered). The bending of the tool in a specific node is modelled by associating a “bending-torsion” parameter to the node that employs the tool’s physical characteristics and its interaction with the walls whilst being pushed by the operator. As a general rule, when the position of the control nodes in the wire is computed, the simulation algorithm selects an optimal position considering the value of the “bending-torsion” parameter, the geometry of the obstacle and the force feedback of the push. After any push operation (stretch creation) the previous control nodes, which are considered non-relevant for the motion, are discarded, a procedure which significantly limits the number of nodes in the model, increasing the speed of the simulation. However, when the node head hits a boundary voxel the simulation algorithm computes a “friction” value for each valid voxel in the head node’s

596

F. Soler et al. / A Model for Flexible Tools

neighbourhood, and the voxel with the minimum “friction” value is selected as new location for the head node. The term “friction” in this context is generic and represents the resistance of the head node to changes in the tool’s direction and orientation.

Figure 1. Left: Guidewire representation layout; Right: stretch collision

Each wire node is related with a unique voxel in the voxelized model. We start from a voxelized model built from the same volumetrically model used for generating the polygonal model. A threshold model is applied to characterize voxels within the domain. During the simulation, some collision tests of the last stretch with non-valid voxels are performed. This straightforward test consists on checking if some voxel wrapping each semi-stretch is a non-valid voxel. This set of voxels is computed using a displacement of the vector connecting the two nodes of each semi-stretch. While the head node is moving, the lines connecting the nodes of the last stretch may collide with a non-valid voxel or more. In this situation, the simulation algorithm subdivides the last stretch adding a new stretch in order to adapt the wire representation to the shape of the vessel (Figure 1). However, if the threshold for “bending-torsion” parameter of any of these nodes is exceeded, the movement of the head node is aborted and no subdivision is performed. Any “push” or “pull” operation of the guidewire/catheter is performed using the described procedures and adapting at each step according with the shape of the domain. Blood flow simulation is built using artificial intelligent particles called boids. We consider that the boids flocking group behaviour matches the characteristics of laminar flow (collision avoidance, velocity matching, and flock centring) and is therefore suitable for modelling channel flows. Although a boids model can only be used for visualization purposes, we produce good results by performing a qualitative comparison of our method with existing fluid particle based simulation. Our model uses the idea that each layer of fluid behaves as a flock, interconnected by the parameters that govern the flow dynamics and the fluid physical properties. At the macroscopic level (arterial flow) blood can be considered as a Newtonian fluid and represented using an underlying system of particles. Many structural similarities with existing particle dynamics systems for fluids are considered (e.g., the kernel function in Smooth Particle Hydrodynamics (SPH) is replaced by the flock neighbourhood rule; however the search for nearby particles is still performed in the usual way) (Figure 2, right). The number of particles inside the domain, during the entire simulation, is kept constant in order to satisfy the system’s mass conservation rule. As in SPH method, each particle carries its own physical quantities such as mass, speed and position, which ensure the control over the main physical parameters of the fluid.

F. Soler et al. / A Model for Flexible Tools

597

Figure 2. Left: Stucked particle are dropped from the layer; Right: Comparison between the SPH model and our approach

The simulation consists of two main stages. In the first stage, particles, grouped in layers, are introduced into the domain at equally constant speed. The layers do not mix and if any particle touches the domain’s boundaries it is rendered motionless (velocity reduced to zero, the no-slip condition). The blood particle boids will adapt, recognizing the environment and match their speed and location to the given domain (Example: particles introduced in a tube with homogeneous walls will match their velocity according to Poiseuille’s Law for channel flows). Any obstacle is avoided gradually and particles that during the transition lose speed, and eventually stop, are dropped from their corresponding layers. In this stage, the fluid layers don’t have one single particle leader; the entire group of particles which form the layer act like leaders. This behaviour is implemented when one or more leading particles become “stuck” (with zero velocity) and the layer is forced to move further without them. The number of particles inside the domain during the simulation is kept statistically constant, in order to respect the conservation of mass principle. In the second stage, the simulated flow becomes stable (does not sustain any change over time or the flow is not timedependent anymore). The particles move according to the layer trajectories and any changes in the domain’s geometry automatically triggers the first stage again and the particles start to adapt from the position where that change was made. The results of the blood flow visualization using boids has been compared with existing benchmarks, in particular non-uniform channel flows, with or without obstacles.

2. Results The geometrical and physical aspects of the virtual patient play a very important role in our algorithm's behaviour, its high adaptability being able to minimise the computations required and the information needed to describe the virtual tool. When the inserted virtual guide wire hits a wall of a vessel or an obstacle such as calcification (hardened blockage within an artery) then the tool will respond appropriately. The algorithm dynamically adds bending or friction properties to the tool at the collision point making its further advancement behave accurately (Figure 3). We also use a haptics interface so that the operator can "feel" the force cues when the movement of the tool does become constricted. After the obstacle is passed the extra physical information that was applied will be deleted, preventing the overall computational demands from growing too large. Our model has a very high adaptability being capable to simulate in real time the behaviour of different types of catheters with various

598

F. Soler et al. / A Model for Flexible Tools

properties (The right image in Figure 3 depicts guidewires that have extreme unrealistic values in order to cover the entire example domain).

Figure 3. Guidewire in artery. Control nodes visible (left). Rigid, normal and soft catheters are simulated (right)

3. Conclusions In summary, unless push or pull operations were performed, the tool representation keeps its length constant at every time point, which together with the physically-based parameters involved makes this model extremely fast and accurate. One example application for our model, from interventional radiology, is the accurate simulation of the insertion of a catheter and guidewire into an artery as a part of the Seldinger Technique, and the real-time interaction of these tools with each other and the vessel walls [5, 6]. When combined with our ultrasound and fluoroscopy interfaces, and realtime blood flow simulation, our model can provide a complete environment for medical training, for procedures such as angioplasty and stent placement within a vessel.

References [1] S. Bhata, T. Kesavadasa, K.R. Hoffmann “A physically-based model for guidewire simulation on patient-specific data“, International Congress Series, 1281, 479–484, 2005. [2] C.J. Hughes, S.R. Pop, N.W. John, “Macroscopic blood flow visualization using boids”, Proceedings of the 23rd International Congress of Computer Assisted Radiology and Surgery, Berlin, Germany. 4 (supplement 1), S68-S69, 2009. [3] N.W. John, C.J. Hughes, S.R. Pop, F.P. Vidal, O. Buckley, "Computational Requirements of the Virtual Patient", First International Conference on Computational and Mathematical Biomedical Engineering (CMBE09), Swansea, United Kingdom, June 2009, 140-143. [4] S.D. Laycock, A.M. Day, “Incorporating haptic feedback for the simulation of a deformable tool in a rigid scene”, Computers & Graphics, 29, 341-351, 2005. [5] W. Lawton, R. Raghavan, S.R. Ranjan, R.R. Viswanathan, “Tubes in tubes: catheter navigation in blood vessels and its Applications”, International Journal of Solids and Structures, 37, 3031-3054, 2000. [6] V. Luboz, C.J. Hughes, D.A. Gould, N.W. John, F. Bello, "Real-time Seldinger Technique Simulation in Complex Vascular Models", International Journal of Computer Assisted Radiology and Surgery. 4(6), 589-596, 2009. [7] C.W. Reynolds, “Flocks, herds and schools: A distributed behavioural model”, ACM SIGGRAPH Computer Graphics. 21(4), 25-34, 1987.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-599

599

Segmentation of 3D Vasculatures for Interventional Radiology Simulation Yi SONG a,1 , Vincent LUBOZ b, Nizar DIN b, Daniel KING b, Derek GOULD c, Fernando BELLO b and Andy BULPITT a a School of Computing, University of Leeds b Department of Biosurgery and Surgical Technology, Imperial College London c Department of Radiology, Royal Liverpool University Hospitals

Abstract. Training in interventional radiology is slowly shifting towards simulation which allows the repetition of many interventions without putting the patient at risk. Accurate segmentation of anatomical structures is a prerequisite of realistic surgical simulation. Therefore, our aim is to develop a generic approach to provide fast and precise segmentation of various virtual anatomies covering a wide range of pathology, directly from patient CT/MRA images. This paper presents a segmentation framework including two segmentation methods: region model based level set segmentation and hierarchical segmentation. We compare them to an open source application ITK-SNAP which provides similar approaches. The subjective human influence such as inconsistent inter-observer errors and aliasing artifacts etc. are analysed. The proposed segmentation techniques have been successfully applied to create a database of various anatomies with different pathologies, which is used in computer-based simulation for interventional radiology training. Keywords. Level set, hierarchical segmentation, simulation.

Introduction Interventional radiology (IR) uses angiographic imaging to guide specialized instruments inside vascular systems through tiny incisions. Computer-based simulation proposes an alternative to traditional training since it offers a safe framework to practice specific skills as often as needed. CRaIVE2 is developing such an augmented reality training simulator. One of the key tasks is patient specific segmentation of highly variable vascular networks in terms of shape and texture that represent vascular disease. Many vascular segmentation techniques have been proposed and implemented in recent years, see [1,2,3] for recent reviews. Model-driven, knowledge based image analysis is one of the techniques often used. It aims to describe and capture a priori information regarding the shape, size and position of each structure. Deformable models [4], statistical shape models [5] and probabilistic atlases [6] can also be employed. The limitation of these approaches is the generation of a suitable template 1

Corresponding Author: School of Computing, University of Leeds, LS2 9JT, UK; E-mail: [email protected]. 2 Collaborators in Radiological Interventional Virtual Environments, http://www.craive.org.uk

600

Y. Song et al. / Segmentation of 3D Vasculatures for Interventional Radiology Simulation

model which is able to capture both natural variations and deformations caused by pathology. As our system needs to handle various abnormalities, sometimes with extreme shape deformation, current model-driven approaches are not well suited to our task. Approaches based on the level set method [7,8] have also been used successfully for medical image segmentation. However, the problem of “leaks” on the boundary is still open to research. Other drawbacks include the non-trivial work of tuning parameters and placing seeds. Skeletonization techniques [9] generate very accurate vasculatures but they often require a lot of user interaction to improve the skeleton, especially around artifacts. In this paper we propose a segmentation framework including two general solutions to overcome such limitations: a region model based level set method and an interactive hierarchical method. In section 2, the two methods are compared with ITKSNAP3 [10] which provides both a classic level set approach (semi- automatic) and interactive segmentation tools.

1. Segmentation Methods 1.1. Region Model Based Level Set Method (RMLS) This method [11] segments anatomic structures from CT/MRA images by evolving an implicitly described active surface through the zero level set of a level set function. Traditional level set based approaches require the placement of seeds to initialize the algorithm. Since the segmentation result is sensitive to both the seed positions and seed numbers, an inaccurate initialization can result in slow convergence to the targeted boundary and increase the risk of leaks. To overcome the problem, we replace the seed placement by a region model, which is a close approximation to the actual structure shape. This can significantly reduce the time of convergence and thus reduce the risk of “leaking” during the level set evolution process.

Figure 1. Edge enhancement. (a) Original slice in sagittal view (image I1). The area inside the rectangular box indicates the location of the abdominal aorta. (b) The gradient magnitude of image I1 (only partial image I1, i.e. the area inside the rectangular box of (a) is displayed). (c) Spurious edge suppression. (d) Image I2 after subtracting the enhanced edge from image I1 (e) Feature map of image I0.

3

SNAP is an application developed on the National Library of Medicine Insight Segmentation and Registration Toolkit (ITK). http://www.itksnap.org

Y. Song et al. / Segmentation of 3D Vasculatures for Interventional Radiology Simulation

601

First, we apply edge enhancement (Figure 1). The gradient magnitude of image I1 (Figure 1a) at each pixel is computed (Figure 1b) and a lower hysteresis threshold T is applied to the result to suppress spurious edges, as shown in Figure 1c. T is estimated using the histogram of the gradient image. The remaining edges are enhanced to the maximum intensity value C of image I1, Eq. (1). I 1' ( x)

C if 0

I 1 ( x) else

T

(1)

Next, we employ mathematical morphology operations on the enhanced image I2 = I1 - I1 (Figure 1d, 1e) and capture the major object structures to form the region model which is then embedded into the level set function for further refinement. As the level set starts from a close approximation to the actual structure shape, this approach can improve the accuracy of results and dramatically decrease processing time. Consequently, the risk of leaks which is common to previous level set based approaches can be minimized. In our approach, a user selects intensity values of the target tissues and no seed placement is required. 1.2. Hierarchical Segmentation (HS) An alternative segmentation algorithm, not dependent on the gradient of the dataset, is provided by hierarchical segmentation. This algorithm starts by creating a hierarchy in a bottom up approach. Initially, there are as many regions as there are voxels in the dataset. These regions are then merged together iteratively based on edge strength between adjacent regions to converge into homogenous regions of similar intensities. Combining the homogeneity and edge stability constraints leads to a single edge measure [12]: I . arctan

I 2

(2)

I

I

Where I is a region mean intensity, 2

is the absolute difference between the two

adjacent regions’ mean intensities, and I is the average of the mean Laplacians of the two regions. At the end of this process, the initial hierarchy is computed. The method subsequently allows the user to define interactively the inside and outside of the required anatomical structure by marking seed points in the image (Figure 2, left). Starting from these points, the algorithm iteratively merges adjacent regions of the hierarchy based on their intensity and therefore separates the inside and outside of the anatomical structure. Griffin et al. [12] proposed to apply this basic hierarchical segmentation to segment a whole dataset in two different ways: perform it on each 2D slice of a whole dataset, or use it on several slices and use the same seeds to propagate the segmentation in 3D to all other slices automatically. The first method can be extremely slow (depending on the number of slices) while the second one is often not very accurate as the intensity of the wanted anatomical structure might vary along the dataset.

602

Y. Song et al. / Segmentation of 3D Vasculatures for Interventional Radiology Simulation

Figure 2. Left: Hierarchical segmentation of the aorta arch. Right: The intersection of the 2D segmentations produces seeds in perpendicular slices, in black for the exterior and in white for the interior seeds.

We propose a third way to adapt it: “2.5D propagation”. It is based on the segmentation of few 2D slices and a new propagation method computing the segmentation results more quickly and accurately. In the automatic propagation phase, a perpendicular slice intersects the user’s 2D segmentations to form lines of seeds (Figure 2). These seeds are used to perform new 2D segmentations on the next slices as they provide enough information for a hierarchical segmentation in those perpendicular slices. Because this algorithm only performs a 2D perpendicular segmentation at a time and each resulting segmentation is stored, it requires significantly less memory than the 3D propagation algorithm, and it is therefore faster. The user can place more seeds, which leads to more accurate results. No parameter or threshold has to be set for this technique.

2. Evaluation 2.1. Evaluation Datasets and Methods The RMLS and HS methods have been evaluated on 20 patient datasets. 19 cases are contrast enhanced images with average resolution of 0.7×0.7×1mm3 and typically 600 slices with 512x512 in-plane pixels (provided by St. Mary’s London Hospital). The remaining dataset is a magnetic resonance angiography image provided by Royal Liverpool Hospital with resolution of 1.5×1.5x2mm3 and 384 slices with 256x256 pixels. The criteria of segmentation includes ascending aorta, aortic arch, brachiocephalic arteries, common carotids, subclavian arteries, descending aorta, celiac artery, superior mesenteric artery, renal artery, common iliac arteries, external lilac arteries, internal iliac arteries and common femoral arteries, if visible in each dataset. ITK-SNAP provides an intuitive and easy to use interactive segmentation tool to assist manual segmentations which are taken as evaluation references. As ITK-SNAP also provides a classic semi-automatic level set implementation, we study the differences between our RMLS approach and the SNAP level set approach. The results of manual segmentation, ITK-SNAP (level set) and HS were provided by Imperial College London (and by the same person). The results of the RMLS method were completed at the University of Leeds. Both research centres used the same PC configurations and recorded the segmentation time. The evaluation of accuracy was conducted by using the MESH4 tool. The metric is the symmetric mean error which is the approximation of Hausdorff distance between discrete 3D surfaces represented by triangular meshes. We created surface meshes from 4

Measuring Error between Surfaces using Hausdorff distance: http://mesh.berlios.de/

Y. Song et al. / Segmentation of 3D Vasculatures for Interventional Radiology Simulation

603

segmentation results using the marching cubes algorithm from the VTK5 library. The same parameters are applied on all datasets. 2.2. Performance 2.2.1. Accuracy For each of the three methods, the symmetric mean error (SME) of each patient is plotted in Figure 3. Summary results for Figure 3 are given in Table 1, including the minimum, maximum, average and RMS of the symmetric mean errors of the 20 datasets. 2.2.2. Efficiency The evaluation was conducted on an Intel Core2 2.66GHz processor. As expected, even with the assistance of the interactive segmentation tool, the manual segmentation was still the most time consuming, taking on average 527.8mins with a standard deviation (SD) of 182mins. Although HS is an interactive method, it is much more efficient with a mean of 104mins and a SD of 37mins. Since the ITK-SNAP level set implementation involves many time-consuming seeds placement operations, it takes a mean of 120mins and a SD of 60mins. On the contrary, RMLS does not require seeds placement and takes less time to converge, it takes the least time with an average of 10 minutes. Figure 4 gives the running time for each patient with each of the three methods.

Figure 3. Comparison of the symmetric mean errors for each dataset and for ITK-SNAP, RMLS and the HS.

Figure 4. Graph of time efficiency for each segmentation method. 5

Visualization Toolkit (VTK): http://www.vtk.org/

604

Y. Song et al. / Segmentation of 3D Vasculatures for Interventional Radiology Simulation

2.3. Human Factors As all the methods require human interventions, the subjective influence from observers is unavoidable. For example, although there is a basic segmentation guidance on which part of the vasculature should be segmented, it is impossible to define case by case on how far those tiny branches should be represented in the segmentation result (as shown in Figure 5). In this study, the manual segmentation, HS and ITK-SNAP (level set) segmentation were completed by one person with a medical background, while the RMLS was done by a different person from a non-medical background. Therefore, the RMLS results include larger artifacts than other approaches. These artifacts mainly involve inconsistent inter-observer errors. Figure 5 visualizes a case with the largest symmetrical mean errors. As marked by the circles, the blue depicts those voxels which are missed by the RMLS but included by the manual segmentation; the yellow illustrates voxels which are missed by the manual segmentation but correctly identified by the RMLS. Here, red indicates voxels segmented by both methods. The segmentation resolution can also cause differences between two methods. This is also illustrated in Figure 5, where the descending aorta has been amplified and displayed at the left corner of (a) and (b), respectively. As shown in Figure 5b, being completed slice by slice on DICOM data, the manual segmentation normally has many aliasing artifacts in the result. The HS method is based on 2.5D propagation which requires fewer 2D slice segmentations. Therefore its results contain less aliasing artifacts compared to manual segmentation. RMLS and ITK-SNAP (level set) are both 3D segmentation approaches, resulting in smoother blood vessel walls, as shown in Figure 5a. HS and ITK-SNAP (level set) demand users to place relatively large amounts of seed points at appropriate positions. Consequently human operation time and the computational time are different from case to case. On the other hand, requiring only minimal user interventions, the RMLS method can be used by researchers with little anatomy knowledge. Another advantage is that the segmentation time varies little between datasets.

Figure 5. Visualization of the differences between the results of the RMLS and the manual segmentation. (a) RMLS result. (b) Manual segmentation result. (c) Visualization of differences.

Y. Song et al. / Segmentation of 3D Vasculatures for Interventional Radiology Simulation

605

Table 1. Summary of the symmetric mean errors. ITK-SNAP RMLS HS

Min. SME (mm) 1.22 1.46 0.904

Max. SME (mm) 5.24 10.74 6.44

Avg. SME (mm) 2.36 4.93 2.37

RMS SME (mm) 7.45 14.96 5.99

3. Conclusion This study presents and evaluates two segmentation methods which aim to overcome the shortcomings of traditional level set and interactive approaches. It also analyses the subjective influence of human observers on the segmentation results. The proposed segmentation techniques have been successfully applied to create a database of various anatomies with different pathologies, which is used in computer-based simulation for interventional radiology training.

4. Acknowledgements This work is partly funded by the UK Engineering and Physical Sciences Research Council (EP/E002749).

References [1]

J.S. Suri, K. Liu, L. Reden, and S. Laxminarayan, “A Review on MR Vascular Image Processing: Skeleton Versus Nonskeleton Approaches II”. IEEE Trans. on Information Technology in Biomedicine, vol. 6, No. 4, December, 2002. [2] C. Kirbas and F. Quek, “A Review of Vessel Extraction Techniques and Algorithms”. ACM Computing Surveys, vol. 36, no. 2, pp. 81-121, June, 2004. [3] P. Campadelli and E. Casiraghi, “Liver Segmentation from CT Scans: A Survey”. Lecture Notes in Computer Science, Springer Berlin, vol. 4578, pp. 520-528, August, 2007. [4] L. Gao,, D.G. Heath, E.K. Fishman, “Abdominal Image Segmentation Using Three-dimensional Deformable Models”. Investigative Radiology, 33(6), pp. 348-355, 1998. [5] T. Heimann, H.P. Meinzer and I. Wolf, “A Statistical Deformable Model for the Segmentation of Liver CT Volumes”. MICCIA 2007 workshop proceedings, pp. 161-166, 2007. [6] H. Park, P.H. Bland and C.R. Meyer, “Construction of an Abdominal Probabilistic Atlas and Its Application in Segmentation”. IEEE Transactions on Medical Imaging, vol. 22, no. 4, pp. 483-492, April, 2003. [7] J.A. Sethian, “Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational”. Geometry, Fluid Mechanics, Computer Vision and Materials Science, Cambridge Univ. Press, 1999. [8] V. Caselles, R. Kimmel and G. Sapiro, “Geodesic Active Contours”. International Journal of Computer Vision, 22(1) 6179, 1997. [9] V. Luboz, X. Wu, K. Krissian, C.F. Westin, R. Kikinis, S. Cotin, and S. Dawson, “A segmentation and reconstruction technique for 3D vascular structures”. Proceedings of the MICCAI Conference, pp 43-50, Palm Spring, CA, October 2005. [10] P.A. Yushkevich, J. Piven, H.C. Hazlett, R.G. Smith, S. Ho, J.C. Gee and G. Gerig, “User-guided 3D active contour segmentation of anatomical structures: significantly improved efficiency and reliability”. NeuroImage, 31 (3), 1116-1128, 2006. [11] Y. Song, A.J. Bulpitt and K.W. Brodlie, “Efficient Semi-automatic Segmentation for Creating Patient Specific Models for Virtual Environments”. MICCAI 2008 workshop on Computer Vision for Intravascular Imaging (CVII), pp.22-34, 2008. [12] L.D. Griffin, A.C.F. Colchester, S.A. Roell, “Hierarchical Segmentation Satisfying Constraints”. BMVC94, 135-144, 1994.

606

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-606

EEG-based “Serious” Games and Monitoring Tools for Pain Management Olga SOURINA1, Qiang WANG, and Minh Khoa NGUYEN Nanyang Technological University, Singapore

Abstract. EEG-based “serious games” for medical applications attracted recently more attention from the research community and industry as wireless EEG reading devices became easily available on the market. EEG-based technology has been applied in anesthesiology, psychology, etc. In this paper, we proposed and developed EEG-based “serious” games and doctor’s monitoring tools that could be used for pain management. As EEG signal is considered to have a fractal nature, we proposed and develop a novel spatio-temporal fractal based algorithm for brain state quantification. The algorithm is implemented with blobby visualization tools for patient monitoring and in EEG-based “serious” games. Such games could be used by patient even at home convenience for pain management as an alternative to traditional drug treatment. Keywords. EEG, serious games, pain management, neurofeedback, fractal dimension

Introduction Electroencephalogram (EEG) is a non-invasive technique recording the electrical potential over the scalp with multiple electrodes. Neurofeedback (NF) is a technique that allows the user voluntary change his/her brain state based on the visual or audio feedback corresponding to the recognized brain state of the user from his/her EEG signals. Some research reveals that the EEG and Event Related Potential (ERP) distortion can reflect psychological disorders such as Attention Deficit Hyperactivity Disorder (ADHD) [1-2], Autistic Spectrum Disorders (ASD) [3-4], Substance Use Disorders (SUD) including alcoholic and drug abuse [5-6], etc. Similar to other parts of our body, the brain function can be trained as well. Neurofeedback is an alternative choice as a treatment to these disorders besides a medical treatment. Many neurofeedback games were assessed and it was proved that they have a healing effect on psychological disorders, e.g. ADHD [7]. Current treatments for pain syndrome employ multidisciplinary approach such as chemical (drugs), physical (therapeutic exercise, acupuncture), psychological approach (relaxation with music, biofeedback, and hypnosis) or a combination of the abovementioned approaches. Recently, virtual games with the distraction effect were proposed for pain management [8-10]. 3D games and Virtual Reality (VR) games were used during the burn dressing of children in pain [8], during treatments of wounds [910], etc. It was also reported successful applications of EEG-based games for Central 1

Corresponding Author: Olga Sourina, Nanyang Technological University, 50 Nanyang Avenue, Singapore 639798; E-mail: [email protected].

O. Sourina et al. / EEG-Based “Serious” Games and Monitoring Tools for Pain Management

607

Pain Syndrome (CPS) pain management [11], and migraine management [12]. Although a great improvement in pain relief is reported by the therapy with EEG-based games, fundamental research and clinical experiments are needed to validate the results. Novel algorithms of pain level quantification and adaptive algorithms for pain therapy should be proposed. From our preliminary study, there are four main directions in game therapy for the pain management: 1) through distraction, 2) alteration of mood (emotion induction), 3) the use of relaxation, and 4) improved sense of control. In this paper, we describe EEG based “serious” games we developed that could be used for pain management through distraction and improved sense of control by relaxation versus concentration.

1. Methods & Materials The EEG-based game design includes two main parts: signal processing algorithms and a 2D/3D or VR game part. Raw EEG signals recorded from the user’s brain are filtered and analyzed by signal processing methods, and the resulting values are interpreted in the game as an additional game control just by the user’s “brain power”. A therapeutic effect of such games consists of combination of the distraction effect of the game and effect from the learning by the user/patient how to control the game by changing voluntary his/her brain state. The mainly used signal processing algorithm in the neurofeedback implementation is a power spectrum analysis in different frequency bands, i.e. band (<4Hz), band (4-7Hz), band (8-12Hz), band (12-30 Hz), and band (>30 Hz) of the EEG signals. Each frequency band is related to different brain functions [13]. ERP analysis, e.g. SCP and P300 component analysis, is another useful tool in neurofeedback that widely used in ADHD treatment [7], and drug abuse rehabilitation [6]. The linear features such as a power spectral density or amplitude extracted from EEG cannot represent the brain activities perfectly well due to the nonlinearity of the EEG signal. Thus, nonlinear methods, e.g. entropy analysis and fractal dimension analysis, are used for EEG processing in many medical applications and could be applied to neurofeedback systems to model brain activities. Fractal dimension (FD) is a measurement of complexity and irregularity of a signal. In our work [14-15], we studied a generalized Renyi approach based on Renyi entropy and applied it to quantification of EEG signals. Entropy is a measure of disorder in physical systems. Thus, with calculation of the fractal dimension we could estimate the brain signal complexity. There are different fractal dimension algorithms. In our work [1617], for calculation of fractal dimension values, we implemented and analyzed two well-known Box-counting [18] and Higuchi [19] algorithms. Both of them we evaluated using Brownian and Weierstrass functions where theoretical true values are known. The Higuchi algorithm gave a better accuracy as its FD values were closer to the theoretical ones [17]. Thus, we used Higuchi algorithm to quantify level of concentration as opposite to relaxation. In this paper, we proposed to apply the fractal-based algorithm to quantify levels of concentration for “serious” game implementation that could be used for pain management through the user distraction and the improved sense of the user control. The concentration/relaxation level values could be interpreted in the games with any visual/audio effects or a change of characters behavior. We also proposed and implemented real-time doctors monitoring tools named “VisBrain” system that could be used for spatio-temporal visualization of the EEG signals. Amplitude and/or

608

O. Sourina et al. / EEG-Based “Serious” Games and Monitoring Tools for Pain Management

calculated FD values are mapped to a 3D head using the time-dependent blobby model [20]. “VisBrain” could be applied for a study of spatio-temporal patterns of patient cases with pain syndromes of different origins. With the proposed system, EEG data could be collected during the doctor patient sessions, and a questioner on the pain assessment should be given to each patient. Based on the EEG monitoring and the questioner, the pain level could be quantified, and spatio-temporal maps of the pain location in the brain for different cases including CPS cases could be created.

2. Results “Brain Chi” and “Dancing Robot” are two simple single-player games that we developed and applied for the pain management. In the “Brain Chi” game, the quantified level of the user concentration is associated with the radius of the “growing/shrinking” ball that allows a “little boy” character to fight enemies by “growing” the ball. In the “Dancing Robot” game, the relaxation/concentration level is associated with a “robot” character behavior. When the concentration level increases, the “robot” character starts to move faster. Examples of the game settings are shown in Figure 1 (a) and (b). In order to play the developed EEG-based games, the user needs an EEG reading device and computer. In our study, EEG data is collected by the Emotiv device or Pet 2. Currently, only O1 electrode in the occipital lobe following the American Electroencephalographic Society Standard is active in our games. EEG signal is transmitted to the computer with Bluetooth. Our final target is to implement series of “Brain Chi” games that allow the user just by playing the games to “reprogram” the corresponding parts of the brain that could be monitored as changing fractal dimension values. Such brain exercises could lead to the pain relief as it was validated in our preliminary study. Different games could be proposed for groups of patients. To monitor the patient state, we use real-time system “VisBrain”. The raw data is read by Emotive device and then filtered by 2-42 Hz bandpass filter. All electrodes are active in the system. A doctor could choose the appropriate mode of 3D signal mapping. It could be a 3D blobby, “pins”, or color visualization. Currently, we visualize the changing signal amplitude and FD values. The 3D blobby mapping allows the doctor to assess the spatio-temporal pattern of the patient brain states. In Figure 2 (a), (b), a spatio-temporal visualization of EEG signals is shown with the blobby and color mapping correspondingly.

Figure 1. Examples of neurofeedback concentration/relaxation games: a) “Brain Chi”; b) “Dancing Robot”

O. Sourina et al. / EEG-Based “Serious” Games and Monitoring Tools for Pain Management

609

Figure 2. Online EEG monitoring tools: a) with blobby visualization; b) with color mapping.

3. Discussion The EEG-based “serious” games for pain management could be played by a patient not only in the doctor’s office but also at home convenience as well. Thus, the doctor during the session using the “VisBrain” monitoring tools could advice to the patient the schedule of the game therapy. More research should be done in future on the use of audio and visual stimuli in the games to improve the therapeutic effect of the games. In work [21], music stimuli for emotion induction were studied, and in [22], we proposed and implemented a real-time fractal-based emotion recognition algorithm where we mapped fractal dimension values to a 2D Arousal-Valence emotion model. To evoke emotions in the game, different stimuli could be used: visual, auditory, and/or the combined ones. Currently, we can recognize online the following emotions: satisfied, pleasant, happy, frustrated, sad, fear, and neutral with 89.13% and 90% for arousal and valence levels respectively. Only 3 channels are necessary. In future, we are going to use the results of the study in pain management game development by adding emotional dimension into the game. The following innovations were proposed and are expected in the future. Technological: We proposed a novel EEG-based technology that includes spatiotemporal fractal-based algorithms of brain state recognition, and the doctor’s monitoring tools as software implementation. We have to further improve EEG-based pain management therapy tools. Scientific: We proposed a new spatio-temporal fractalbased model of pain level quantification and EEG-based pain management approach leading to the implementation of doctor’s and patient tools for the EEG-based monitoring and pain treatment. Economic: We proposed a cost-effective EEG-based technology for the pain treatment. The cost of the developed software is included in the current project cost and can be minimized just to the cost of CD copy. The final cost of the treatment could include a doctor session fee and one EEG headset with an access to PC computer that currently has a tendency to reduce. Then, the patient could have any amount of sessions prescribed by the doctor free of charge. Social: By research among institutionalized elderly, 71% to 83% of them report at least one pain problem, and 60% of adult men and women experience some pain [23]. Central Pain Syndrome not only causes physical discomfort, but also interferes with social relationships, family life and self-esteem, and there is a high correlation between the chronic pain and depression. Considering all above the proposed pain treatment tools could improve quality of life of 60% of adult and 70-80% of elderly people giving the patients a good alternative to more expensive traditional drug treatment.

610

O. Sourina et al. / EEG-Based “Serious” Games and Monitoring Tools for Pain Management

Acknowledgment This project is supported by grant NRF2008IDM-IDM004-020 “Emotion-based personalized digital media experience in Co-Spaces” of National Research Fund of Singapore.

References [1] J. F. Lubar, et al., Evaluation of the effectiveness of EEG neurofeedback training for ADHD in a clinical setting as measured by changes in T.O.V.A. scores, behavioral ratings, and WISC-R performance, Biofeedback and Self-Regulation 20 (1995), 83-99. [2] T. Fuchs, et al., Neurofeedback treatment for attention-deficit/hyperactivity disorder in children: A comparison with methylphenidate, Applied Psychophysiology Biofeedback 28 (2003), 1-12. [3] R. Coben, et al., Neurofeedback for autistic spectrum disorder: A review of the literature, Applied Psychophysiology Biofeedback 35 (2010), 83-105. [4] M. E. J. Kouijzer, et al., Neurofeedback treatment in autism. Preliminary findings in behavioral, cognitive, and neurophysiological functioning, Research in Autism Spectrum Disorders 4 (2010), 386399. [5] E. Saxby and E. G. Peniston, Alpha-theta brainwave neurofeedback training: An effective treatment for male and female alcoholics with depressive symptoms, Journal of Clinical Psychology 51 (1995), 685693. [6] T. M. Sokhadze, et al., EEG biofeedback as a treatment for substance use disorders: Review, rating of efficacy, and recommendations for further research, Applied Psychophysiology Biofeedback 33 (2008), 1-28. [7] H. Gevensleben, et al., Distinct EEG effects related to neurofeedback training in children with ADHD: A randomized controlled trial, International Journal of Psychophysiology 74 (2009), 149-157. [8] http://videos.howstuffworks.com/sciencentral/2888-virtual-pain-relief-video.htm. [9] http://www.myfoxaustin.com/dpp/news/local/111909-Video-Game-Therapy-Helping-Soldiers. [10] http://www.impactlab.com/2006/03/19/introducing-video-game-therapy. [11] http://www.youtube.com/watch?v=6qocxopS5fc&feature=player_embedded. [12] http://www.youtube.com/watch?v=SKY-TlAt4co. [13] J. N. Demos, Getting Started with Neurofeedback, WW Norton & Company, New York, 2005. [14] V. Kulish, A. Sourin, O. Sourina, Human electroencephalograms seen as fractal time series: mathematical analysis and visualization, Computers in Biology and Medicine, Elsevier-Pergamon 36 (2005), 291-302. [15] V. Kulish, A. Sourin, O. Sourina, Analysis and visualization of human electroencephalograms seen as fractal time series, Journal of Mechanics in Medicine & Biology 26 (2006), 175-188. [16] Q. Wang, O. Sourina, M. K. Nguyen, A Fractal Dimension Based Algorithm for Neurofeedback Games, In Proc. of CGI 2010, SP(26), Singapore, 8-11 Jun 2010. [17] Q. Wang, O. Sourina, M. K. Nguyen, EEG-based “Serious” Games Design for Medical Applications, In Proc of 2010 Int. Conf. on Cyberworlds, 270-276, Singapore, 20-22 Oct 2010. [18] A., W. Block, V. Bloh, and H. J. Schellnhuber, Efficient box-counting determination of generalized fractal dimensions, Physical Review A 42(1990), 1869-1874. [19] T. Higuchi, Approach to an irregular time series on the basis of the fractal theory, Physica D: Nonlinear Phenomena 31(1988), 277-283. [20] O. Sourina, A Sourin, V. Kulish, EEG Data Driven Animation and its Application, In Proc. of International Conference Mirage 2009, 380–388, 4-5 May 2009. [21] O. Sourina, V. Kulish, A Sourin, Novel Tools for Quantification of Brain Responses to Music Stimuli, In Proc. of 13thInternational Conference on Biomedical Engineering ICBME, 411-414, Singapore, 3–6 Dec 2008. [22] Y. Liu, O. Sourina, M. K. Nguyen, EEG-based Human Emotion Recognition and Visualization, In Proc of 2010 Int. Conf. on Cyberworlds, 262-269, Singapore, 20-22 Oct 2010. [23] L. Galieze, Chronic Pain in Elderly People, Pain 70 (1997), 3-14.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-611

611

A New Part Task Trainer for Teaching and Learning Confirmation of Endotracheal Intubation Dr Cyle SPRICK a,1 Prof Harry OWEN a Dr Cindy HEIN b Dr Brigid BROWN c a Flinders University b SA Ambulance c Southern Adelaide Health Service Abstract. Endotracheal intubation is a skill employed by a diverse range of healthcare professionals in a wide variety of circumstances. Failure to put this tube in the right place (in the trachea) can result in serious injury and death. There are a wide variety of methods for verification of proper placement. Some are more widely accepted than others. A universal guideline should be adopted to allow consistent and safe practice in all situations by all who do this procedure. Training for endotracheal intubation must also include training in the verification methods. We have developed a new airway part-task trainer that allows the use of all of the methods of tube placement verification.. Keywords. Simulation, endotracheal intubation, ETI, SimTools, confirmation

Introduction Healthcare practitioners from acute care disciplines need to be proficient in a range of airway management techniques. At our hospital, intensive care and emergency medicine trainees, paramedics, Royal Flying Doctor Service nursing staff, rural GPs and medical students have clinical placements in the operating theatre suite with the objective of acquiring or refreshing airway intervention skills. Most want to be able to intubate but the educational attachments are typically short and individuals have had varying amounts of preparation. The consequences of unrecognised oesophageal intubation are catastrophic but this event can be avoided through verifying correct placement of an endotracheal tube. Staff and patients move between clinical areas (e.g. an ICU nurse may transfer to anaesthesia or the RFDS) and patients may be handed over (e.g. from paramedics to the ED) so a single, universal, evidence-based process of checking the position of an ET tube would be valuable for patient safety. This would also facilitate cross-checking of the process by other team members which is an important consideration.

1

Corresponding Author: [email protected]

612

C. Sprick et al. / A New Part Task Trainer for Teaching and Learning Conﬁrmation

1. Methods A search was undertaken for methods of confirming (verifying) endotracheal intubation for anaesthesia and in other areas of patient care. We determined there were two distinct populations that required endotracheal intubation: patients with an effective circulation and those without. The first group includes patients being intubated for ventilation during anaesthesia or needing respiratory support as a part of intensive care. These patients had an effective circulation, some preparation was possible (e.g. preoxygenation) and the intubation usually took place in a controlled environment. The second group was made up of patients that had severe shock or have had a cardiac arrest so that little or no CO2 is being delivered to the lungs. These patients needed oxygenated cerebral blood flow now and anything that delayed that would worsen outcome. In addition they could be in a noisy or dirty environment with inappropriate lighting. Methods for confirming tracheal intubation needed to suit both of these groups. Part-task trainers (desktop simulators) are used for initial training in endotracheal intubation. Whole-body manikins (patient simulators) of varying sophistication are used mostly for practising difficult intubation scenarios. The investigators searched for and then evaluated as many of these models as they could and reviewed their features. A new part task trainer was constructed that incorporates ALL the recommended methods of confirmation of proper tube placement. This allows trainees to practice the full placement confirmation procedure each time they undertake this task.

2. Results Many ways of confirming endotracheal intubation have been described and these are listed in table 1. Currently, auscultation is the only clinical method that identifies the tip of the ET tube is in the trachea or a bronchus. No single method of confirming endotracheal intubation can be relied on, not even the reputed ‘gold standard’ of waveform capnography, so several methods must be used in combination. A universal process needs to be applicable to all intubation settings, be quick to perform, not require major additional investment and have high face validity. This eliminated the majority of methods listed in table 1 and left only those recommended in resuscitation guidelines from ILCOR. Standardised teaching and practise of confirmation of intubation through a consistent sequence of observations is a desirable objective. It is stressed that the tube must be seen to pass between the vocal cords and these methods are then used for confirming endotracheal intubation. Whilst a single process of checking would be ideal, alternatives for patients that have or don’t have a pulse (circulation) reduced the number of steps for most patients (See table 2). Healthcare professionals are already familiar with resuscitation guidelines on the management of patients with and without a pulse and this approach to confirmation of intubation is an extension of that. A summary of findings when we tried to use these methods on the various airway trainers is given in Table 3. Unfortunately, none of the airway trainers incorporated all of the recommended methods for confirming tube placement. This results in learners not having the opportunity to practice the whole procedure including confirmation of placement. When learners rehearse only part of a procedure, they are much more likely to perform only that part of the procedure in actual practice.

C. Sprick et al. / A New Part Task Trainer for Teaching and Learning Conﬁrmation

613

Table 1: Methods of confirmation of tracheal intubation

Method

Comments

Bilateral chest rise

Not always possible to assess

Auscultation of axillae and epigastrium

Sounds can be transmitted. Can differentiate tracheal from bronchial intubation

Breath by breath (continuous) CO2

Very useful if patient has an effective circulation.

Oesophageal detector device

Very useful and quick but rarely taught. Less reliable after breaths have been delivered

Condensation in tube (misting, fogging)

Not reliable (ref) but widely taught

“Clicking” of bougie on tracheal rings

Useful, requires a bougie with a Coude tip

Visualisation of tracheal rings or carina

Very specific but uses expensive equipment

Transcricothyroid membrane ultrasound

Appears useful but requires additional equipment and training

Transillumination

Inexpensive but needs shade

Computerised breath sound analysis

Requires electronic stethoscope and proprietary software

Transthoracic impedence

Little evidence of value yet

Reflected sound waves

Under development, could be useful

Magnet and sensor or RFID device

No commercial product available

Repeat direct laryngoscopy

Error prone and likely delay CPR during resuscitation

Pulse oximetry

Should not be used for verification of intubation

Chest x-ray

Not to be used for verification but may show need to reposition Table 2: Suggested guidelines for verifying endotracheal intubation

ET tube seen entering the trachea through the vocal cords, then… Patient has pulse (effective circulation)

No pulse (Cardiac arrest/severe shock)

Oesophageal detector device *

Oesophageal detector device

Bilateral chest rise**

Bilateral chest rise**

Auscultation of R and L axillae and epigastrium

Auscultation of R and L axillae and epigastrium

CO2 detected with every breath

CO2 detector attached***

Pulse oximeter checked

Pulse oximeter checked

Note tube length at teeth and fix

Note tube length at teeth and fix

Record above observations and tube length at teeth. If patient is moved, head or neck position is changed, expired CO2 changes or SpO2 falls unexpectedly, tube marking at teeth is different, a cuff leak develops or lung compliance changes the position of the ETT must be re-evaluated urgently.

We modified existing airway trainers to incorporate all of the recommended methods of tube placement including: Oesophageal Detector Device, unilateral and bilateral chest rise, auscultation of the epigastrum and axillae, and CO2 detection. This was done using a combination of mechanical modifications to commercial airway trainers and addition of the SimTools CO2 capnometer emulator (which is modeled after the EasyCapII from Nellcor Puritan Bennett) and the SimTools stethoscope. (Figure 1)

614

C. Sprick et al. / A New Part Task Trainer for Teaching and Learning Conﬁrmation Table 3: Functionality of airway trainers for verifying endotracheal intubation

Model TRM LAT LFC LDA LTA SAK KOK CLA AST LAT AAT STT

ODD – () – – – – – – –

Bilat CR () – – – () – – – – –

Ausc. axillae – – – – – – – –

Ausc. epigastrium – – – () – () – – – –

CO2 – – – – – – – – – – – –

The key to airway models is available from the authors. = feature is present, () feature present but not well emulated and – feature not present.

The SimTools stethoscope was described previously at MMVR as part of a suite of tools to provide simulated patient information for manikins or actors.[1] In this case, the stethoscope provides presence or absence of breath sounds as appropriate in the chest, axillae and epigastrum. The SimTools capnometer emulator (shown in Figure 2) uses a pressure sender placed in the airway of the model (Figure 3) to detect ventilations.

Figure 1: Customised Airway Trainer

Figure 2: Prototype display and breath detector

The modular Resus Anne system was chosen because it provides a full torso and is fairly easily upgraded. In particular, the oesophagus is sufficiently long and soft enough to allow the ODD to function as designed. A low resistance one-way valve was added to the end of the oesophagus to allow exhaust of ventilations yet give a positive indication with the ODD. (Figure 3) The exhaust of the one-way valve could be plumbed to the abdominal bladder to simulate stomach distention. An alternate method of deflation is necessary. Figure 4 shows the one-way valve in position next to the breath detector attached to the lung. A future modification would be to split the lung into left and right. Placement of the breath detector in the left lung helps to distinguish right mainstem bronchus intubation. Oesophageal intubation will not trigger the breath detector. When the lungs are ventilated, a wireless signal is sent to the display to trigger a colour change display to the clinician. Another wireless message signals the stethoscope to play the appropriate

C. Sprick et al. / A New Part Task Trainer for Teaching and Learning Conﬁrmation

615

sound as selected by the facilitator based on chestpiece placement and clinical situation. (Figures 3 & 4)

Figure 3: Chest view of airway and oesophagus

Figure 5: No CO2

Figure 4: Breath detector attached to trainer

Figure 6: CO2 detected

The colour of the central portion of the display smoothly changes from purple to tan with each breath. (Figures 5 & 6) The prototype uses a PDA as a display. Future models are planned which use a dedicated in-line display which sits between the ET tube and the ventilation device and looks much more like the real device. 3. Conclusion Resuscitation and anaesthetic guidelines have recommended several approaches to verification of endotracheal tube placement. We have surveyed the available methods and devices and summarised these guidelines in Table 2. Current part-task trainers do not allow the user to practice the full guideline of tube placement verification. Skipping steps during training can lead to omission of these steps during clinical practice. We have created a new part-task airway trainer that allows the entire procedure to be practiced. We are currently investigating various sequences of tube placement verification to determine one that will minimise the time to detection of an incorrect placement. 4. References [1]

Sprick, C.; Reynolds, K. J. & Owen, H. (2008), SimTools: a new paradigm in high fidelity simulation., Stud Helath Technol Inform 132(2008), 481-483 Published by IOS Press.

616

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-616

Mobile Three Dimensional Gaze Tracking Josef STOLL a,1 , Stefan KOHLBECHER b , Svenja MARX a , Erich SCHNEIDER b and Wolfgang EINHÄUSER a a Neurophysics, Philipps-University Marburg, Germany b Institute for Clinical Neurosciences, University Hospital, Munich, Germany Abstract. Mobile eyetracking is a recent method enabling research on attention during real-life behavior. With the EyeSeeCam, we have recently presented a mobile eye-tracking device, whose camera-motion device (gazecam) records movies orientated in user’s direction of gaze. Here we show that the EyeSeeCam can extract a reliable vergence signal, to measure the ﬁxation distance. We extend the system to determine not only the direction of gaze for short distances more precisely, but also the ﬁxation point in 3 dimensions (3D). Such information is vital, if gaze-tracking shall be combined with tasks requiring 3D information in the peri-personal space, such as grasping. Hence our method substantially extends the application range for mobile gaze-tracking devices and makes a decisive step towards their routine application in standardized clinical settings. Keywords. Mobile eyetracking, 3D gaze calibration, vergence eye movements

Introduction Gaze allocation in natural scenes has been a subject of research for nearly a century [1,2]. Possible applications reach from advertisement [1,3,4], over basic research to clinical applications [5,6,7]. Most experimental studies, however, measure eye movements in constrained laboratory settings. While such data have some predictive quality for gaze allocation in real-world environments, plenty of qualitative features remain unexplorable for principled reasons [8]. Recently, we have introduced a wearable eye-tracking device (EyeSeeCam) that allows recording gaze-centered movies while an observer pursues natural tasks in a natural environment [9]. Unlike stimuli presented on a screen, however, the real world is inherently 3D. Despite of research in virtual reality (VR), where eye trackers have been coupled with VR goggles [10,11] and in remote eye-tracking applications [12], most of today’s commercial eye tracking systems ignore this fact and restrict their calibration to one plane or use a recording setup that avoids parallax errors2 . Here we propose a solution that in addition yields distance information. To achieve robust 3D gaze-tracking, each eye needs to be represented in its own coordinate system under the constraint that the gaze directions of both eyes converge onto the ﬁxation point. Fulﬁlling this condition allows the measurement of disjunctive eye movements, yielding a vergence signal for depth measurement. Here we present an extension 1 Corresponding

Author: Josef Stoll, AG Neurophysik, Philipps-Universität Marburg, Karl-von-Frisch-Str. 8a, 35032 Marburg, Germany; E-mail: [email protected]. 2 e.g., ISCAN, Woburn MA, USA, http://www.iscaninc.com

J. Stoll et al. / Mobile Three Dimensional Gaze Tracking

617

of the EyeSeeCam software that allows calibration in depth. Besides the identiﬁcation of ﬁxated targets in space, this allows the system to compensate for inevitable errors of parallax arising from the distance between the gaze-controlled camera-motion device (gazecam) and eyes. We quantify advantages in calibration accuracy and provide a proof of principle that eye-tracking can be used to tag objects in 3D space.

1. Methods EyeSeeCam Hardware The basic design of the EyeSeeCam has been described previously [13]. In brief: The EyeSeeCam consists of a binocular video-oculography (VOG) device and a head-mounted camera-motion device (gazecam), that is continuously oriented to the user’s point of regard by the eye movement signals . The gazecam captures images nearly identical to the user’s retina-centered visual stimulus, thus operating as an artiﬁcial eye. The whole apparatus is lightweight, mobile, battery-driven and controlled and powered by one laptop computer (Apple, MacBook). Altogether four digital cameras (Point Grey, Fireﬂy MV) are integrated in the EyeSeeCam (Figure 1,B). The gazecam reaches the dynamic properties of the human ocular motor system - velocities above 500 deg/s and accelerations of up to 5000 deg/s2 - with a latency of 10 ms. The workspace lies in the range of ± 30 deg for horizontal and vertical movements [9]. For minimal user restriction and high orientation accuracy, a compact, light-weight, noiseless, and precise system is realized by a parallel kinematics setup with small piezo linear motors (Physik Instrumente, Germany). The camera platform, which is connected by a gimbal joint to the supporting head mount, gets rotated via two universal joints and push rods through two parallel displaced sleds, driven by the piezo actuators (modelled in ﬁgure 1C). Model-Free Gazecam Calibration In routine usage, the direction of the gazecam is aligned to the observer’s direction of view by the following procedure. The gazecam is moved towards 25 pre-deﬁned locations on a 5x5 grid, whose central point is approximately aligned with the user’s gaze straight ahead. The user is asked to ﬁxate the location a laser aligned with the optical axis of the gazecam pointer indicates. The mapping between known camera commands and VOG signal is interpolated and extrapolated to

Figure 1. Setup A) EyeSeeCam device; VOG cameras (600Hz, low-res) are visible to behind the hot mirrors; gazecam(752x480 pixels, 60Hz) on top; wide-angle head ﬁxed camera (below gaze cam) is not used in the present experiment. B) Optical axes under two distance conditions; note the difference in eye vergence and parallax between gazecam and eyes. C) Simpliﬁed mechanical model simulated by CAD software; dependence between gazecam orientation and piezo sled position. By symmetry, the perpendicular position of the platform joint plane generates its zero position; actual shift by 0.0009 is the consequence from the push rod’s inclination.

618

J. Stoll et al. / Mobile Three Dimensional Gaze Tracking

arrive at a mapping valid for most of the visual ﬁeld. Since the whole setup (including the laser pointer) moves with the head, this calibration method is insensitive to head movements and thus allows an accurate alignment of gaze and gazecam. Limitations of Model-Free Calibration In the model-free calibration, the mapping between VOG and camera-motion commands is computed directly, but no information is obtained on the angular orientation of the eye in its orbit. The validity of the calibration is restricted to the distance the laser pointer is projected to. This is tolerable if the distance is always large (i.e., virtually inﬁnite) or all operations happen near this plane. Real-world tasks, however, often require switching between peri-personal and far space, such that a depth-corrected calibration is required. Similarly, parallax errors due to the inevitable misalignment of eye and gazecam need to be corrected in near space. Since such depth corrections require vergence information, thus orientation of the eye, the modelfree calibration approach is insufﬁcient for parallax correction and 3D calibration. Eye-in-Head Calibration In routine usage, the gazecam calibration is complemented by an independent calibration for eye-in-head orientation [14]. This has to be done for each individual and session, as the device’s position relative to the observer’s head may vary when the device is taken off and back on. For this calibration, an additional headﬁxed laser is mounted between the eyes. The laser is equipped with a diffraction grating that generates a face-centered grid of dots on, e.g., a nearby wall (Figure 1B). The 4 ﬁrst-order diffraction maxima have an angular distance of 8.5◦ to the central zerothorder maximum. This central laser projection deﬁnes the primary position, i.e., the origin of the gaze angle coordinates. By use of an eyeball model, these 5 gaze directions are integrated to map the VOG signal to the coordinates over the full calibration plane. So far, eye-in-head calibration and model-free calibration were performed independently. Although this allowed mutual veriﬁcation of the two methods, a depth-correct allocation of the gazecam was impossible. Thus we here present a novel method that combines both strategies to arrive at a calibration and distance estimation in 3D space. Offset Correction To align the gazecam with the calibrated eye position, a second laser pointer is mounted in parallel to the optical axis of the gazecam. The offset of the gazecam is then adjusted by the experimenter through the graphical user interface such that this pointer matches the central (0th order) maximum of the projected calibration grid. This represents the uniquely adjusted origin of the gazecam coordinates plus a parallax correction matching the calibration distance. This procedure is independent from eye-in-head calibration. Only after offset correction is performed, the gazecam is set to follow gaze (i.e., is in tracking mode). During this usage of the EyeSeeCam the pointer can be used to verify the calibration against drift and to tag items of interest. 3D Calibration The 3D-information about the user’s ﬁxation point is observable, if both eyes are calibrated in the same reference (coordinate) system. At ﬁxation in inﬁnite distance, the zero direction of each eye is parallel to the calibration laser grid’s central ray. For ﬁxation at ﬁnite distances, a vergence eye movement adds to the zero direction. Since calibration must be performed at ﬁnite distances, we correct for the vergence angle occurring at each calibration point. By adding this angle, we adjust the coordinate systems such that both eyes point in parallel directions if their measured gaze coordinates are equal. For this correction, the relative positions between the eyes and the calibration points are needed and thus the projection distance and the relative position between the eyes and the source of the laser grid are required. While the distance between pupils has to be adjusted individually, inter-individual differences in the distance to the source of

J. Stoll et al. / Mobile Three Dimensional Gaze Tracking

619

the diffraction pattern, i.e., the laser, are negligible. Given the distance of the eyes to the laser and the application of the offset correction, the eye-in-head calibration yields a vergence angle as the difference of the spherical angles of both eyes according to the following calculation. Equal polar angles imply ﬁxation at inﬁnite distance. Any vergence angle greater than zero implies that rays originating from both eyes cross. The vergence angle is the inclination in the ﬁxation point that is equal to the difference in both eyes polar angles. Due to the symmetry of the problem, the azimuth angle does not inﬂuence this computation. Using the polar angles and the interpupillary distance, we construct two linear equations whose solution is the ﬁxation point in 3D space. Gazecam Vergence Correction Now we change the axis of symmetry and ignore the polar angle for the parallax correction. The gaze-direction in azimuth, the ﬁxation distance b and the relative position of the gaze-camera’s gimbal joint allow spanning a triangle, whose unknown angle γ is the difference in azimuth between eyes and gazecam rays and equals the parallax correction - the gimbal joint lies on the optical axis of the gazecam. The distance between the center of the eyeballs and the gimbal joint is used in a. γ is the difference of the averaged gaze azimuth and the angle included by gaze zero direction and the direction from eyeballs center to the gimbal joint (Figure 1B). This problem is solved in plain trigonometry by formulas valid for oblique triangles, where γ a−b two sides a, b and their included angle γ are available. Thus, tan( α−β 2 ) = a+b cot 2 and α + β + γ = π yield the parallax correction: γ a−b α = π−γ 2 arctan( a+b cot 2 ). Parameterized Gazecam Positioning Distance variations of a user’s ﬁxation imply an adaptation of the angle, which corrects for the parallax error of gazecam orientation. This requires the positioning of the gazecam to be implemented as a function of spherical angles that stays constant given the present mechanical conditions. It needs only the geometry of the systems to be known and thus is a one time measurement. The gimbal joint as the center of rotation coincides with the optical axis of the gazecam. This facilitates the mapping from a gazecam direction to the linear positions of the two piezo actuator sleds. The function is built on the holonomic constraints of the three-point coupling behind the camera platform, whose two universal joints are displaced by the piezo actuators via push rods. The transformation from angle to linear sled position replaces the previously employed point-by-point matching, but not the origin-offset adjustment, which depends on the individual ﬁt of the EyeSeeCam on the user’s head. This means, the novel gazecam control still would need the angle relative to its zero position, in which the gazecam is oriented parallel to the primary eye position. The issue of offset-independent positioning is solved by a separation of the orientation into a vertical rotation followed by a horizontal rotation. First, the tilt given by the azimuth is virtually executed by a symmetric displacement of both piezo actuator bars, which is related by a simple sine mapping. Then the new positions of the pivots mounted on the camera platform are computed by projecting them on the plane of polar rotation, the pan. The resulting triangle gets rotated by the polar angle and the ﬁnal positions of the universal joints are reached. Their projection on a linear parallel to the linear motor direction provides the asymmetric sled displacement. Additionally taking into account their perpendicular projection enables correction pushrod inclination. This approach simpliﬁes the previous solution [15] and is easily adaptable to future systems with different geometries due to its parametric nature. The resulting mapping from direction angles to piezo actuator bar positions ensures an accurate

620

J. Stoll et al. / Mobile Three Dimensional Gaze Tracking

rotation of the gazecam. We veriﬁed the analysis with a 3D mechanical CAD program (SolidWorks, Dassault Systèmes, France).

2. Results To compare the novel calibration method to the previous one, we performed an experiment with predeﬁned ﬁxation targets. The projection distance during calibration was held constant at 2 m, alike in normal usage.The measurement process included ﬁxations at 1, 2 and 4 m with a pattern of ﬁxation targets, whose dot size and extension was scaled proportional to the distance. The dots were distributed on the corners and edge midpoints of squares, whose edge midpoints are oriented in the cardinal directions and have an angular distance of 2.4◦ - like the corners of the board game mill. To compare the accuracy of both calibration methods, the images from gazecam recordings were analyzed for deviations of the ﬁxated target from image center. To quantify the parallax error, the statistics from the absolute vertical pixel deviation were exploited and plotted in angular degrees. We opposed the parallax error resulting from the former method to the vergence corrected error remaining when 3D calibrated. The old method shows a clear decrease of the vertical error with increasing distances (Figure 2A bottom, red diamonds). This parallax error could be diminished substantially, as the level of the vertical error is around 1◦ for all measured ﬁxation distances (black circles). These results provide the proof of concept for a systematical vergence correction and mark a considerable increase for the accuracy with respect to the parallax error. To evaluate the overall performance of both calibration methods, the absolute value (direction independent) of the target-image-center deviation was analyzed. The mean over each eccentricity is plotted separately in Figure 2A, top. 3D calibration increases accuracy for the peri-personal range and also at large distances.

inner middle outer calibration square 1m

0.5

center

inner middle outer calibration square 2m

1.5 vertical error [deg]

vertical error [deg]

3.0

0

center

inner middle outer calibration square

0.5

B

total error [deg]

new old center

4m

3.0

0

center

inner middle outer calibration square 4m

1.5

center

inner middle outer calibration square

0.5

C 4 2 d (m)

0

2m

3.0

vertical error [deg]

total error [deg]

1m

total error [deg]

A 3.0

1

0.5 0.25 center

inner middle outer calibration square

0

5

10

t (s)

15

20

25

Figure 2. A) Accuracy Absolute error measured at validation points for different ﬁxation distances (top of each panel) for four eccentricities (x–axes). Top: Absolute error; bottom: vertical component of error. Red diamonds: old calibration method; black circles: novel approach. Mean and sem over users (n=2), trials (n=3) and ﬁxated points (n=8 for inner, middle and outer calibration square, 23 for center). Note: different scales for total and vertical error. B) User approaches a cube in space while ﬁxating it, online display of distance, note the vergence correction; http://www.staff.uni-marburg.de/~einhaeus/ESC_3D.html for video C) estimated log ﬁxation distance vs. walking time.

J. Stoll et al. / Mobile Three Dimensional Gaze Tracking

621

To investigate the 3D calibration in a truly 3D setting, we hang a small cube in space and ask one user to walk towards it. We observe an accurate measurement of ﬁxated locations in 3D space (Figure 2B). The distances represent a usable estimator with an uncertainty of around 10% in the peri-personal range. During the approach the time course of estimated distance gets rather smooth for distances below 1m (Figure 2C). Above 1.5m, when eyes are almost parallel, the distance estimate gets increasingly instable and eventually looses in accuracy. Nonetheless, the robust and accurate estimation of 3D ﬁxation distance in near space further validates our setup for use in tasks requiring operation in 3D space.

3. Discussion We present a novel calibration that uses angle calibrated binocular eye movements for 3D gaze tracking and extracts a reliable vergence signal that deﬁnes a 3D point for the gazecam’s target direction. In addition to ﬁxation determination in 3D, the new procedure increases accuracy for the near ﬁeld by analytic correction of parallax induced errors. The procedure simpliﬁes calibration and makes it more efﬁcient despite improvements in reliability and accuracy, which is of particular importance in clinical settings where large cohorts of patients have to be measured under strict time limitations. The uncertainty of ﬁxation estimation is inversely proportional to the distance: distance is reliably estimated in the peri-personal range and gets imprecise only above 2m. Although far distances are worse to measure based on the vergence signal, this error does not affect gaze accuracy, as the vergence correction is then converging to zero. The nearer the ﬁxation point, in turn, the lower is the uncertainty for the distance. Over all distances, performance is sufﬁcient to capture human behavior as device accuracy (in terms of standard error) is below usual human variations over ﬁxations, which are about 1◦ [16]. In addition, the human vergence system itself seems to be rather variable - the vergence angle varies between ﬁxations, although the target position is the same. This variation seems to be increased after large shifts in depth of ﬁxation, where vergence movements during slow disjunctive saccades arise. Humans are expected to have their ﬁxation point sharply tuned only after about 800ms [17]. Thus, eye-movement dynamics inﬂuence the evaluable ﬁxation distance considerably. By measuring vergence movements, we can apply our system to address vergence dynamics in a variety of realistic scenarios. Interacting with others in real-world space requires prediction of their intentions, which is often achieved by an estimate of the other’s gaze [18]; impairment of this function is fundamental to several clinical conditions in which social interactions or the perception of others’ intentions are impaired [19,20]. Using gaze direction as clinical tool [7] will thus be greatly fostered if combined with standardized clinical paradigms. As those often require action in peri-personal space or switching from peri-personal to far space, the 3D calibration presented here is inevitable for an eventual combination of gaze tracking with other tasks in standardized clinical settings. First tests in clinical populations with neuro pathological disorders (e.g., schizophrenia, Parkinson’s disease) have demonstrated the applicability of the EyeSeeCam in clinical settings. In addition, the EyeSeeCam has successfully been used as a training tool for surgeons by recording gaze-centered videos from a ﬁrst person perspective [21]. Such videos intuitively visualize the experienced surgeon’s action planning. By applying the new calibration method the camera may be

622

J. Stoll et al. / Mobile Three Dimensional Gaze Tracking

actively focused at the correct distance thereby further improve the ﬁrst-person feeling. Eventually, such application as teaching tool can extend well beyond surgery - for example towards dentists and anaesthesists [22] and even to other professions, like mechanics and engineers.

Acknowledgements We gratefully acknowledge support by the DFG under excellence cluster CoTeSys, BMBF (No. 01EO0901) (SK), grant EI 852/1 (WE), and research training unit 885 (JS).

References [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14]

[15] [16] [17] [18] [19] [20] [21] [22]

G.T. Buswell, How People Look at Pictures: A Study of The Psychology of Perception in Art, The University of Chicago Press, 1935. A.L. Yarbus, Eye movements and vision, Plenum Press, New York, 1967. R. Carmi, L. Itti, The Role of Memory in Guiding Attention during Natural Vision, J Vis 9 (2006), 898-914. N. Höning et al., GoodGaze: Eine Technologie aus der Hirnforschung analysiert Webseiten auf ihre Aufmerksamkeitswirkung, up08 Lübeck, 2008. J. Vockeroth, K. Bartl, S. Pfanzelt, E. Schneider, Medical documentation using a gaze-driven camera, Stud Health Tech Informat 142 (2009), 413-416. Published by IOS Press. R.J. Leigh, C. Kennard, Using saccades as a research tool in the clinical neurosciences, Brain 127 (2004), 460-477. E.H. Pinkhardt et al., Differential diagnostic value of eye movement recording in PSP-parkinsonism, Richardson’s syndrome, and idiopathic Parkinson’s disease, J Neurol 255 (2008), 1916-1925. B.M. ’t Hart et al., Gaze allocation in natural stimuli: comparing free exploration to head-ﬁxed viewing conditions, Vis Cog 17(6) (2009), 1132-1158. E. Schneider et al., EyeSeeCam: An eye movement-driven head camera for the examination of natural visual exploration, Ann N Y Acad Sci 1164 (2009), 461-467. A.T. Duchowski et al., Binocular eye tracking in VR for visual inspection training, VRST 8 (2001). G.P. Mylonas et al., Gaze-contingent soft tissue deformation tracking for minimally invasive robotic surgery, MICCAI (2005), 843-850. C. Hennessey, P. Lawrence, 3d point-of-gaze estimation on a volumetric display, Proceedings of the 2008 symposium on Eye tracking research & applications 59. J. Vockeroth et al., The combination of a mobile gaze-driven and a head-mounted camera in a hybrid perspective setup, IEEE SMC (2007), 2576-2581. T. Dera, G. Boning, S. Bardins, E. Schneider, Low-latency video tracking of horizontal, vertical, and torsional eye movements as a basis for 3dof realtime motion control of a head-mounted camera. IEEE SMC (2006). T. Villgrattner, H. Ulbrich, Piezo-driven two-degree-of-freedom camera orientation system. IEEE ICIT (2008), 1-6. T. Eggert, Eye movement recordings: methods, Dev Ophthalmol 40 (2007), 15-34. C. Rashbass, G. Westheimer, Disjunctive eye movements, J Physiol 159 (1961), 339-360. R. Stiefelhagen, J. Yang, A. Waibel, Estimating Focus of Attention Based on Gaze and Sound, ACM Int Conf Proc Series 15 (2001). A. Frischen, A.P. Bayliss, S.P. Tipper, Gaze Cueing of Attention, Psychol Bull 133(4) (2007), 694-724. M.J. Green, J.H. Waldron, M. Coltheart, Eye Movements Reﬂect Aberrant Processing of Social Context in Schizophrenia, Schizophr Bull 31(2) (2005) 470. E. Schneider et al., Documentation and teaching of surgery with an eye movement driven head-mounted camera: see what the surgeon sees and does, Stud Health Tech Informat 119 (2006), 486-90. C. Schulz et al., Eye tracking for assessment of workload: a pilot study in an anaesthesia simulator environment, Br. J. Anaesth. ﬁrst published online October 30, (2010) doi:10.1093/bja/aeq307

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-623

623

High-Field MRI-Compatible Needle Placement Robot for Prostate Interventions Hao SUa,1, Alex CAMILOa, Gregory A. COLEa, Nobuhiko HATAb Clare M. TEMPANYb and Gregory S. FISCHERa a Worcester Polytechnic Institute, Worcester, MA b Brigham and Women’s Hospital, Harvard Medical School, Boston, MA

Abstract. This paper presents the design of a magnetic resonance imaging (MRI) compatible needle placement system actuated by piezoelectric actuators for prostate brachytherapy and biopsy. An MRI-compatible modular 3 degree-offreedom (DOF) needle driver module coupled with a 3-DOF x-y-z stage is proposed as a slave robot to precisely deliver radioactive brachytherapy seeds under interactive MRI guidance. The needle driver module provides for needle cannula rotation, needle insertion and cannula retraction to enable the brachytherapy procedure with the preloaded needles. The device mimics the manual physician gesture by two point grasping (hub and base) and provides direct force measurement of needle insertion force by fiber optic force sensors. The fabricated prototype is presented and an experiment with phantom trials in 3T MRI is analyzed to demonstrate the system compatibility. Keywords. MRI compatible robot, prostate brachytherapy, biopsy.

1. Introduction Prostate cancer continues to be the most common male cancer and the second most common type of cancer in human. The estimated new prostate cancer cases (192,280) in 2009 account for 25% incident cases in men [1]. The current "gold standard" transrectal ultrasound (TRUS) for guiding both biopsy and brachytherapy is accredited for its real-time nature, low cost, and ease of use. However, TRUS-guided biopsy has a detection rate as low as 20%-30% and the radiation seeds cannot be effectively observed on the images [2]. On the other hand, the MRI-based medical diagnosis paradigm capitalizes on the novel benefits and capabilities of the scanner. These are created by the combination of capability for detecting seeds, high-fidelity soft tissue contrast and spatial resolution. The challenges, however, arise from the manifestation of the bidirectional MRI compatibility requirement - both the device should not disturb the scanner function and should not create image artifacts and the scanner should not disturb the device functionality. Moreover, the confined physical space in closed-bore high-field MRI presents formidable challenges for material selection and mechanical design. Early MRI-compatible robots focus on manual driven or ultrasonic motor driven and the latter cannot run during imaging due to significant signal loss. Chinzei, 1

Corresponding Author. Hao Su, Automation and Interventional Medicine (AIM) Robotics Laboratory, Worcester Polytechnic Institute, Higgins Lab 130, 100 Institute Road, Worcester, MA 01609, USA. Tel.: +1508-831-5191; Fax: +1-508-831-5680; E-mail: [email protected].

624

H. Su et al. / High-Field MRI-Compatible Needle Placement Robot for Prostate Interventions

et al. developed a general-purpose robotic assistant for open MRI [3] that was subsequently adapted for transperineal intraprostatic needle placement. Krieger et al. [4] presented a 2-DOF passive, un-encoded, and manually manipulated mechanical linkage to aim a needle guide for transrectal prostate biopsy with MRI guidance. Stoianovici et al. [5] described a MRI-compatible pneumatic stepper motor PneuStep, which has a very low level of image interference. Song et al. [6] presented a pneumatic robot for MRI-guided transperineal prostate biopsy and brachytherapy. However the scalability, simplicity, size and inherent robustness of electromechanical systems present a clear advantage over pneumatically actuated systems [7-9]. The difficulty arises from the actuator driving controller that usually induces significant image artifact using off-the-shelf control circuits [10-11]. Needle steering is becoming a practical technique to address needle placement accuracy issues in recent years. Mahvash et al. [12] have experimentally demonstrated that increased needle velocity is able to minimize tissue deformation and damage and reduce position error. To bridge the gap between MRI compatible mechatronics and needle steering techniques, the contributions of this paper are: (1) design of a modular needle driver that can be coupled to a base Cartesian motion platform to improve feasibility and accuracy of MRI-guided prostate interventions and their outcome and (2) experimental demonstration of real-time in-situ MRI compatibility and potential for multiple imager compatible surgery. To the authors’ knowledge, this is the first needle steering robot capable of operating in real-time MRI.

2. Methods and Materials In this paper, we presented a 3-DOF needle driver as slave robot to provide haptic feedback as shown in Fig. 1. The overall goal is to provide force feedback using fiber optic force sensor during interventional MRI-guided prostate interventions [13-14]. The primary design requirements and the features of the needle driver include: 1) 3-DOF motion needle driver. It provides cannula rotation and insertion (2DOF) and stylet translation (1-DOF). The independent rotation and translation motion of the cannula can increase the targeting accuracy while minimize the tissue deformation and damage. 2) Safety. Interventional robots require a redundant safety mechanism. Three approaches are implemented to minimize the consequences of system malfunction. a) Mechanical travel limitations mounted on the needle insertion axis that prevents linear motor rod running out of traveling range; b) Software calculates robot kinematics and watchdog routine that monitors robot motion and needle tip position; and c) Emergency power button that can be triggered by the operator. 3) MRI Compatibility. The robot components are primarily constructed of acrylonitrile butadiene styrene (ABS) and acrylic. Ferromagnetic materials are avoided. Limiting the amount of conductive hardware ensures imaging compatibility in the mechanical level. The piezoelectric driver has proven minimal image interference in the electrical level. 4) Operation in confined space. To fit into the scanner bore, the width of the driver is limited to 6cm and the operational space when connected to a base platform is able to cover the perineal area using traditional brachytherapy 60 mm × 60mm templates.

H. Su et al. / High-Field MRI-Compatible Needle Placement Robot for Prostate Interventions

625

5) Sterilization. Only the needle clamp and guide (made of low cost plastic) have contact with the needle and are disposable. 6) Compliance with transperineal needle placement, as typically performed during a TRUS guided implant procedure. This design aims to place the patient in the supine position with the legs spread and raised with similar configuration to that of TRUS-guided brachytherapy.

Figure 1: (Left) System architecture for the master - slave haptic interface. The fiber optic force sensor and robot are placed near the iso-center of the MRI scanner, the master manipulator is connected to the navigation software interface, and the two are couple through the robot controller in the scanner room using a fiber optic network connection. (right) The robot prototype in the bore of a 3T MRI scanner with a phantom.

2.1. Needle Placement Robot Design The needle placement robot consists of a needle driver module (3-DOF) and Cartesian positioning module (3-DOF). The material is rapid prototyped with ABS and laser cut acrylic. Considering the supine configuration and the robot workspace, the width of the robot is limited to 6cm. As shown in Fig. 2 (left), the lower layer of the needle driver module is driven with linear piezoelectric motor and the upper layer provides cannula rotation motion and stylet prismatic motion. To design a needle driver that allows a large variety of standard needles, a new clamping device shown in Fig. 2 (right) rigidly connects the needle shaft to the driving motor mechanism. This structure is a collet mechanism and a hollow screw made of stereolithography ABS is twisted to fasten the collet thus rigidly locks the needle shaft on the clamping device. The clamping device is connected to the rotary motor through a timing belt that can be fastened by an eccentric belt tensioner. The clamping device is generic in the sense that we have designed 3 sets of collets and each collet can accommodate a width range of needle diameters. The needle driver is designed to operate with standard MR-compatible needles of various sizes. The overall needle diameter range is from 25 Gauge to 7 Gauge. By this token, it can not only fasten brachytherapy needle, but also biopsy needles and most other standard needles instead of designing some specific structure to hold the needle handle.

626

H. Su et al. / High-Field MRI-Compatible Needle Placement Robot for Prostate Interventions

Figure 2: (Left) Physical prototype of 6-DOF piezoelectric needle placement robot consisting of needle driver module and Cartesian gross positioning module, (right) a exploded view of the needle clamping mechanism, optical tracking frame and rotary motor fixture with timing belt tensioner.

Once a preloaded needle or biopsy gun is inserted, the collet can rigidly clamp the cannula shaft. Since the linear motor is collinear with the collet and shaft, we need to offset the shaft to manually load the needle. We designed a brass spring preloaded mechanism that provides lateral passive motion freedom. The operator can squeeze the mechanism and offset the top motor fixture then insert the loaded needle through plain bearing housing and finally lock with the needle clamping. This structure allows for easy, reliable and rapid loading and unloading of standard needles. 2.2. Needle Placement Robot Navigation Dynamic global registration between the robot and scanner is achieved by passive tracking the fiducial frame in front of the robot as shown in Fig. 2 (right). The rigid structure of the fiducial frame is made of ABS and seven MR Spot fiducials (Beekley, Bristol, CT) are embedded in the frame to form a Z shape passive fiducial. Any arbitrary MR image slicing through all of the rods provides the full 6-DOF pose of the frame, and thus the robot, with respect to the scanner [7]. Thus, by locating the fiducial attached to the robot, the transformation between patient coordinates (where planning is performed) and the needle placement robot is known. To enhance the system reliability and robust, multiple slices of fiducial images are used to register robot position using principal component analysis method. The end effector location is then calculated from the kinematics based on the encoder positions. 2.3. Piezoelectric Actuator Driver The piezoelectric actuators (PiezoMotor, Uppsala, Sweden) chosen are non-harmonic piezoelectric motor which have two advantages over a harmonic drive: the noise caused by the driving wave is much easier to suppress, and the motion produced by the motors is generally at a more desirable speed and torque. Even though piezoelectric motor does not generate magnetic field, commercial motor driver boards usually induce significant image artifact due to electrical noise according to the most recent result [15]. A new low noise driver was developed and its architecture is shown in Fig. 3 (left) and Fig. 3 (right) shows the board prototype. Waveform tables are stored in RAM and utilized by a synthesizer running on the FPGA to generate four independent control waveforms of

H. Su et al. / High-Field MRI-Compatible Needle Placement Robot for Prostate Interventions

627

arbitrary phase and frequency. These control waveforms are then streamed out to the analog amplification stage at 25 mega samples per second.

Figure 3: (Left) piezoelectric actuator driver architecture using FPGA generated waveform, (right) the piezoelectric driver board prototype, a key aspect of generating the low noise high precision motion.

3. Results Four imaging protocols as shown in Table 1, were selected for evaluation of compatibility of the system: 1) diagnostic imaging T1-weighted fast gradient echo (T1 FGE/FFE), 2) diagnostic imaging T2-weighted fast spin echo (T2 FSE/TSE), 3) highspeed real-time imaging fast gradient echo (FGRE), and 4) functional imaging spin echo-planar imaging (SE EPI). Details of the scan protocols are shown in Table 1. All sequences were acquired with a slice thickness of 5mm and a number of excitations (NEX) of one. Three configurations were evaluated and used in the comparison: 1) baseline of the phantom only, 2) motor powered with controllers DC power supply turned on and 3) system servoing inside MRI board. Three slices were acquired per imaging protocol for each configuration. Table 1: SCAN PARAMETERS FOR COMPATIBILITY EVALUATION Protocol T1W FFE T2W TSE FGRE SE EPI

FOV 240 mm 240 mm 240 mm 240 mm

TE 2.3 ms 90 ms 2.1 ms 45 ms

TR 225 ms 3000 ms 6.4 ms 188 ms

FA 75 90 50 90

Bandwidth 751 Hz/pixel 158 Hz/pixel 217 Hz/pixel 745 Hz/pixel

As can be seen in Fig. 4 (left), the motors and encoders provide very small visually identifiable interference with the operation of the scanner. Fig. 4 (right) depicts one slice of the tracking fiducial frame which provides the full position information of the robot. We utilize signal to noise ratio (SNR) as the metric for evaluating MR compatibility with baseline phantom image comparison. For comparison, the SNR of each configuration was normalized by the average SNR of the 3 baseline images for each imaging protocol. SNR was calculated as the mean signal in the center of the phantom divided by the noise intensity outside the phantom [10]. Statistical analysis with a Tukey Multiple Comparison confirms that no pair shows significant signal degradation with a 95% confidence interval.

628

H. Su et al. / High-Field MRI-Compatible Needle Placement Robot for Prostate Interventions

Figure 4: (Left) Representative results showing the images obtained of baseline and system servoing inside scanner bore conditions, (right) one slice of tracking fiducial frame besides a phantom. This result presents significant improvement over recent research [15].

4. Discussion This paper presents the design of a MRI-compatible needle placement system actuated by piezoelectric actuators for transperineal prostate brachytherapy. It consists of a modular 3DOF needle driver module coupled with a 3-DOF x-y-z stage. Initial comparability testing verified the system architecture and electrical comparability. This test has confirmed that no pair showed significant signal degradation with a 95% confidence interval. Piezoelectric driven robot position control accuracy is being investigated. Future works include integrating the fiber optic sensors and phantom brachytherapy evaluation.

Acknowledgements We gratefully acknowledge the support from the Congressionally Directed Medical Research Programs Prostate Cancer Research Program New Investigator Award W81XWH-09-1-0191 and Worcester Polytechnic Institute internal funds.

References [1] A. Jemal, R. Siegel, E. Ward, Y. Hao, J. Xu, and M. J. Thun, “Cancer statistics, 2009,” CA Cancer J Clin, vol. 59, pp. caac.20006–249, May 2009. [2] J. C. Presti, “Prostate cancer: assessment of risk using digital rectal examination, tumor grade, prostatespecific antigen, and systematic biopsy.” Radiol Clin North Am, vol. 38, pp. 49–58, Jan 2000. [3] K. Chinzei and et al, "MR Compatible Surgical Assist Robot: System Integration and Preliminary Feasibility Study," in MICCAI 2000, 2000, pp. 921–930. [4] A. Krieger, C. Csoma, I. I. Iordachital, P. Guion, A. K. Singh,G. Fichtinger, and L. L. Whitcomb, “Design and preliminary accuracy studies of an MRI-guided transrectal prostate intervention system.,” MICCAI2007, vol. 10, pp. 59–67, 2007. [5] D. Stoianovici, D. Song, D. Petrisor, D. Ursu, D. Mazilu, M. Muntener, M. Mutener, M. Schar, and A. Patriciu, “MRI stealth robot for prostate interventions.,” Minim Invasive Ther Allied Technol, vol. 16, no. 4, pp. 241–248, 2007.

H. Su et al. / High-Field MRI-Compatible Needle Placement Robot for Prostate Interventions

629

[6] S.E. Song, N. B. Cho, G. Fischer, N. Hata, C. Tempany, G. Fichtinger, and I. Iordachita, “Development of a pneumatic robot for MRI-guided transperineal prostate biopsy and brachytherapy: New approaches,” in Proc. IEEE International Conference on Robotics and Automation ICRA, 2010. [7] Fischer GS, Iordachita I, Csoma C, Tokuda J, DiMaio SP, Tempany CM, Hata N, Fichtinger G, MRICompatible Pneumatic Robot for Transperineal Prostate Needle Placement, IEEE / ASME Transactions on Mechatronics - Focused section on MRI Compatible Mechatronic Systems, Vol 13, No 3, pp 295305, June 2008. [8] Y. Wang, H. Su, K. Harrington and G. Fischer, “Sliding Mode Control of Piezoelectric Valve Regulated Pneumatic Actuator for MRI-Compatible Robotic Intervention”, ASME Dynamic Systems and Control Conference, Boston, USA, 2010 [9] H. Su and G. S. Fischer, “High-field MRI-Compatible Needle Placement Robots for Prostate Interventions: Pneumatic and Piezoelectric Approaches”, eds. T. Gulrez and A. Hassanien, Advances in Robotics and Virtual Reality, Springer-Verlag, to appear in 2011 [10] Y. Wang, G. Cole, H. Su, J. Pilitsis, and G. Fischer, “MRI compatibility evaluation of a piezoelectric actuator system for a neural interventional robot,” in Annual Conference of IEEE Engineering in Medicine and Biology Society, (Minneapolis, MN), pp. 6072–6075, 2009. [11] G. Cole, K. Harrington, H. Su, A. Camilo, J. Pilitsis, G. S. Fischer, “Closed-Loop Actuated Surgical System Utilizing In-Situ Real-Time MRI Guidance”, 12th International Symposium on Experimental Robotics (ISER2010), New Delhi & Agra, India, 2010 [12] M. Mahvash and P. Dupont, “Fast needle insertion to minimize tissue deformation and damage,” in Proc. IEEE International Conference on Robotics and Automation ICRA 2009, pp. 3097 – 3102, 2009. [13] H. Su and G. Fischer, “A 3-axis optical force/torque sensor for prostate needle placement in magnetic resonance imaging environments,” 2nd Annual IEEE International Conference on Technologies for Practical Robot Applications, (Boston, MA, USA), pp. 5–9, IEEE, 2009. [14] H. Su, W. Shang, G. Cole, K. Harrington, and F. S. Gregory, “Haptic system design for MRI-guided needle based prostate brachytherapy,” IEEE Haptics Symposium 2010, (Boston, MA, USA). [15] A. Krieger, I. Iordachita, S. Song, N. Cho, G. Fichtinger, and L. Whitcomb, "Development and Preliminary Evaluation of an Actuated MRI-Compatible Robotic Device for MRI-Guided Prostate Intervention," in Proc. of IEEE International Conference on Robotics and Automation (ICRA 2010)

630

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-630

Electromyographic Correlates of Learning during Robotic Surgical Training in Virtual Reality Irene H. SUH a,b, Mukul MUKHERJEE d, Ryan SCHRACK b, Shi-Hyun PARK d, Jung-hung CHIEN a,b,d, Dmitry OLEYNIKOV b,c, Ka-Chun SIU a,b,d,1 a

College of Public Health, b Center for Advanced Surgical Technology c Dept of Surgery, University of Nebraska Medical Center; d Nebraska Biomechanics Core Facility, University of Nebraska at Omaha, Omaha, Nebraska, USA.

Abstract. The purpose of this study was to investigate the muscle activation and the muscle frequency response of the dominant arm muscles (flexor carpi radialis and extensor digitorum) and hand muscles (abductor pollicis and first dorsal interosseous) during robotic surgical skills training in a virtual environment. The virtual surgical training tasks consisted of bimanual carrying, needle passing and mesh alignment. The experimental group (n=5) was trained by performing four blocks of the virtual surgical tasks using the da VinciTM surgical robot. During the pre- and post-training tests, all subjects were tested by performing a suturing task on a “life-like” suture pad. The control group (n=5) performed only the suturing task without any virtual task training. Differences between pre- and post-training tests were significantly greater in the virtual reality group, as compared to the control group in the muscle activation of the hand muscle (abductor pollicis) for both the suture tying and the suture running (p < 0.05). In conclusion, changes in electrographic activity shows that training in virtual reality leads to specific changes in neuromotor control of robotic surgical tasks. Keywords. Electromyography, Training, da VinciTM Surgical System, Simulation

1.

Introduction

Some of the problems commonly encountered in traditional laparoscopy are visual constraints and reduced dexterity [1]. Robotic surgical systems commonly used in minimally invasive surgery can overcome some of these drawbacks of traditional laparoscopy. Virtual Reality (VR) has been used to improve training for traditional laparoscopy and to give surgeons superior performance in the operating room [2]. VR simulations can also provide user-friendly, attractive and inexpensive environments to learn robotic surgical skills. In our previous research studies, we have shown that robotic surgical skills learning through VR simulations are comparable with real world surgical skill improvement tasks [3, 4, 5]. We implemented the VR simulations as part of a training 1

Corresponding Author: Email: [email protected] web: http://www.unmc.edu/cast/

I.H. Suh et al. / Electromyographic Correlates of Learning During Robotic Surgical Training in VR 631

program and determined the effect of learning on a common real-world surgical task, such as suturing. Despite significant increases in the popularity of robotic surgeries, current literature has seldom addressed the physiologic effects of training with a robotic surgical system, particularly muscle activity [6]. The purpose of this study was to investigate the muscle activation and the muscle frequency response of signals of dominant arm and hand muscles (flexor carpi radialis (FCR), extensor digitorum (ED), abductor pollicis (AP), and first dorsal interosseous (DI)) after VR training using the da VinciTM Surgical System (Intuitive Surgical, Sunnyvale, CA).

2.

Methods

Subjects Eleven young healthy student volunteers from the University of Nebraska Medical Center and the University of Nebraska at Omaha participated in this study. Subjects were randomly assigned to either the VR group or the control group. Experimental Protocol Subjects in the VR group performed the three tasks in four blocks. In each block, each of the three tasks was performed five times. The order of tasks was randomized within each block. The Webots software (Cyberbotics, Lausanne, Switzerland) was used to build the VR environment which was driven by kinematic data streaming in real-time from the operating console of the da VinciTM surgical system. Subjects in the control group performed only the pre- and post-test before and after a gap of 2.5 hours (the average time to complete the VR training). Training Tasks Subjects who were in the VR training group performed three tasks in a virtual environment (Figure 1): bimanual carrying (BC), needle passing (NP) and Mesh Alignment (MA). (A)

(C)

(B)

Figure 1. The surgical tasks in the virtual environment: (A) Bimanual Carrying, (B) Needle passing, (C) Mesh Alignment.

632 I.H. Suh et al. / Electromyographic Correlates of Learning During Robotic Surgical Training in VR

For the BC task, the volunteers simultaneously transferred two plastic pieces in opposite directions five times consecutively. The NP task required the passing of a surgical needle through six pairs of holes. In the MA task, a virtual rolled-up mesh was opened up by the simulated arms of the robot and placed on a pre-marked virtual task platform. These tasks were designed to mimic real-life surgical skills training in terms of their cyclic nature (BC task), decision-making skills (determining location of touch sensors to unroll the mesh in the MA task) and grasping, transferring, and release skills in both BC and NP tasks. Testing Task For the testing task, all subjects performed three trials of the following procedure repairing an enterotomy on a life-like suture pad (Figure 2). This procedure consisted of using the da VinciTM Surgical System for making three single knots (suture tying), five running passes (suture running) followed by three single knots again between predefined locations on the suture pad. The two surgical skill components (suture tying and suture running) were used for data analysis.

Figure 2. Repairing an enterotomy on the life-like suture pad: One subject’s performance at the Pre-testing (left) and Post-testing (right)

Data Collection and Analysis The electromyography (EMG) data was collected using the Delsys Myomonitor® Wireless EMG system (Delsys, Inc., Boston, MA) and was sampled at 1,000 Hz using the EMGworks Acquisition software (Delsys, Inc., Boston, MA). Surface electrodes were placed over the bellies of the following four muscles of the dominant forearm (FCR and ED) and hand (AP and DI). The EMG signals were then analyzed according to Narazaki et al. [7] and Judkins et al. [8]. To quantify the extent of muscle activation, the normalized EMG signals (i.e., percentage of raw EMG outputs relative to maximal EMG output) for each muscle in each trial were integrated for the entire task completion time, and the total volume of muscle activation (EMGv) was obtained. Moreover, the activation rate (EMGr) was calculated by dividing EMGv by time. Frequency-domain analysis of EMG provides a window into muscle fatigue and motor unit recruitment [9]. Raw EMG was first filtered using a 2nd order Butterworth band-pass filter from 20-300 Hz. Raw EMG was then converted to the frequency domain using a Fast Fourier Transform to determine the power spectrum. Median frequency and frequency bandwidth were computed from the power spectrum. Median frequency was computed as the frequency at half of the integrated power spectrum as given by the following equation [9]:

I.H. Suh et al. / Electromyographic Correlates of Learning During Robotic Surgical Training in VR 633

Eq.(1)

where P(f) is the power at frequency f, fmed is the median frequency (Fmed), and fmax is the maximum frequency of the power spectrum. Frequency bandwidth (Fband) is the difference between the highest and lowest frequency where the power exceeds half the maximum power of the power spectrum. The differences between the pre- and post-training tests were calculated for statistical analysis. The relative differences were also calculated between the pre- and post-training tests for the descriptive statistics of each muscle. Statistical Analysis Separate independent t-tests were applied for both suture tying and suture running of the testing task. Dependent variables: EMGv, EMGr, Fmed and Fband

3.

Results

Our results showed that the differences between pre- and post-training tests were significantly greater in the VR group, as compared to the control group for the EMGv of the AP muscle for both suture tying and the suture running (p < 0.05) (Figure 3). For the FCR muscle, the EMGv showed a significantly greater difference but only for suture running (Figure 4).

0.25

* 0.2

FCR Right

CTRL VR

*

0.15

0.1

0.05

Differences between Pre - Post training test EMGv (mV sec)

Differences between Pre - Post training test EMGv (mV sec)

AP Right

CTRL

VR

0.06

*

0.05 0.04 0.03 0.02 0.01 0

0

Suture Tying

Suture Running

Figure 3. The hand muscle (abductor pollicis (AP)) activations between pre- and post-training tests in suture tying and running

Suture Tying

Suture Running

Figure 4. The forearm muscle (flexor carpi radialis (FCR)) activations between pre- and post-training tests in suture tying and running

634 I.H. Suh et al. / Electromyographic Correlates of Learning During Robotic Surgical Training in VR

The relative differences in EMGr for the AP muscle were 26% for control group and 55% for VR group in suture tying. For suture running, the relative differences were 28% and 57% for control and VR groups respectively. For the FCR muscle, the relative differences in EMGr were 10% for control group and 3% for VR group in suture tying. For suture running, the relative differences were 20% and 32% for control and VR groups respectively. No differences were found in both EMGv and EMGr for DI and ED muscles as well as in both Fmed and Fband between two groups for all muscles.

4.

Conclusions

Our results are highly encouraging in indicating that training with simulated surgical tasks may result in improvement of actual surgical skills. These results were also reflected in our previous study [10], in terms of the differences in the speed of the instrument tips between the pre and post-training tests on the suture pad. In that study, the VR group had shown a higher increase in speed in both aspects of the surgical task. We had shown these improvements previously in kinematic parameters [6] and through this study. We demonstrated that these improvements also have electrophysiological correlates, especially in those muscles (AP and FCR) which seem to be the primary contributor for controlling the telemanipulators of the surgical system. Further improvement of the VR environment can enhance the learning effect even more.

5.

Acknowledgement

This work was supported by the Nebraska Research Initiative and the Center for Advanced Surgical Technology at the University of Nebraska Medical Center.

References [1]

J.D. Hernandez, S.D. Bann, Y. Munz, K. Moorthy, V. Datta, S. Martin, A. Dosis, F. Bello, A. Darzi and T. Rockall, Qualitative and quantitative analysis of the learning curve of a simulated surgical task on the da Vinci system. Surgical Endoscopy 18 (2004), 372-378. [2] A.G. Gallagher, E.M. Ritter, H. Champion, G. Higgins, M.P. Fried, G. Moses, C.D. Smith, R.M. Satava, Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann Surg. 241 (2005), 364-72. [3] D. Katsavelis, K.C. Siu, B. Brown-Clerk, I. Lee, Y.K. Lee, D. Oleynikov, N. Stergiou, Validated robotic laparoscopic surgical training in a virtual-reality environment. Surg Endosc. 23 (2009), 66-73. [4] B. Brown-Clerk, K.C. Siu, D. Katsavelis, I. Lee, D. Oleynikov, N. Stergiou, Validating advanced robotassisted laparoscopic training task in virtual reality. Stud Health Technol Inform. 132 (2008), 45-9. [5] M.J. Fiedler, S.J. Chen, T.N. Judkins, D. Oleynikov, N. Stergiou, Virtual reality for robotic laparoscopic surgical training. Stud Health Technol Inform.125 (2007), 127-9. [6] T.N. Judkins, D. Oleynikov, N. Stergiou, Electromyographic response is altered during robotic surgical training with augmented feedback. J Biomech. 5 (2009), 71-6. [7] K. Narazaki, D. Oleynikov, N. Stergiou, Robotic surgery training and performance: identifying objective variables for quantifying the extent of proficiency. Surg Endosc, 20 (2006), 96-103. [8] T.N. Judkins, D. Oleynikov, K. Narazaki, N. Stergiou, Robotic surgery and training: electromyographic correlates of robotic laparoscopic training. Surg Endosc. 20 (2006), 824-9. [9] Basmajian & De Luca., Muscles alive, their functions revealed by electromyography, Williams & Wilkins, Baltimore, MD, 1985 [10] M. Mukherjee, K.C. Siu, IH Suh, A. Klutman, D. Oleynikov, N. Stergiou, A virtual reality training program for improvement of robotic surgical skills. Stud Health Technol Inform. 142 (2009), 210-4.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-635

635

Web-Based Interactive Volume Rendering Stefan SUWELACK, Sebastian MAIER, Roland UNTERHINNINGHOFEN and Rüdiger DILLMANN Institute for Anthropomatics, Karlsruhe Institute of Technology, Germany Abstract. In this paper we present a web-based remote visualization system. The system makes use of video stream based techniques to reduce the network bandwidth requirements and is capable of performing interactive volume rendering on computed tomography data in real-time. The technique allows embedding interactive volume rendering into a website. The concrete contribution of this paper is twofold. First, we outline a Microsoft Silverlight based implementation of the prototype and describe the applied video encoding techniques. Furthermore we present experimental results that allow evaluating the system in terms of latency and image quality. In particular, we show that the additional delay of stream based remote visualization is very small if compared to picture based techniques. Keywords. interactive remote visualization, volume rendering, web-based visualization

1. Introduction Current medical visualization and simulation techniques often have demanding hardware requirements. A typical example is the volume rendering of computed tomography (CT) data. In order to perform interactive volume rendering on portable devices such as netbooks or tablet PCs the actual rendering has to be done on a remote server [1]. Most systems use compressed images to deliver the rendered data to the client [3]. The amount of data that has to be streamed over the network can be signiﬁcantly reduced by using video encoding techniques [2]. The disadvantages of this method are the additional encoding time for the video stream and the additional computational effort for the decoding on the client side. In this paper we present a stream-based remote visualization system whose client is based on the Microsoft Silverlight application framework. Thus no additional software has to be installed on the client as even complex medical visualizations can be delivered as web-based content. Additionally the GPU based video decoding capabilities of MS Silverlight reduce the computational load of the client. We analyze the system with respect to encoding time and image quality for the remote visualization of volume rendered CT images. 2. Methods Web-Based Video Streaming The system can encode the data either as a H.264 or a WMV1 video stream. The encoding parameters for both formats have been optimized for real-time encoding. In particular no

636

S. Suwelack et al. / Web-Based Interactive Volume Rendering

Figure 1. Close-ups of a volume rendered CT image with a resolution of 512x512 pixels. The image is encoded using the H.264 algorithm (middle column) and the WMV1 algorithm (right column) at bitrates of 1000kbit/s (upper row) and 250 kbit/s (lower row).

bidirectional frames are allowed. On the client side the video images are extracted and the video stream is decoded natively using the Silverlight GPU accelerated algorithms, thus keeping the client hardware requirements low. Remote Visualization Pipeline The complete system can be integrated into the visualization toolkit (VTK) [4]. The mouse and keyboard events which trigger the update of the visualization pipeline are transferred from the client to the server using a custom protocol. The realtime transport protocol (RTP) is used for video transmission. On the client side the encoded video stream is delivered to a MS Silverlight MediaElement through the MediaStreamSource API. The server component that handles the socket connections to the clients allows transmitting several video streams per client. GPU Raycasting The volume rendering is performed using GPU based ray casting. In order to increase the computational efﬁciency the algorithm adapts the sampling step size based on a spectral analysis of the dataset and the transfer function [5].

3. Results The system was evaluated in terms of image quality and latency. It has to be noted that the encoded video frames differ in quality and size. In particular there is a huge difference between keyframes and predicted frames (P-frames). We address this problem by averaging the performance measurements over 1000 frames. In order to assess the quality of the images we compare typical non-keyframe sequences. The results of the performance measurements are displayed in Table 1. We point out that the encoding time is very low for the WMV1 algorithm (9ms for a resolution of 768x768 pixels). In comparison the encoding using H.264 compression takes more than

S. Suwelack et al. / Web-Based Interactive Volume Rendering

637

Table 1. Encoding time for the H.264 and WMV1 algorithms and image resolutions of 512x512 and 768x768 pixels. Delay of the system for different networks and image resolutions (far right column). Video resolution

Encoding time (ms)

Network delay

@ bit rate

WMV1

H264

(ms)

5122 @1000 kbit/s

3.8

24.0

1.1

7682 @2500 kbit/s

9.0

57.0

2.0

LAN

5122 @1000 kbit/s 7682 @2500 kbit/s

3.8 9.0

24.0 57.0

38.7 37.3

W-LAN

5122 @1000 kbit/s

3.8

24.0

122.3

7682 @2500 kbit/s

9.0

57.0

165.3

5122 @1000 kbit/s 7682 @2500 kbit/s

3.8 9.0

24.0 57.0

209.4 282.5

Localhost

Internet

six times as long. Table 1 also reveals that the delay introduced by the WMV1 encoding is nearly negligible if compared to the network delay. The difference in image quality between the two encoding algorithms is clearly visible when low bandwith settings are used (see Fig. 1). Although the H.264 encoding gives better results in all scenarios, the difference in image quality is low for high bandwidth streams. 4. Conclusions We presented a novel approach for web-based remote visualization of volume rendered CT images. The system allows embedding interactive volume rendering into a website as a MS Silverlight application. A detailed analysis shows that the encoding time is much smaller than the network delay. We thus conclude that the additional delay of stream based remote visualization is very small if compared to picture based techniques. Furthermore, stream based techniques signiﬁcantly reduce the bandwith requirements and can deliver high quality content over end-user internet connections. Future work includes the reduction of the H.264 encoding time by using GPU accelerated algorithms or special purpose hardware. Also, we are currently integrating the presented system into an online anatomy learning tool which features volume rendered images to improve the learning experience of the students. References [1] [2] [3]

[4] [5]

K. Engel, O. Sommer, and T. Ertl. A framework for interactive hardware accelerated remote 3dvisualization. In Proceedings of EG/IEEE TCVG Symposium on Visualization. Citeseer, 2000. F. Lamberti and A. Sanna. A streaming-based solution for remote visualization of 3D graphics on mobile devices. IEEE Transactions on Visualization and Computer Graphics, 2007. B. Paul, S. Ahern, E.W. Bethel, E. Brugger, R. Cook, J. Daniel, K. Lewis, J. Owen, and D. Southard. Chromium renderserver: Scalable and open remote rendering infrastructure. IEEE Transactions on Visualization and Computer Graphics, 2008. Will Schroeder, Kenneth M. Martin, and William E. Lorensen. The visualization toolkit (2nd ed.): an object-oriented approach to 3D graphics. Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 1998. Stefan Suwelack, Eric Heitz, Roland Unterhinninghofen, and Rüdiger Dillmann. Adaptive gpu ray casting based on spectral analysis. In Medical Imaging and Augmented Reality, Lecture Notes in Computer Science, pages 169–178. Springer Berlin / Heidelberg, 2010.

638

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-638

A Method of Synchronization for Haptic Collaborative Virtual Environments in Multipoint and Multi-level Computer Performance Systems Kazuyoshi Tagawa a,1, Tatsuro Bito a, and Hiromi T. Tanaka a a Ritsumeikan University, Japan

Abstract. We have developed a novel volume-based haptic communication system. It allows participants at remote sites on a network to simultaneously interact with the same target object in virtual environments presented by multilevel computer performance systems. It does this by only exchanging a small set of manipulation parameters for the target object and an additional packet to synchronize the status of the binary tree and the deformation of the local region of the shared volume model. We first developed online-remesh volume models, which we call dynamic adaptive grids, to simulate deformable objects such as soft tissues at remote sites. Then, haptic sensation during interaction with the target object was achieved by rendering the reflection force from the object, which was simulated with the online-remesh volume model from the manipulation parameters and additional packets exchanged among all remote sites. We investigated the efficiency of our system via experiments at remote locations on a WAN between Tokyo, Osaka, and Kyoto. Keywords. Elastic object, haptic collaborative virtual environment, multi-level computer performance systems

Introduction Virtual reality technology has improved as computers and networks have become faster and more powerful. Some telecommunication systems that allow users to work at remote locations on the network have been developed, and medical and educational applications are expected. The VizGrid project [1] has enabled the volume modeling of output data to be developed from simulation or experimental results and the volume modeling of images in the real world by using a multiple-view camera. Hikichi et al. [2] proposed a system of haptic collaboration without loss of quality of service (QoS). They conducted an experiment to evaluate their system using a rigid and a surface object, and the delay time, packet loss, and information loss were measured. Mortensen et al. [3] presented a study on remote collaboration between people in a shared virtual environment. Two people, one at University College London (UCL) and the other at the University of North Carolina, Chapel Hill (UNCCH), met together in the shared virtual environment 1

[email protected]

K. Tagawa et al. / Synchronization for Haptic Collaborative Virtual Environments

639

and lifted a rigid body object together and moved it to another place. Gunn et al. [4] proposed techniques allowing long-distance sharing of haptic-enabled, dynamic scenes. At the CSIRO Virtual Environments Laboratory, they have successfully used their system to connect a prototype of a surgical-simulation application between participants on opposite sides of the world in Sweden and Australia, over a standard Internet connection. However, previous work did not achieve realistic sensations for the representational model or real-time performance, and the sense of touch was not well defined. We propose a novel system of volume-based haptic communication in this paper, which allows participants at remote sites on the network to simultaneously interact with the same target object in virtual environments presented by multi-level computer performance systems by only exchanging a small set of manipulation parameters for the target object and an additional packet to synchronize the status of the binary tree and the deformation of the shared volume model. We first developed an online-remesh volume model, which we called a dynamic adaptive grid, to simulate deformable objects such as soft tissues at remote sites. Then, haptic sensation during interaction with the target object was achieved by rendering the reflection force from the object, which was simulated with the online-remesh volume model from the manipulation parameters exchanged among all remote sites. Finally, we investigated the efficiency of our system via experiments at remote locations on a WAN between Tokyo, Osaka, and Kyoto.

1. Online-Remesh Deformation Model We developed a mesh generator in our previous work [5], where an input mesh model such as an organ was represented using a binary tree of a set of tetrahedrons without any cracks being formed. The model was based on a tetrahedral adaptive mesh for the parallel hierarchical tetrahedralization of volume data. We used a dynamic tetrahedral adaptive grid of volume data [6] in the onlineremesh deformation model to rapidly simulate deformation of a visco-elastic object. This algorithm could refine this tetrahedral adaptive mesh interactively. Figures 1 and 2 outline examples of binary refinement and simplification.

Figure 1. Binary refinement and simplification

Figure 2. Examples of online-remesh

640

K. Tagawa et al. / Synchronization for Haptic Collaborative Virtual Environments

2. Synchronization Method 2.1. Overview We proposed a volume-based haptic communication system [7] to present the same deformation to all users, which allowed participants at remote sites on the network to simultaneously interact with the same target object in virtual environments by only exchanging a small set of manipulation parameters for the target object. Each PC had the same deformation model, and then the same manipulation parameters were input to all deformation models. As a result, the same simulation results were obtained and presented to the users. This method enabled the target object’s deformation to be shared by enabling small packets (manipulation parameters) to be exchanged via the network. However, if multi-level systems of computer performance are used in the communication system, interaction is apt to be unstable because of the long time-delays caused by late simulation on low-spec PC(s). To solve this problem, we extended the communication system. In the approach overviewed in Figure 3, additional packets to synchronize the status of the binary tree and the deformation of the local region (Figure 4) of the shared volume model are exchanged.

Figure 3. Overview of our synchronization method

Figure 4. Definition of local region

2.2. Send/Receive Packet The manipulation parameters (position, orientation, time stamp, and simulation time) are exchanged at a rate of 1 kHz. Additional packets are sent and received according to need in three steps (where location ID subject to synchronization is 0): Step 1 Step 2

Step 3

Exchange manipulation parameters. Search location_max, where location_max is a location ID of a PC that has maximum simulation time SimTimelocation _ max . If SimTime ࠉ0 SimTime location _ max ! D then Send an additional packet to location_max else if SimTime location _ max SimTime 0 ! D then Receive an additional packet from location_max

641

K. Tagawa et al. / Synchronization for Haptic Collaborative Virtual Environments

In Step 3, D is the threshold of difference in the simulation time. The additional packet contains the status of the binary tree and the deformation of the local region. Details on this packet and synchronization method are described in the following subsections. 2.3. Synchronization of Binary Trees The binary trees are synchronized in four steps (where location ID subject to synchronization is 0). Step 1

The status of the binary trees are expressed by bit sequences to reduce the amount of communications traffic. As shown in Figure 5, we define tetrahedrons that belong to the binary tree as 1, otherwise 0.

Step 2

These bit sequences are exchanged via the network.

Step 3

By using logical operation, the difference T0c

between a bit

sequence of binary tree T0 and bit sequences of other binary trees

T1 ,, TN 1 sent from other locations is obtained as:

T0c (T0 T1 TN 1 ) T0 .

(1)

There is an example of this logical operation in Figure 6. Step 4

Add tetrahedrons indicated by T0c to T0 .

We can obtain a synchronized and crack-free (without any cracks forming) binary tree at low computation cost and with low communications traffic by using this algorithm.

Figure 5. Coding of binary tree

Figure 6. Synchronization of binary trees

2.4. Synchronization of Deformation Tetrahedrons that belong to a large deformation region are subdivided locally in the online-remesh deformation model. These tetrahedrons appear at deep leaf nodes of the binary tree as the subdivision levels of these subdivided-tetrahedrons increase. In our approach to synchronization, deformations of these tetrahedrons are shared with other locations as follows:

642

K. Tagawa et al. / Synchronization for Haptic Collaborative Virtual Environments

Step 1

Begin at user’s manipulating tetrahedrons.

Step 2

Search adjacent tetrahedrons recursively to find which subdivision levels are E and over where E is the threshold of the subdivision level.

Step 3

Send IDs, positions, and velocities of nodes that consist of retrieved tetrahedrons to location_max.

3. Experiment We conducted an experiment to evaluate how effective our method of synchronization was by using a liver model of initial level 3 (the number of initial nodes was 96) and this level changed to level 6 (the number of nodes was 3757). Displacements based on a sine function were given as the input for three nodes, and then coordinates of one arbitrary point at each location were measured as shown in Figure 7.

Figure 7. Experimental model

3.1. Experimental Conditions Three multi-level computer performance PCs were used as shown in Table 1. Highspec. PC was allocated in Tokyo, middle-spec. PCs in Osaka, and low-spec. PCs in Kyoto. These PCs were connected with a peer-to-peer wide area network (WAN) in which we used a TCP/IP connection and network delay emulation software to emulate the JGN2plus network between Tokyo, Osaka, and Kyoto. The average round trip times were about 10 [msec] between Tokyo and Osaka and between Tokyo and Kyoto, and about 1 [msec] between Kyoto and Osaka. Table 1. Specifications of PCs Low-spec. PC Middle-spec. PC High-spec. PC

CPU Intel Xeon X5355 2.6 GHz Intel Core i7 920 2.6 GHz Intel Core i7 975 3.3 GHz

OS CentOS 5.5 CentOS 5.5 RedHat Enterprise Linux 6.0 beta

Memory 8 GB 10 GB 12 GB

3.2. Experimental Results Figures 8 and 9 plot the trajectories of nodes measured at locations with asynchronous and synchronization methods. Figure 10 is a close-up of Figure 9. The solid lines, short-dashed lines and long-dashed lines correspond to trajectories measured at low-

K. Tagawa et al. / Synchronization for Haptic Collaborative Virtual Environments

643

spec., middle-spec. and high-spec. PCs. The trajectories of each pair were more similar in the synchronous method (Figures 9 and 10) than those in the asynchronous method (Figure 8), because the differences in coordinates were periodically modified. Table 2 lists the average errors in coordinates where the threshold of difference in simulation time ( D ) was changed to 0.01, 0.05, 0.1, 0.3, and 0.5. The average errors in coordinates were reduced by using the synchronous method. Figure 11 shows examples of interaction force. Discontinuous force was observed in Figure 11(c); however, each author’s subjective opinion is that unpleasant force was not felt.

Figure 8. Deformation (asynchronous)

Figure 9. Deformation (synchronous,

Figure 10. Deformation (synchronous,

D

=0.1,

E =3, closeup)

Table 2. Average errors in coordinates [mm] Threshold of difference in simulation time ( D )

0.01

0.05

0.1

0.3

0.5

Low-spec. PC Middle-spec. PC

0.002 0.0007

0.01 0.003

0.05 0.004

0.37 0.02

0.63 0.04

D =0.1, E =3)

644

K. Tagawa et al. / Synchronization for Haptic Collaborative Virtual Environments

(a) High-spec. PC

(b) Middle-spec. PC

(c) Low-spec. PC Figure 11. Interaction force (synchronous,

D

= 0.1,

E = 3)

4. Conclusion We described a volume-based system of haptic communication that shares an adaptive volume model between remote locations and provides haptic communication to users. The model of shared volume in virtual environments was presented by multi-level computer performance systems and was shared by only exchanging a small set of manipulation parameters and additional packets to synchronize the status of the binary tree and the deformation of the local region of the model. The efficiency of the proposed algorithm was confirmed through experiments at three remote locations on a WAN between Tokyo, Osaka, and Kyoto.

References [1] VizGrid Project, Final Report of Vizgrid Project. Technical report, Japan, 2007. [2] K. Hikichi, H. Morino, I. Fukuda, S. Matsumoto, K. Sezaki, and H. Yasuda. Proposal and evaluation of system for haptics collaboration. Journal of The Institute of Electronics, Information and Communication Engineers, J86-D(2):268. 278, 2003. [3] J. Mortensen, V. Vinayagamoorthy, M. Slater, A.Steed, B. Lok, and M. Whitton. Collaboration in TeleImmersive Environments. In 8th EurographicsWorkshop on Virtual Environments, pages 93.101, 2002. [4] C. Gunn, M. Hutchins, and M. Adcock. Combating latency in haptic collaborative virtual environments. Presence: Teleoperators and Virtual Environments, 14(3):313.328, 2005. [5] Y. Takama, A. Kimura, and H. T. Tanaka. Tetrahedral adaptive mesh for parallel hierarchical tetrahedralization of volume data. Journal of The Information Processing Society of Japan, 48(SIG 9(CVIM 18)):67.78, 2007. [6] Y. Takama, H. Yamashita, and H. T. Tanaka. Dynamic tetrahedral adaptive mesh generation of volume data. In Symposium on VC/GCAD2007. [7] S.Yamaguchi, H. T. Tanaka. Toward Real-time Volume-based Haptic Communication with Realistic Sensation. Proc. of IEEE/RSJ 2007 Intr. Conf. on Intelligent Robots and Systems(IROS2007) Workshop on Modeling, Identification, and Control of Deformable Soft Objects, pp.97-104, 2007.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-645

645

A Hybrid Dynamic Deformation Model for Surgery Simulation Kazuyoshi TAGAWAa,1 and Hiromi T. TANAKA a a Ritsumeikan University, Japan

Abstract. A hybrid dynamic deformation model that is capable of haptic interaction with dynamically deformable objects is presented. This approach is also capable of taking into account large deformation and topological changes of objects. The amount of computation required in this approach is proportional to the sum of the square of the number of shared nodes and the number of whole nodes of an online-remesh model. We implemented the approach in our prototype system and confirmed the feasibility and performance through experiments. Keywords. Elastic object, deformation model, impulse response deformation model, haptic rendering

Introduction Haptic interaction with elastic objects is an important topic in the field of haptics. The greatest difficulty in such interaction is the large computation cost required to calculate the object’s deformation and the user’s reaction force. Recently, the use of multi-core processors (e.g. multi-core CPU or GPU) and novel deformation models (e.g., a record reproduction model) [2, 3, 5] have enabled haptic interaction with relatively complex objects. However, operations that involve topological changes (e.g., cutting or rupturing operations) of such objects are still difficult. We present here a hybrid deformation model as an approach to solve the above problems. This approach is capable of simulating large deformation and topological changes of dynamically deformable objects at a haptic rate.

1. Related Work 1.1. Deformation Model Researchers in the field of virtual reality often prefer to use the mass-spring model (MSM) and an explicit method to solve the model [7] because it is easy to change boundary conditions. However, the relationship between the parameters of the model and the physical properties of the target object is not clear.

1

E-mail: [email protected]

646

K. Tagawa and H.T. Tanaka / A Hybrid Dynamic Deformation Model for Surgery Simulation

In contrast, the finite element method (FEM) [9] is based on continuum dynamics. However, we have to solve large simultaneous equations; thus, the amount of computation of FEM is very large. 1.2. Acceleration Method for Simulation of Deformable Object Of course, some fast computation methods have been proposed. Kikuuwe et al. proposed a computationally efficient formulation and an algorithm for tetrahedral finite-element simulation of elastic objects subject to the Saint Venant-Kirchhoff (StVK) material law [4]. However, in cases where the model is solved by an implicit method, it is difficult to take topological changes into consideration. In previous papers, the authors proposed an online-remesh model that uses tetrahedral adaptive mesh [10]. Moreover, the authors proposed a deformation model based on a record reproduction approach which we call the impulse response deformation model (IRDM) [5]. In IRDM, the computation amount of calculation of the interaction force is not dependent on the model complexity. This is advantageous for haptic interaction; however, there are two problems. One is that the model is based on the idea of linear assumption. The other is that it is difficult to represent topological changes. This is because we must retain a large number of impulse responses (a combination of patterns of the user’s interaction and topological changes). 1.3. Rupture Model Kume et al. proposed a rupture model for a cholecystectomy simulator [6]. This is a FEM-based soft tissue destruction model that behaves according to variable tearing manipulation. However, haptic feedback is difficult because of the computation cost of recalculating a stiffness matrix. Moreover, this model is a static model; therefore, the dynamic feature of deformation is unavailable. As an approach to solve the problem above, Hirota et al. [1] and Cotin et al. [8] proposed a tensor mass model (TMM) which uses the finite element model and an explicit method. In the approach, it is easy to change boundary conditions; however, numerical computation of the model is apt to be unstable. In general, internal organs subject to surgery are connected to other organs by blood vessels or adhesion. In addition, deformation and topological change operations are applied to a specific portion of the organ. Cotin et al. proposed a hybrid deformation model [8]. In their approach, the organ is simulated by combining the TMM (deformation and topological change operations are applied) and linear FEM models. However, use of the online-remesh model was not discussed, and the dynamic feature of deformation was unavailable.

2. Hybrid Deformation Model In this paper, a hybrid deformation model is presented. In this model, IRDM and the online-remesh model are combined via shared nodes in order to perform coupled analysis. To do this analysis, we define shared nodes in each of the two models, as shown in Fig. 1. The hybrid deformation model is calculated according to the following procedure:

K. Tagawa and H.T. Tanaka / A Hybrid Dynamic Deformation Model for Surgery Simulation

647

0.

Obtain the position of the stylus of the force-feedback device.

1.

Calculate internal forces of all nodes of the online-remesh model.

2.

Send the present internal forces of the shared nodes of the online-remesh model as the present external forces of shared nodes in IRDM.

3.

Calculate displacements of shared nodes of IRDM using a convolution integral [5].

4.

Receive displacements of shared nodes of the online-remesh model from displacements of shared nodes of IRDM.

The numbers of shared nodes in IRDM and the online-remesh model are not equivalent because the tetrahedral adaptive mesh in the online-remesh model is locally subdivided based on the maximum shear strain of tetrahedrons. Therefore, we used interpolated values based on the initial positions of the shared nodes. This calculation is easy because tetrahedrons are recursively subdivided equally.

Figure 1. Hybrid deformation model.

3. Experiments We implemented our approach with our prototype system and confirmed the feasibility and performance through experiments. 3.1. Experimental Setup As shown in Fig. 2, we employed three rectangular objects that each had a width and breadth of 128 mm and 32 mm. The height of each model was 128 mm, 192 mm, and 256 mm, respectively. The bottom nodes of each object were attached to the floor. We assigned the online-remesh model to the upper side of each object. The number of nodes of the online-remesh model of each object at the initial mesh was 135, and the number of shared nodes of the models was 27. In IRDM, there were 135, 243, and 351 nodes for the respective objects. The impulse responses were obtained by FEM. The sampling frequency of the impulse response was set to 500 Hz, and the response duration was set to 1 s. We employed Young’s modulus E=2000N/m2, Poisson’s ratio =0.49, and density =110kg/m3. In the online-remesh model, we employed MSM. Young’s modulus E=10000N/m2 and density =100000kg/m3. In some part whose height was from quarter to middle of

648

K. Tagawa and H.T. Tanaka / A Hybrid Dynamic Deformation Model for Surgery Simulation

the online-remesh model, Young’s modulus was multiplied by 0.1 in order to rupture easily. We used a PC (CPU: Core i7 3.3GHz, Memory: 12GB, OS: Linux) and a PHANTOM Omni device.

Figure 2. Experimental models.

3.2. Experimental Results Fig. 3 shows an example of dynamic deformation and topological change of an object done by user manipulation. The user pulled up a node on the upper surface of the object because the superior half of the model was the online-remesh model; tetrahedral elements with large deformation were locally subdivided. The computation time of the online-remesh model was 0.35 ms. Compared to this, the computation time of IRDM of each model was 9.6 ms. Computation times of IRDM of these objects were about the same because the computation cost of IRDM is not dependent on the complexity of the model (i.e., the number of whole nodes of an object) but is dependent on the number of shared nodes of the object.

Figure 3. Example of dynamic deformation and topological change.

K. Tagawa and H.T. Tanaka / A Hybrid Dynamic Deformation Model for Surgery Simulation

649

3.3. Acceleration by GPU Parallelization of computation of IRDM is possible because the algorithm of IRDM consists of sets of convolution integrals. Therefore, we tested the acceleration of the computation using a graphics processing unit (GPU). The computation time of IRDM was 0.11 ms; for graphics cards, we used nVIDIA GeForce GTX 295 and CUDA 3.1. This speed was sufficiently fast for haptic interaction.

4. Conclusion This paper introduced a novel deformation model that can consider both dynamic behavior and topological changes of elastic objects. A simple shape of a hybrid model was implemented, and through experiments, we confirmed that the approach was feasible and had good performance.

References [1] K. Hirota and T. Kaneko. A study on the model of an elastic virtual object. Trans. of the Society of Instrument and Control Engineers, 34(3):232–238, 1998. [2] Doug L. James and Kayvon Fatahalian. Precomputing interactive dynamic deformable scenes. Proc. ACM SIGGRAPH 2003, pages 879–887, 2003. [3] K. Hirota and T. Kaneko. Haptic representation of elastic object. Presence, 10(5):525–536, 2001. [4] R. Kikuuwe, H. Tabuchi, and M. Yamamoto. An edge-based computationally efficient formulation of saint venant-kirchhoff tetrahedral finite elements. ACM Trans. on Graphics, 28(1):8:1–8:13, 2009. [5] K. Tagawa, K. Hirota, and M. Hirose. Impulse response deformation model: an approach to haptic interaction with dynamically deformable object. Proc. IEEE Haptic 2006, pages 209–215, 2006. [6] N. Kume, M. Nakao, T.Kuroda, H. Yoshihara, and M. Komori. Simulation of soft tissue ablation for a vr simulator. Trans. of Japanese Society for Medical and Biological Engineering, 43(1):76–84, 2005. [7] A. Norton, G. Turk, B. Bacon, J. Gerth, and P. Sweeney. Animation of fracture by physical modeling. Visual Computer, 7:210–219, 1991. [8] Cotin Stëphane, Delingette Hervë, and Ayache Nicholas. A hybrid elastic model for real-time cutting, deformations, and force feedback for surgery training and simulation. The Visual Computer, 16:437– 452, 2000. [9] G. Yagawa and S. Yoshimura. Computational Dynamics and CAE Series 1: Finite Element Method. Baifu-kan, Tokyo, 1991. [10] S. Yamaguchi and H. T. Tanaka. Toward real-time volume-based haptic communication with realistic sensation. Proc. of IROS2007, pages 97–104, 2007.

650

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-650

Single and Multi-User Virtual Patient Design in the Virtual World D TAYLOR MSc1, V PATEL MRCS, D COHEN MRCS, R AGGARWAL PhD, K KERR PhD, N SEVDALIS PhD, N BATRICK FRCS and A DARZI PC KBE HonFREng FMedSci Division of Surgery, Department of Surgery and Cancer, Imperial College London

Abstract. This research2 addresses the need for the flexible creation of immersive clinical training simulations for multiple interacting participants and virtual patients by using scalable open source virtual world technologies. Initial development of single-user surgical virtual patients has been followed by that of multi-user multiple casualties in a field environment and an acute hospital emergency department. The authors aim to validate and extend their reproducible framework for eventual application of virtual worlds to whole hospital major incident response simulation and to multi-agency, pan-geographic mass casualty exercises. Keywords. Virtual Patient, Virtual Worlds, Medical Training, eTraining, Major Incident Response, Simulation, Mass Casualty, Trauma, Emergency Response

Background Virtual Patients are computer simulations designed to train or assess clinicians in information gathering, diagnostic reasoning and management of individual patients[1]. The majority of online Virtual Patient designs have focused on a single user or group interaction with a single patient. The aim of this research was to develop a series of complex Virtual Patients for both single and multi-user simulations, using a reproducible design methodology and subsequently to validate their use in clinician and emergency responder training and assessment. Its novelty lies in the use of open source based technology and Second Life®/Opensim[2] for the flexible development of scalable multi-user and multi-Virtual Patient scenarios.

1 Corresponding

Author: Division of Surgery, Imperial College London, 10th Floor QEQM, St Mary's Hospital, South Wharf Road, London W2 1NY, UK; E-mail: [email protected]

2

Acknowledgements: Phase 1 was supported by the London Deanery under their Simulation Technology-enhanced Learning Initiative (STeLI). Phase 2 was supported by funding from the Health Protection Agency. Aggarwal and Sevdalis are supported by the UK’ s National Ins titute for Health Research through a Clinician Scientist award and the Imperial Centre for Patient Safety and Service Quality, respectively.

651

D. Taylor et al. / Single and Multi-User Virtual Patient Design in the Virtual World

1. Method The development of the Virtual Patients was undertaken in two phases. Phase 1 involved the development of a series of single Virtual Patients (Figure 1) for postgraduate surgical trainees to individually assess and manage. Phase 2 focused on the development of mass casualty scenarios for multiple simultaneous emergency responders and multi-disciplinary teams to assess and treat. Clinical decision trees to control Virtual Patients' responses were modelled using an editor based on an open source framework and subsequently compiled to a web player that was developed to maintain the state of multiple Virtual Patients in an open source and scalable architecture. A message broker was developed to communicate between the virtual world (flexibly Second Life or the open source equivalent, Opensim) and the web player (Figure 2). In the second phase, 3 existing modalities of training were studied: field training for HART paramedics[3] responding to an incident involving hazardous materials, an Emergo Train tabletop[4] exercise for hospital emergency departments - approved by the Department of Health as an acceptable alternative to a live exercise, and a desktop mass casualty exercise involving multiple agencies. A training needs analysis was undertaken through semi-structured interviews with trainers and users who had experienced 1 or more of the 3 exercise types. Subsequently an integrated exercise scenario was conceptualised by a multi-disciplinary team so as to meet the identified training needs, particularly those that were not adequately or costeffectively met by the 3 existing exercise modalities.

Second Life or Opensim Broker

Virtual World Client

VPs and devices

Avatars

Figure 1 Virtual Patient in Ward

Figure 2 System Architecture

Web Browser

Servlet

Web Server

VP data: Logic and Decision Trees

Open Source Application Framework MySQL Database

Editor

652

D. Taylor et al. / Single and Multi-User Virtual Patient Design in the Virtual World

Figure 3 HART Paramedics in Hazmat Suits

Figure 4 A Bomb Blast Casualty

2. Results For the first phase, 3 general surgical Virtual Patients in 3 different ward settings (Figure 1) at 3 training levels were created. These are undergoing face and construct validity testing with both junior and Consultant grade surgeons [3]. For the second phase, 3 aspects of the scenario (referred to as vignettes) were designed and implemented and their face and content validity assessment is now being undertaken with paramedics and hospital practitioners. The 1st vignette is set in a simulated field environment involving hazardous materials (Figures 3 and 4), where paramedics and other first responders could undergo concurrent training and assessment. The other 2 vignettes are set in a simulated emergency department receiving multiple casualties.

3. Discussion A reproducible framework for both single and multi-user virtual patient development was achieved. Face and content validity testing is being carried out separately for each phase 2 vignette but it is clear that the technology could support multiple teams at different virtual sites exercising at the same time in a fully integrated exercise. This might include multiple agencies and teams at the site of the incident, several hospitals and regional control centres, each managing their resources as in a real mass casualty event.

References [1] [2] [3] [4]

Triola, M.M., et al., An XML standard for virtual patients: exchanging case-based simulations in medical education. AMIA Annu Symp Proc, 2007: p. 741-5. http://opensimulator.org/wiki and http://education.secondlife.com/ (both accessed on 14 July 2010). http://www.ambulancehart.org.uk/about_hart/ (accessed on 14 Oct 2010). http://www.emergotrain.com (accessed on 14 Oct 2010).

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-653

653

Terahertz Imaging of Biological Tissues Priyamvada TEWARIa,b, Zachary. D. TAYLORa,b,1, David BENNETTa,b , Rahul. S. SINGHa,c,g, Martin O. CULJATa,b,c,g, Colin P. KEALEYc, Jean Pierre HUBSCHMANd, Shane WHITEe, Alistair COCHRANf, Elliott . R. BROWNg, Warren S. GRUNDFESTa,b,c, a Center for Advanced Surgical and Interventional Technology (CASIT) b Biomedical Engineering IDP, cDepartment of Surgery, dJules Stein Eye Institute, e School of Dentistry, fDepartment of Pathology, University of California, Los Angeles; g Dept. Electrical and Computer Engineering, University of California Santa Barbara

Abstract. A reflective THz imaging system sensitive to small variations in water concentrations has been developed. Biological tissues such as skin, eyes and teeth were imaged to ascertain the systems response to tissue hydration. Difference in water concentrations translated to contrast in the THz images. Contrast was also seen in THz images of skin cancer and burns suggesting the potential diagnostic capability of THz imaging system in clinical settings. All specimens analyzed were freshly excised ex-vivo tissues. These encouraging preliminary results have motivated us to explore the in vivo potential of our imaging system. Keywords. Terahertz imaging, hydration, skin burns, dental, corneas

1. Introduction The terahertz band occupies the part of the electromagnetic spectrum between infrared and microwaves spanning the frequency range 0.3 THz – 3 THz. Properties like strong absorption and reflection by water, non-ionizing photon energy, low loss transmission through common fabrics, and reduced scatter compared to infrared and better resolution than microwave have motivated the exploration of THz radiation for medical imaging [1-2]. Studies have reported the capability of THz in identifying inflamed skin regions, cancers, burns, stratum corneum hydration profiling, and early detection of dental caries [3-4]. These findings coupled with recent advances in instrumentation have motivated the development of medical THz imaging systems [5-6]. From a clinical standpoint, reflective imaging systems are more practical since they allow in vivo assessment of body surface abnormalities and may provide contrast currently unavailable with existing medical imaging modalities. We have developed a reflective THz imaging system that operates at a center frequency of 0.525 THz with a bandwidth of 0.125 THz [7]. Previous results obtained by our group suggest the capability of our system in imaging biological tissues and generating a contrast between areas differing in water concentrations [7-8]. This paper provides a summary of preliminary experiments done on various biological samples to explore the sensitivity of our imaging to hydration gradients in a range of biological tissues. 1

Corresponding Author: Zachary. D. Taylor, [email protected]

654

P. Tewari et al. / Terahertz Imaging of Biological Tissues

2. Results 2.1. Skin Grafts Skin grafts play an important role in the closure of large wounds, and in reconstructive surgery. The thickness and type of skin grafts is dictated by procedural requirements. Hydration state assessment of skin grafts before and after implantation can be useful in monitoring the success of functional and cosmetic assimilation [9]. To gauge the ability of our system to assess hydration in skin grafts, different layers of skin were imaged. Porcine skin was obtained and sliced into grafts of varying thicknesses using a dermatome. Specimens were placed face-down onto a polypropylene sample mount, and imaged using the THz system (Figure 1).

Figure 1. (Left) Representative skin graft data scan, with metal tape and specimen’s labeled, and (Right) average THz reflectivity as function of skin thickness

THz reflectivity increased monotonically with increasing graft thickness. It has been experimentally shown that the deeper dermal layer of skin is more hydrated than the superficial epidermis [10]. Our findings are consistent with these earlier studies and demonstrate the ability of our system to detect small differences in water content between different layers of the skin. These findings suggest the utility of THz imaging as a method for assessing pre- and post-operative graft hydration levels, potentially alerting clinicians to at-risk grafts prior to failure. 2.2. Skin Burns 1.25 million patients are treated for burns annually in the United States with burn injuries being the fifth leading cause of injury-related deaths [11]. Since accurate assessment of burn extent and depth is important for making clinical decisions, the need for a noninvasive imaging tool is immense. For evaluating the capability of our imaging system in visualizing skin burns, fresh porcine skin was obtained and sectioned into pieces 1” x 1”. A brass brand in the shape of cross was heated to 350°C and pressed against the skin for 5-7 sec to induce a full thickness burn. The sample was then mounted on a polypropylene mount and imaged.

Figure 2. (Left) visible picture, and (Right) THz image of full thickness burn on porcine skin

P. Tewari et al. / Terahertz Imaging of Biological Tissues

655

The cross-shaped burnt area is clearly visible in the THz image. A contrast is obtained between the burnt and unburnt area where lighter shades of gray correspond to areas of higher reflectivity. Burning involves local evaporation of water making the burnt areas relatively dehydrated as compared to the surrounding normal areas in skin. There was a 98.4% drop in reflectivity between burnt and unburnt area. The SNR of the image was calculated as 17 dB. Scattering due to skin surface roughness contributed to the overall variance seen in the image. This result is in accordance with a previous published experimental result [12]. 2.3. Skin Cancer Tumors are associated with higher water content than normal tissues. Since THz imaging is so sensitive to water and changes in absorption and refractive indices, which are found to vary between normal and malignant tissues, a difference in THz reflection is expected to generate contrast [4,13]. A skin scalp biopsy sample with recurrent melanoma was procured with the permission of UCLA Institutional Review board. The specimen was placed on a polypropylene mount and imaged.

Figure 3. (Left) Skin scalp biopsy specimen, and (Right) THz image

The red outline in the THz image denotes the outer boundary of the sample delineating it from the polypropylene mount. The sample gives an overall lower THz reflectivity as compared to the mount. Further contrast is seen within the specimen (Figure 3). The dark pigmented lesion in the center of the scalp is found to be relatively less reflective to THz as compared to the surrounding scalp. Areas of higher reflectivity (outlined in red) close to the lesion in normal skin boundary are also observed in the THz image. In order to demonstrate the clinical utility of terahertz technology, we are working closely with pathologists to develop procedures that map histological findings to terahertz images. 2.4. Tooth Enamel and Dentin A horizontal cross-section of a molar was studied; this comprised of a ring of surface enamel encircling deeper dentin. The tooth was fully hydrated and embedded in epoxy resin for imaging (Figure 4). Enamel and dentin contrasted markedly. The yellow region in the 2-D image corresponds to the thin outer ring of enamel surrounding the dentin. Darker regions within the red area can be mapped to thinner areas of circumpulpal dentin. Even though the shape of the underlying subsurface pulp horn shape in the dentin was not exactly replicated in the image, different dentin regions in the sample can be differentiated. The THz imaging contrast between enamel and dentine is due to a difference in refractive indices. Higher THz reflectivity is obtained from enamel (3.1), which has a higher refractive index than dentine (2.6) [14]. The THz image obtained delineates enamel from dentin suggesting that our system can be used to image the dentino-enamel

656

P. Tewari et al. / Terahertz Imaging of Biological Tissues

junction. Previously most of THz images have been obtained from dry samples. The tooth sample was fully hydrated at the time of embedding so as to simulate inherent the moistness of teeth in vivo.

Figure 4. (Left) Tooth sample encased in epoxy, and (Right) corresponding THz image

2.5. Cornea Hydration profiling of cornea can help in early detection of corneal edema, trauma, inflammation, corneal endothelial cell pathology and monitoring laser ablation rate during corrective surgeries [15]. The following experiment was done to see whether THz imaging could play a role in hydration detection and monitoring in cornea. Pig eyes were obtained and corneal flaps ~ 130 µm were sliced from the eyes using a microkeratome. Flaps were placed on the polypropylene mount with an Al strip on one side to calibrate the system. The height of stage was adjusted to point of maximum reflectance on the top surface. Setting the maximum as the origin, line scans were taken across the corneal flaps with a step size of 0.5 mm at a bias of 30 V. Each line scan crossed over polypropylene, porcine cornea, and aluminum (Figure 5).

Figure 5. (Left) 2-D image of cornea, (Middle) horizontal cut, and (Right) vertical cut through middle of cornea

A radially varying hydration profile is evident from line scans of the corneal flap. The point of maximum reflectance was found to be coincident with the center of cornea. Reflectance decreased from the origin to the edges, suggesting that center of cornea was more hydrated than the edges. The edges were found to have similar reflectivites, indicating symmetry in the corneal structure [16]. Though the results are obtained by forcing a curved corneal flap onto a flat scanning surface they are promising, and in concert with previous experimental observations. We expect improved accuracy with the development of systems for imaging curved surfaces.

3. Conclusions/Future direction Preliminary results indicate that a reflective THz imaging system is able to generate a contrast in biological specimens between regions differing in water concentrations. This can be used not only in distinguishing between tissues but also in identifying diseased states. The successful imaging of ex-vivo tissues like skin and eyes along with the non-invasive nature of THz radiation opens up many potential clinical uses of THz imaging technology. The next step is repeating some of the above listed experiments on

P. Tewari et al. / Terahertz Imaging of Biological Tissues

657

live animals. The Animal Research Committee at UCLA has approved our project. In vivo THz imaging is currently underway.

4. Acknowledgements The authors would like to thank CASIT collaborators in UCSB for their support of this project, along with CASIT collaborators at UCLA: Dr. Jean-Louis Bourgeis, Dr. Benjamin Burt, Dr Matthew DeNicola and student researchers Jon Suen, Shijun Sung and Ashkan Maccabi. The authors most gratefully appreciate the funding provided by the Telemedicine and Advanced Technology Research Center (TATRC)/ Department of Defense under award W81XWH-09-Z-0017 and the National Science Foundation under grant number IHCS-801897.

References [1] [2] [3]

[4]

[5] [6] [7] [8] [9] [10] [11] [12] [13] [14]

[15] [16]

J.E Bjarnason, T.L.J Chan, A.W.M Lee, A Celis & E.R Brown, “Millimeter-wave, terahertz, and midinfrared transmission through common clothing,” Applied Physics Letters, 85(4) (2004), 519-521 P.H Siegel, “Terahertz technology in biology and medicine”, IEEE transactions on microwave theory and techniques, 52(10) (2004), 2438-2446, A.J Fitzgerald, E Berry, N.N Zinv’ev, S Homer-Vanniasinkam, R.E Miles, M Chamberlain & M.A Smith, “Catalogue of Human Tissue Optical Properties at Terahertz Frequencies ,” J. Bio. Phys. 129 (2003), 123-128 R.M Woodward, B.E Cole, V.P Wallace, R.J Pye, D.D Arnone, E.H Linfield & M Pepper, “Terahertz pulse imaging in reflection geometry of human skin cancer and skin tissue,” Phys. Med. Biol. 47, 38533863 (2002). D.R. Grischkowsky & DM Mittleman, “Introduction. In: Mittleman DM, editor. Sensing with terahertz radiation. Berlin, Heidelberg, New York: Springer ( 2003), 1-38 Z.D. Taylor, R.S. Singh, M.O. Culjat, J.Y. Suen, W.S. Grundfest, E.R Brown, “THz imaging based on water-concentration contrast,” Proc . of SPIE 6949 (2008) R.S. Singh, Z.D. Taylor, M.O. Culjat, W.S. Grundfest & E.R. Brown, “Towards THz Medical Imaging; Reflective Imaging of Animal Tissues,” MMVR 16 (2008) Z.D. Taylor, R.S. Singh, M.O. Culjat, J.Y. Suen, W.S. Grundfest, H Lee & E.R. Brown, “Reflective terahertz imaging of porcine skin burns,” Optice letters 33(11) ( 2008) D.Ratner, “Skin grafting,” Seminars in cutaneous mediucine and surgery 22(4) (2003), 295-305 K. Martin, "In vivo measurements of water in skin by Near-Infrared Reflectance," Applied spectroscopy 52(7) (1998), 1001-1007 Burn incidence and treatment in the US: 2007 fact sheet. D.M .Mittleman, M Gupta, R Neelamani, R.G. Baraniuk, J.V. Rudd & M Koch, “Recent advances in THz imaging,” Applied Physics B: Lasers and Optics 68 (1999), 1085-1094 R.M. Woodward, B Cole, Wallace V.P, D.D. Arnone, Pye R, Linfield E.H, Pepper M & Davies A.G. “Terahertz pulse imaging of in-vitro basal cell carcinoma samples.” CLEO (2001); 329-330. D.A. Crawley, C. Longbottom, B.E. Cole, C.M. Ciesla, D. Arnone, V.P. Wallace & M. Pepper, "Terahertz pulse imaging: A pilot study of potential applications in dentistry," Caries Research 37 (2003), 352-359 S Mishima, “Corneal thickness,” Surv Ophthalmol 13 (2) (1968),57-96 R. S. Singh, P. Tewari, J.L. Bourges, J.P. Hubschman, D.B. Bennett, Z.D. Taylor, H. Lee, E.R. Brown, W.S. Grundfest, M.O. Culjat, "Terahertz Sensing of Corneal Hydration," Proc. of IEEE EMBS 2010

658

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-658

Quantifying Surgeons’ Vigilance during Laparoscopic Operations Using Eyegaze Tracking Geoffrey TIEN a,1 , Bin ZHENG b and M. Stella ATKINS a a School of Computing Science, Simon Fraser University, Canada b Department of Surgery, University of British Columbia, Canada Abstract. The vigilance of surgeons while operating is an important consideration for patient safety. Using a lightweight mobile eyegaze tracker, we can objectively observe and quantify a surgeon’s vigilance measured as the frequency and duration of time spent gazing at an anaesthesia monitor displaying various patient vital signs. Expert surgeons and training surgical residents had their eyegaze recorded while performing a mock partial cholecystectomy on a computer simulator. Results show that experts glanced at the patient vital signs more than the residents, indicating a higher level of surgical vigilance. Keywords. Laparoscopic surgery simulator, Head-mounted eyegaze tracker

Introduction Vigilance is the state of being watchful to avoid danger. In an operating room (OR) setting, surgical vigilance can be extended to encompass awareness of potential dangers to a patient. A high level of mental judgment ability inclusive of awareness of patient condition is an important part of ensuring patient safety [1,5,6]. When observing surgical performance in the OR, it is noticeable that the senior surgeon usually keenly detects signs that may concern patient safety. But little is known whether vigilance is associated with a surgeon’s competency in performing the surgical procedure. The ﬁrst goal of this study is to examine the relationship between vigilance and surgical skills. To achieve this ﬁrst goal, we asked surgeons with a wide range of surgical experience to perform a laparoscopic procedure in a simulated environment. We chose laparoscopy due to a simple fact that a sufﬁcient level of vigilance can be more difﬁcult to maintain in a laparoscopic setting where only a part of the surgical ﬁeld is visible by a video display from an endoscope, and additional mental processing is needed to maintain orientation of the patient anatomy, further compounded by the increased difﬁculty of precisely controlling the laparoscopic instruments compared to open surgery. A problem with observing the vigilance of surgeons is the lack of a method of measuring this skill. To this end we propose to use eyegaze tracking as an approach to objec1 Corresponding Author: Geoffrey Tien, School of Computing Science, Simon Fraser University, 8888 University Drive, Burnaby, B.C. Canada; E-mail: [email protected].

G. Tien et al. / Quantifying Surgeons’ Vigilance During Laparoscopic Operations

659

tively quantify surgical vigilance. Our second goal is to prove the value of using eyetracking in a surgical context, based on our earlier work showing how the eyegaze of novices and experts differ in a virtual laparoscopic training environment [3]. We hypothesize that as surgical experience increases, cognitive effort in performing the primary surgical task will decrease, hence freeing attentional resources to observe the patient condition. In this study we aim to track surgeons’ eye movements during a mock laparoscopic procedure and to use this as a measure of awareness of changes in a patient’s condition displayed on a simulated anaesthesia monitor.

1. Method 1.1. Apparatus The study was conducted in the surgical skills training lab at the Centre of Excellence for Surgical Education and Innovation (CESEI) of the University of British Columbia (UBC). Two high-ﬁdelity simulators were used to create patient scenarios. The ﬁrst, a SurgicalSim VR manufactured by Medical Education Technologies, Inc. (METI) provided the main visual and tactile interface of the apparatus. This PC-based simulator includes a set of slender tools and a foot pedal to mimic the form and function of laparoscopic instruments, and a 17" LCD monitor as the simulated laparoscopic display. SurgicalSim was used to create a virtual surgical training environment for our participants to perform a partial cholecystectomy. A separate MacOS-based METI Emergency Care Simulator (ECS) includes a lifesized pneumatically controlled mannequin, whose simulated vital signs such as heart rate (with audible beep), blood pressure, and blood oxygen saturation were displayed on a 15" LCD monitor placed to the right side of the main SurgicalSim VR display. It is important to note that the ECS and SurgicalSim VR systems were placed closed to each other creating a sensation for the participant that they were operating on one single patient; however, the ECS and SurgicalSim VR do not communicate with one another. Finally, eyegaze tracking was accomplished by a head-mounted PT-Mini system manufactured by Locarna Systems, Inc. The PT-Mini headgear consists of two linked video cameras—one aimed at the wearer’s eye, and one facing forward to capture the scene relative to the wearer’s head. The two video feeds were saved to a portable notebook computer for post-processing. The components of the experimental apparatus are shown in Figure 1. 1.2. Task The experimental task required a surgeon to hold a grasper and a monocautery hook to dissect the gall bladder from the liver. A foot pedal placed on the ground controlled cautery. For each participant, the partial cholecystectomy exercise was performed on the SurgicalSim VR under two different patient conditions. One patient presented a stable heartbeat controlled by ECS, while the other patient’s heartbeat became slightly erratic at set intervals. Because ECS and SurgicalSim VR are unlinked, changes in patient condition on ECS do not alter the scene on SurgicalSim VR.

660

G. Tien et al. / Quantifying Surgeons’ Vigilance During Laparoscopic Operations

(a)

(b)

Figure 1. (a) METI SurgicalSim VR and ECS, (b) Locarna PT-Mini headgear.

1.3. Participants Participants included surgical residents, laparoscopic fellows, and attending surgeons from the surgery department at UBC. A pre-test questionnaire was administered to gather demographic data and to measure their laparoscopic surgical experience score as detailed by Zheng et al [6]. 1.4. Procedure After signing their consent to participate and completing the pre-test, each participant was allowed to complete the simulated partial cholecystectomy once without the patient vitals to learn the characteristics of operating the SurgicalSim VR. Each participant then put on the Locarna headgear and was guided through a short calibration procedure to ensure that his eye could be reliably tracked across the scene camera’s ﬁeld of view. Participants then performed the partial cholecystectomy task once for each of the two patient conditions, for a total of 2 trials. The patient histories were presented on a printed sheet of paper before each trial. The order of the patient conditions was counterbalanced across participants. 1.5. Data processing and analysis Eyegaze was recorded over the duration of each trial of the cholecystectomy task and analyzed using Locarna’s Pictus Eyegaze Calculation software. Eyegaze ﬁxation detection was done using a dispersion threshold algorithm with a minimum duration of 100 ms and a maximum dispersion of 40 pixels relative to the captured video scene frame. Results were analyzed in a 2×2 ANOVA using statistical software from SPSS Inc., where P<0.05 is considered signiﬁcant.

2. Results 16 surgeons and medical students participated in this study. Based on the pre-test questionnaire, the participants were divided into 8 novices (3 female, aged 25-49, mean age 32) and 8 experts (all male, aged 34-57, mean age 39), based on their years of training

G. Tien et al. / Quantifying Surgeons’ Vigilance During Laparoscopic Operations

661

Table 1. Number of glances made to the vitals display. Novice Mean, std.dev Min, median, max

Stable

Unstable

Expert

1.3 ± 1.8 0, 0.5, 5

2.1 ± 3.4 0, 0, 9

Mean, std.dev Min, median, max

Stable

Unstable

1.3 ± 3.5 0, 0, 10

3.4 ± 6.1 0, 1, 18

Table 2. Eyegaze ﬁxation times on the surgical and vitals displays. Total time (s)

Stable Surgical (% total)

Vitals (% total)

Total time (s)

Unstable Surgical (% total)

Vitals (% total)

Novice Mean Std.dev Min

162 48.8 72

151 (94.4) 42.8 70

0.9 (0.4) 1.6 0.0

176 62.4 78

169 (95.9) 59.1 77

1.6 (0.8) 3.0 0.0

Median Max

168 219

157 212

0.2 4.5

177 262

168 254

0.0 8.6

Expert Mean Std.dev Min Median

186 49.6 143 174

176 (94.5) 41.5 128 167

0.8 (0.3) 2.3 0.0 0.0

201 66.0 114 208

185 (92.9) 52.0 104 200

3.2 (1.1) 7.8 0.0 0.3

Max

303

269

6.5

318

243

22.5

and their surgical experience scores. Novices had around 6 months of laparoscopic training, whereas experts had at least 2 years training and had performed many laparoscopic cases as the primary surgeon. Aggregated eyegaze saccade results appear in Table 1, and the eyegaze ﬁxation results are shown in Table 2, where the ﬁrst column for each condition describes the total recording time; the second and third columns describe only the time captured in ﬁxations. For the stable patient condition, novices spent approximately the same mean duration of time looking at the anaesthesia monitor (0.9 s) as experts (0.8 s). Novices and experts also glanced at the patient vitals the same number of times during the operation on the stable patient (1.3 times). When operating on the unstable patient, novices spent less time looking at the anaesthesia monitor (1.6 s) compared to experts (3.2s) and did so by looking over less frequently (2.1 times vs. 3.4 times). Also, only 3 novices glanced at the vitals screen of the unstable patient whereas 5 experts checked the patient vitals. However, with our relatively small and variable participant sample, we are unable to associate these results with statistical signiﬁcance.

3. Discussion The power of the results obtained in this study is limited somewhat by the small sample size. Secondly, in an OR setting, the anaesthesia monitor is often oriented on a plane not visible to the primary surgeon. Nevertheless, we found that surgeons with extensive laparoscopic OR experience still glanced at the unstable patient’s vital signs when the display was made available to see. The situation is different for the stable patient, where only one expert looked at the vital signs. The experts who had performed a high num-

662

G. Tien et al. / Quantifying Surgeons’ Vigilance During Laparoscopic Operations

ber of OR cases noted that the regular audible beep of the simulated patient’s heart rate was sufﬁcient to conclude that the patient was stable, and the operation could proceed as normal. In contrast, novices who had not yet mastered the manual skills for laparoscopic operations likely had most of their mental resources occupied by the primary task, leaving few resources available to monitor the patient condition through auditory or visual channels. Such a compromise of performance easily occurs under high workload [4,2]. Despite extensive OR experience and heightened ability to match the audible heart rate to condition, experts still tended to visually reafﬁrm their knowledge when the anaesthesia monitor was available, demonstrating that our chosen experimental setup could still distinguish expert and novice behaviours with respect to patient safety. Furthermore, any difference in eye movement characteristics between the two patient conditions within a single group can be safely attributed to noticeable changes in the patient vitals through audio and visual channels; since ECS and SurgicalSim VR are not linked, the actual cholecystectomy task on SurgicalSim VR is identical across both conditions. With these promising early results, we will continue to recruit more participants with the aim of observing statistically signiﬁcant differences in the eyegaze measures. Finally, we would like to correlate eyetracking data with surgical performance data and mental workload assessment. Future analysis will include these data dimensions. 4. Conclusion Awareness of patient condition during laparoscopic surgery is an important skill in a procedure which demands intense focus on a laparoscopic display. Expert surgeons were more aware of changes in patient condition, and were more able to effectively distribute their attention between two surgical and anaesthetic displays compared to novices who concentrated on only the surgical display and were inattentive to the patient condition. Acknowledgements The Locarna eyegaze tracker was purchased with funds from the Canadian Natural Sciences and Engineering Research Council grant. References [1] R. Aggarwal, J. Leong, D. Leff, O. Warren, G.-Z. Yang, A. Darzi. New technologies for the surgical curriculum, World Journal of Surgery 32:2 (2008), 213–216. [2] K. E. Hsu, F.-Y. Man, R. A. Gizicki, L. S. Feldman, G. M. Fried. Experienced surgeons can do more than one thing at a time: effect of distraction on performance of a simple laparoscopic and cognitive task by experienced and novice surgeons, Surgical Endoscopy 22:1 (2007), 196–201. [3] B. Law, M. S. Atkins, A. Kirkpatrick, A. Lomax, C. L. MacKenzie. Eye gaze patterns differentiate skill in a virtual laparoscopic training environment, Proceedings of Eye Tracking Research and Applications (2004), 41–47. [4] R. G. Loeb. Monitor surveillance and vigilance of anesthesia residents, Anaesthesiology 80 (1994), 527– 533. [5] D. Stefanidis, M. Scerbo, J. J. Korndorffer, D. Scott. Redeﬁning simulator proﬁciency using automaticity theory, The American Journal of Surgery 193:4 (2007), 502–506. [6] B. Zheng, M. A. Cassera, D. V. Martinec, G. O. Spaun, L. L. Swanström. Measuring mental workload during the performance of advanced laparoscopic tasks, Surgical Endoscopy 24:1 (2009), 45–50.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-663

663

Modeling of Interaction between a Three-Fingered Surgical Grasper and Human Spleen Mojdeh TIREHDASTa,b, Alireza MIRBAGHERIa,b, Mohsen ASGHARIa, Farzam FARAHMANDa,b,1 a

School of Mechanical Engineering, Sharif University of Technology, Tehran, Iran b RCSTIM, Tehran University of Medical Sciences, Tehran, Iran

Abstract. The aim of this study was to develop a more sophisticated model of the spleen tissue and investigate its interactions with a three-fingered laparoscopic grasper. The spleen tissue, modeled as a hyper viscoelastic material, was subjected to external loadings, imposed by rigid grasping jaws. The tissue deformation as well as the sliding occurrence between tissue and jaws was investigated using nonlinear finite element method. Results indicated that a grasping configuration which aimed a sufficiently large piece of spleen with small radius of curvature was more successful for effective grasping. The trends and magnitudes of the tooltissue interaction forces obtained during effective and ineffective grasping were quite different. A force with progressively increasing trend toward a high magnitude was found to be indicative of effective and safe grasping. This finding might be used to predict the effectiveness of different grasping configurations and sliding thresholds during spleen and other soft organs surgery. Keywords. Finite element method, soft tissue grasping, sliding

Introduction Laparoscopic surgery procedures have become common clinical practices in recent years as a result of the rising trend towards more minimally invasive surgery (MIS). They are thought to provide better clinical results and less overall costs through shorter hospital stays, shorter recovery times, and reduced need for repeated surgery [1]. However, considering the loss of direct visual and tactile information during laparoscopic surgery, surgery trainees need to be well prepared for the hand-eye coordination and complex gestures required. Surgical simulators, incorporating a virtual reality based interactive graphical environment and a force feedback device, provide a safe and efficient environment for the training process [1, 2]. They enable the trainee to repeat the surgical procedure unlimitedly, while the system parameters have been set to simulate different situations, including complications. They could be also used for pre-planning of surgical operations.

1 Corresponding Author: School of Mechanical Engineering, Sharif University of Technology, and RCSTIM, Tehran University of Medical Sciences, Tehran, Iran; E-mail:[email protected]

664

M. Tirehdast et al. / Interaction Between a Three-Fingered Surgical Grasper and Human Spleen

A key problem in development of an effective surgical simulator is the realistic simulation of the mechanical interactions between surgical tools and body organs. Assuming the deformations of the surgical tool to be negligible, this would need accurate modeling of the mechanical properties of the soft organs, as well as the appropriate implementation of the changing boundary and force conditions. There have been lots of research studies concerning the mechanical modeling of soft tissues, e.g., brain, liver and kidney [3-5]. Also the interactive simulating of tool- tissue interactions, e.g., needle insertion and tissue cutting has been addressed by several investigators [1, 6-8]. However, a very limited number of studies are available in the literature concerning the modeling of spleen tissue and simulation of grasping procedure. Davies et al. in 2002 fitted an exponential curve, based on a Mooney-Rivlin formulation of strain energy function, to the experimental data to develop a mathematical model for the spleen tissue suitable for laparoscopic surgery [9]. Alsaraira et al. [10], on the other hand, used finite element method to simulate the tool-tissue interactions during gripping. This study attempted to develop a more sophisticated model for the spleen tissue and investigate its interactions with a three-fingered laparoscopic grasper, especially designed for manipulation of large body organs [11].

Method A 3-D geometrical model of spleen was developed in SOLIDWORKS environment using direct measurements conducted on a normal specimen during an open surgery operation. It was then meshed into 27342 tetrahedral elements, each of four 3 degrees of freedom nodes, and analyzed using the ABAQUS finite element software (Fig 1). In order to avoid extra computations, we used biased seeding so that finer elements were used in regions with large strains.

Figure 1. The 3D model of spleen and three grasping jaws in ABAQUS

The spleen tissue was considered to be a deformable isotropic, homogeneous, incompressible material with hyper viscoelastic properties so that its time-dependent

M. Tirehdast et al. / Interaction Between a Three-Fingered Surgical Grasper and Human Spleen

665

non-linear stress-strain behavior under large deformations could be well addressed. A polynomial strain energy density function was considered to derive the constitutive equation needed to describe the complicated mechanical behavior of the tissue [12]. The parameters of the model were obtained by fitting to the stress-strain relationships reported by Davies et al. [9]. The surgical grasper was considered to include three jaws that interact and apply loads to the surface of the soft tissue. The grasper jaws were modeled as rigid bodies considering their negligible deformation in comparison with the spleen tissue. The three jaws were placed 120ᵒ apart around the spleen and were positioned parallel to each other during opening and closing functions, considering the design of the grasper [11]. The mechanical interaction between the jaws and the spleen was modeled using general contact elements of ABAQUS, so that the varying boundary conditions, as well as the occurrence of sliding between jaws and tissue could be addressed. These elements were preferred to other possible options due to imposing least limitation to surface selection, modeling finite sliding condition and minimizing the penetration between contacting objects. The coefficient of friction between jaws and spleen was assumed to be 0.15. In order to avoid rigid body motion of the spleen, the nodes located at its concave side were fully constrained to represent the effect of connective surrounding tissues (Fig 1). In order to facilitate the recognition of contact by ABAQUS software, a small gap was inserted between the tissue and rigid jaw. At the first step of analysis a prescribed displacement was applied to instrument jaws to just touch the tissue. Tissue-tool interaction was investigated at the second step. The dynamic, explicit procedure was chosen to solve the problem. This procedure is best suited for our analysis since in some situations the soft tissue escapes before jaws can grasp it, so the static problem changes into a dynamic problem. External loading was applied to the tissue in the form of the displacement of the jaws reference points. This model of tool- tissue interaction was analyzed for different initial positions of instrument relative to spleen in order to investigate the effectiveness of the grasping configuration.

Results Sample illustrations of the analysis of the model, starting from different initial grasping configurations, are shwon in figures 2 and 3. In general, the grasping configurations which aimed a sufficiently large piece of spleen with small radius of curvature were more successful for effective grasping (Fig 2). When grasping of small pieces of the spleen or with large radius of curvature was attempted, the tissue often slid out and escaped from being bit by the jaws (Fig 3).

666

M. Tirehdast et al. / Interaction Between a Three-Fingered Surgical Grasper and Human Spleen

Figure 2. A sample illustration of tool-tissue configuration with effective grasping

(a)

(b)

(c) Figure 3. A sample illustration of the sequence of sliding out of a tool-tissue configuration with ineffective grasping.

The tool-tissue interaction forces imposed to the jaws during grasping for sample effective and ineffective grasping conditions of figures 2 and 3 are illustrated in figure 4 and 5, respectively. Each diagram includes three curves for the resultant force imposed to the three jaws of grasper. By comparing the force results, it could be observed that much larger tool-tissue interaction forces occurred for effective grasping

M. Tirehdast et al. / Interaction Between a Three-Fingered Surgical Grasper and Human Spleen

667

condition than ineffective grasping. In particular, for the sample effective and ineffective grasping conditions examined, the maximum resultant forces were 12N and only 4.5 N, respectively. Moreover, the trend of the force diagrams for the effective and ineffective grasping conditions were quite different. For effective grasping, the tool-tissue interaction forces increased progressively with increasing displacement (closure) of the grasper jaws. For ineffective grasping, however, these forces increased initially to their maximum magnitudes, corresponding to a particular configuration of the grasper jaws, and then decreased due to sliding out and escaping from being bit by the jaws.

Figure 4. The tool-tissue interaction forces imposed on the three jaws during effective grasping

Figure 5. The tool-tissue interaction forces imposed on the three jaws during ineffective grasping

Discussion In this paper, a hyper viscoelastic model of spleen was developed and its interaction with a three fingered surgical grasper was investigated. The mathematical model of the spleen presented in this work is more sophisticated than those of previous studies [9]. However, the most significant feature of our work is providing a framework for

668

M. Tirehdast et al. / Interaction Between a Three-Fingered Surgical Grasper and Human Spleen

realistic simulation of large tissues grasping procedure in which the occurrence of sliding between tissue and surgical instrument is incorporated. Such presentation of the sliding event and its effects on the tool-tissue interaction has not been investigated in previous studies. The results of our study, concerning the different magnitudes and trends of the tool-tissue interaction forces imposed to the jaws during effective and ineffective grasping are consistent with the general sense. However, experimental studies are required to verify our results. In particular, the magnitudes of forces obtained in our study are largely dependent upon our assumptions for the mechanical properties of the spleen tissue and the coefficient of friction between jaws and spleen. Detailed experimental investigations are needed to provide accurate data necessary for calibration of the model, as well as validation of its result. Work is now in progress to develop an appropriate set up for such experiments. The results of our model, after proper validation, might be used to train surgeons to predict the effectiveness of the grasping configuration and the sliding thresholds during soft organs, e.g., spleen, surgery. Such information is valuable for surgeons since an effective grasping, by which the soft organ is hold fixed, can greatly affect the outcome of the surgical operation. For instance, sliding out of the spleen during spleenectomy might cause the surgeon to cut or damage the surrounding tissues by mistake, resulting in surgical complications. The computational time and cost is a major concern for our model to be incorporated in a surgical simulator system and be utilized for training the surgery trainees. Considering the complexity of the hyper visco elastic model of the tissue material and the high nonlinearity imposed by the general contact elements, the only solution for a real-time response is to solve the model for plenty of different conditions and construct a looking up table. By using modern intelligent algorithms, e.g., neural network methodology, this could be inherently used for a real time simulation of the surgical operation. Work is now in progress to develop such looking up table and implement the neural network algorithm to provide real-time response by the model.

Acknowledgment This study was supported by grant No. 86/48490 from Hi-tech Industries Projects of I.R. Iran. The help of the staff of the Robotic Surgery Lab of RCSTIM is greatly appreciated.

References [1] [2]

[3] [4]

E. Basafa, F. Farahmand, Real-time simulation of the nonlinear visco-elastic deformations of soft tissues, International Journal of Computer Assisted Radiology and Surgery (2010), 1-11. E. Basafa, M. Sheikholeslami, A. Mirbagheri, F. Farahmand, G.R. Vossoughi, Design and implementation of series elastic actuators for a haptic laparoscopic device, Engineering in Medicine and Biology Society, 2009. EMBC 2009. Annual International Conference of the IEEE, (2009), 60546057. K. Miller, Constitutive model of brain tissue suitable for finite element analysis of surgical procedures, Journal of Biomechanics 32 (1999), 531-537. J.D. Brown, J. Rosen, M.N. Sinanan, B. Hannaford, In vivo and postmortem compressive properties of porcine abdominal organs, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 2878, (2003) 238-245.

M. Tirehdast et al. / Interaction Between a Three-Fingered Surgical Grasper and Human Spleen [5]

669

M. Farshad, M. Barbezat, P. FluÌˆeler, F. Schmidlin, P. Graber, P. Niederer, Material characterization of the pig kidney in relation with the biomechanical analysis of renal trauma, Journal of Biomechanics 32 (1999), 417-425. [6] H.W. Nienhuys, A.F. Van Der Stappen, A computational technique for interactive needle insertions in 3D nonlinear material, Proceedings - IEEE International Conference on Robotics and Automation, vol. 2004, New Orleans, LA, (2004), 2061-2067. [7] J.T. Hing, A.D. Brooks, J.P. Desai, Reality-based needle insertion simulation for haptic feedback in prostate brachytherapy, Proceedings - IEEE International Conference on Robotics and Automation, vol. 2006, Orlando, FL, (2006), 619-624. [8] T. Chanthasopeephan, J.P. Desai, A.C.W. Lau, Measuring Forces in Liver Cutting: New Equipment and Experimental Results, Annals of Biomedical Engineering 31 (2003), 1372-1382. [9] P.J. Davies, F.J. Carter, A. Cuschieri, Mathematical modelling for keyhole surgery simulations: A biomechanical model for spleen tissue, IMA Journal of Applied Mathematics (Institute of Mathematics and Its Applications) 67 (2002), 41-67. [10] A. Alsaraira, I. Brown, R. McColl, F. Lim, Instrument-tissue segment interaction using finite element modeling, Conference proceedings : ... Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Conference 2007 (2007), 2760-2763. [11] A. Mirbagheri, M. Yahyazadehfar, F. Farahmand, Conceptual Design of Novel Laparoscopic instrument for Manipulation of Large Internal Organs, Proceedings of 5th ASME Frontiers in Biomedical Devices Conference (BioMed2010), Newport Beach, California, USA, 2010. [12] M. Tirehdast, A. Mirbagheri, F. Farahmand, M. Asghari, Finite element modeling of spleen tissue to analyze its interaction with a laparoscopic surgery instrument, Proceedings of 10th Biennial ASME Conference on Engineering Systems Design and Analysis, (ESDA2010), Istanbul, Turkey, 2010.

670

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-670

Quantizing the Void: Extending Web3D for Space-Filling Haptic Meshes Sebastian ULLRICH a,1 , Torsten KUHLEN a Nicholas F. POLYS b Daniel EVESTEDT c Michael ARATOW d and Nigel W. JOHN e a RWTH Aachen University, Germany b Virginia Polytechnic Institute and State University, United States c SenseGraphics, Sweden d Web3D Consortium, United States e Bangor University, United Kingdom Abstract. In this paper we summarize the progress of the Web3D scene graph model, and associated standards, speciﬁcally Extensible 3D (X3D) in the domain of medical simulation. Historically, the Web3D nodesets have focused on the representation and rendering of point, line or surface geometry. More recently, signiﬁcant progress in X3D Volume rendering has been made available through the co-operative DICOM work item, n-Dimensional Presentation States. However, here we outline the need for a standard for simulation meshes and review several related approaches. As a result, we propose preliminary requirements for a simulation mesh standard and provide several use case scenarios of how Web3D and haptic technologies can aid the fulﬁllment of these requirements. We conclude with an X3D proposal to describe simulation meshes for soft (deformable) bodies. Keywords. X3D, Haptics, Data Sets, Simulation Meshes, Fidelity

1. Introduction Medical simulation plays an increasingly important role in the training of new clinicians and the pre-operative rehearsal of surgery and prosthetics. A virtual reality-based medical simulation system usually consists of some kind of hardware setup (e.g., computer with specialized input/output devices) and a simulation software engine and medical datasets [1]. The ﬁeld is burgeoning with research teams and companies providing various solutions to a common problem: an engine that loads, evaluates and renders some data in a tight, interactive loop. Haptic rendering is particularly important and presents requirements beyond surface or volume representations. Here, we focus on the crucial role of medical datasets in the simulation enterprise: for a speciﬁc simulation scenario, datasets of body regions or organs etc. need to be generated and represented. This can be very tedious work, how1 Corresponding

E-Mail Address: [email protected]

S. Ullrich et al. / Quantizing the Void: Extending Web3D for Space-Filling Haptic Meshes

671

ever [2,3]. There are basically three ways of acquiring new datasets: (1) buying commercially modeled or segmented datasets [4,5], (2) segment patient speciﬁc data from medical scans, or (3) generate models procedural by mathematical formulas and heuristics. A brief survey leads to the following three conclusions: (1) although highquality models are available, the choice is very limited and might not cover one’s speciﬁc requirements; (2) patient-speciﬁc material is (relatively) easy to acquire by medical scans and provides an ever increasing level of detail, however segmentation is a major ﬁeld of research and full automation in the foreseeable future is not achievable; (3) while shape variations of existing models are possible, completely new (ab initio) models cannot be accurately synthesized in most cases. In conclusion, (2) is the most popular approach but is also demanding on manpower for manual segmentation, mesh conversion, etc. Our long-term goal is to reduce this workload by specifying enabling standards to share and reuse medical datasets.

2. Methods & Materials 2.1. Web3D and MedX3D The International Standards Organization (ISO) standard for 3D graphics over the Internet is Extensible 3D (X3D), which is maintained and developed by the Web3D Consortium. The initiative of the Web3D Consortiums Medical Working Group is to specify and implement MedX3D–an extension to the open and royalty-free X3D standard to support advanced medical visualization functionality and medical data exchange [6]. This initiative has speciﬁed and demonstrated cross-platform volume rendering styles (i.e., transfer functions), segmentation and ontology support [7,8,9], and data import/export capabilities for interactive presentation. Expressing the presentation of medical data through the X3D scene graph provides signiﬁcant advances in functionality, interoperability and durability [10, 11]. There are both open-source and commercial implementations supporting the Volume Component and numerous X3D engines deployed as standalone applications and web-integrated plug-ins. Recent integration with X3DOM has also shown great potential for native X3D in HTML5 content [12]. Most directly, the demonstrated pipeline for DICOM data in X3D has led to a Digital Imaging and Communications in Medicine standard (DICOM) Work Item for n-Dimensional Presentation States [Base Standard]. Discussions have progressed across multiple modalities and application groups toward critical mass and baseline functionality in Working Group 11 [Display Function Standard]. For this work item, there are interests and requirements in many aspects of medical care and DICOM standardization. The current speciﬁcation is a signiﬁcant opportunity to provide broad and capable functionality for the health-care enterprise.

672

S. Ullrich et al. / Quantizing the Void: Extending Web3D for Space-Filling Haptic Meshes

2.2. Survey of Simulation Meshes Both surface meshes and voxel representations are speciﬁed as part of the X3D ISO standard and the MedX3D extension respectively. However, volumetric mass spring systems and Finite Element Method (FEM) approaches often require a space-ﬁlling mesh representation, e.g., a tetrahedral mesh. We survey generators and ﬁle formats for such meshes. There are two popular systems to extract volumes from surface meshes: Tetgen and NetGen [13,14]. While Tetgen primarily supports Delaunay tetrahedralization, the advancing front algorithm is implemented in NetGen. Although Delaunay is faster and thus allows prototyping with diﬀerent parameters, it can slightly modify the surface and so lead to false negatives or false positives. Other tools often integrate one (e.g., Salome) or both (e.g., Gmsh) of these approaches [15,16]. Recently, another mesh generator tool called SimBio-Vgrid has been released as an open source project [17]. It creates hexa- and tetrahedral meshes directly from medical scans by building an octree partition and applying marching simplices to construct volume elements [18]. Furthermore, the Visualization Toolkit (VTK) which compromises state-of-the-art visualization algorithms, and modeling techniques, has a generic ﬁle format that also supports tetrahedral meshes [19]. To conclude this section we provide an overview of ﬁle formats in table 1. 2.3. Fidelity in Medical Simulators An important issue with the deployment of Web3D to medical simulators is that of scalability (of the visualization, of the models, of the haptic feedback, and of the interaction and simulation) to handle diﬀerent platforms and diﬀerent data rates. Decisions often have to be made about gaining extra performance at the expense of the overall quality of the simulation. Historically, this has resulted in Web3D oﬀering a low-ﬁdelity solution for procedural training tools [20]. However, validation studies show that these tools can be eﬀectively used in the training process–as was demonstrated for the web-based lumbar puncture simulator (Fig. 1, left) [21]. The model used here was created from a data set of a female volunteer who went through a CT scanner in the crouched position that would normally be taken up by a patient about to undergo a lumbar puncture. The CT Table 1. Overview of existent and proprietary ﬁle formats for tetrahedra. Format

Description

Tetgen

collection of ﬁles for: nodes, tetrahedra, triangles, boundary edges and neighbors

NetGen Msh (Gmsh)

sections in one ﬁle for: nodes, tetrahedra, and triangles ASCII or binary, sections in one ﬁle for: node, elements, region names and additional data for nodes and elements

GMV (Vgrid)

sections in one ﬁle for: nodes, cells, faces and additional data per node, face or cell

Vtk

unstructured grid in the vtk data format supports tetrahedra cells, additionally each cell can have arbitrary scalar data

S. Ullrich et al. / Quantizing the Void: Extending Web3D for Space-Filling Haptic Meshes

673

Figure 1. Examples of ﬁdelity in needle puncture simulators. From left to right: low ﬁdelity web-based simulator for training lumbar puncture, high ﬁdelity with real-time ultrasound and puncture simulation (ImaGiNe-S), and high ﬁdelity VR-based regional anaesthesia simulation (RASim).

data was then manually segmented, slice by slice, with a triangulation process lastly being applied to create the ﬁnal mesh. More recently the ﬁdelity that can be achieved has vastly improved as medical simulations have beneﬁted from developments in hardware and the associated lowering in costs. One example is the ImaGiNe-S Imaging Guided Needle puncture Simulation (Fig. 1, middle) [22]. It uses the H3D API, which extends the X3D scene graph with support for Volume rendering and haptic interaction. The anatomy models are segmented from a CT data scan of a real patient. In this application it is important that the virtual patient exhibits movement due to respiration. The use of mesh models allows us to animate the anatomy to implement respiratory motion. This is achieved in real time. The models themselves were obtained using a semi-automatic segmentation tool that provided a number of novel features such as the ability to partially “rewind” iterative segmentation algorithms such as the level set method to correct errors and methods to deﬁne constraints for segmentation algorithms such as using segmentations of kidney and ribs to constrain a liver segmentation. The segmentation is still time consuming and has to be carried out as a pre-processing step. Validation of the ImaGiNe-S simulator has produced excellent results, particularly for liver biopsy [23]. Another example is the regional anesthesia simulator (RASim project) [24]. This simulator allows the training of palpation and needle interaction (Fig. 1, right) for RA procedures. Electric impulse stimulation, which is used in RA to navigate the needles tip close to nerve structures, is approximated by path ﬁnding and morphing algorithms. In order to acquire source images non-invasively, protocols for MRI and MRA without contrast agent have been used [25]. Unfortunately, similar to aforementioned project, the generation of datasets and the semi-automatic segmentation create a bottleneck and are very time consuming. Ongoing work is focused on increasing the ﬁdelity by deformable soft-tissue and plausible haptic feedback. The implementation of the project is utilizing the open source projects ViSTA VR Toolkit and SOFA [26,27]. Prototypes of the simulator have been successfully evaluated with two diﬀerent datasets in an user study with ten subjects, both residents and consultants [24]. The importance of diﬀering anatomies for training has been proven by the study.

674

S. Ullrich et al. / Quantizing the Void: Extending Web3D for Space-Filling Haptic Meshes

3. Results From the previous section, own experience and from related work we have identiﬁed the following requirements for simulation meshes in Table 2. Additionally, we deduced the following use case scenarios in Table 3. The X3D standard [28] covers already some of the requirements (e.g., surface geometry, volume data (DICOM) and LOD), is ISO certiﬁed and extensible. More concretely, it supports surface meshes (e.g., IndexedFaceSet, Sec. 13.3.6) and also has a RigidBodyPhysics component (Sec. 37). Subsequently, a SoftBodyPhysics component could be introduced to accommodate the simulation mesh. Similar to the RigidBodyPhysics component, the physics nodes should be separated from the visual representation. This allows to use for example a coarse simulation mesh and a higher resolution mesh for visualization. Sometimes (for development, veriﬁcation, comparison, etc.) it might be also useful to visualize diﬀerent physics output properties of the nodes of the mesh. In order to make these properties easier to share between surface and volume meshes maybe they can be speciﬁed per vertex with nodes that already exist. For example the FloatVertexAttribute can be stored directly in an IndexedFaceSet and from there can be used by a shader.

4. Conclusions/Discussion We have outlined the beneﬁts of a straightforward extension to an established standard and proposed a way to extend X3D to create a more powerful represenTable 2. Collected requirements for simulation meshes. Technical • Task-speciﬁc geometric representations (e.g., surface mesh, voxel grid, tetrahedral mesh, etc.) • Material properties for physics-simulation and haptics (e.g., mass, stiﬀness, young modulus, etc.) • Level-Of-Detail (LOD) hierarchies • Mapping between simulation representation and visual, haptic and collision detection model Interactivity

• Realistic motion of mesh models due to physiology processes, e.g., respiration • Deformation of mesh models, e.g., palpation • Support for topological changes, e.g., cutting

Usability • The ability to quickly use patient-speciﬁc data • Standardized ﬁle format for arbitrary medical simulation engines

Table 3. Exemplary use case scenarios and matching X3D proﬁles. Scenario

X3D proﬁle

Easy deployable Web-based simulator setup

Interactive (Annex C)

High-ﬁdelity simulator

Immersive (Annex E)

Data-exchange, e.g., between research teams

Interchange (Annex B)

S. Ullrich et al. / Quantizing the Void: Extending Web3D for Space-Filling Haptic Meshes

675

tation of soft-body haptic simulation meshes. The full value of such an interchange speciﬁcation is compounding across domains and applications, where benchmark and patient-speciﬁc data can be reliably shared and experienced. In our proposed paradigm, a simulation scenario can be described in a structured data ﬁle format that enables the exchange of models from an arbitrary simulator engine with custom system setup to a web-based simulator to reach a broader audience (i.e. low-tech medical). Both the geometric simulation basis and interactive deformations can be represented with our extended X3D, which already includes a rich palette of animation, sensors and interaction. Leveraging the X3D scene graph as a scenario description layer and a deployment medium for medical applications holds great promise. Through the interoperable pipelines with existing imaging and informatics standards points, lines, planes, surfaces, volumes, and volumetric meshes all co-exist in an interactive immersive environment. Worlds and collaborative events within can be shared across multiple users on a network. Perhaps most interestingly, the metadata and demonstrated ontology support in the X3D scene graph could be expanded to practical integration with other standards for online curricula for education and training, providing additional context and semantics for representing space-ﬁlling meshes. Conformance supports progress since all science must be repeatable. From Human-Computer Interaction to medical education and clinical training, a common haptic representation of (deformable) anatomical geometry will yield farreaching beneﬁts. Inevitably, future computational work will include the further development of methods to quickly and accurately derive space-ﬁlling FEM meshes from volumetric scan data. Ultimately, our vision is for an open repository of shared anatomical datasets that can be used for visualization and simulation purposes.

References [1] Vidal FP, Bello F, Brodlie KW, John NW, Gould D, Phillips R, Avis NJ. Principles and Applications of Computer Graphics in Medicine. Computer Graphics Forum. 2006 March;25(1):113–137. [2] Teich C, Liao W, Ullrich S, Kuhlen T, Ntouba A, Rossaint R, Ullisch M, Deserno TM. MITK-based segmentation of co-registered MRI for subject-related regional anaesthesia simulation. In: Proceedings SPIE Medical Imaging 2008. San Diego, USA; 2008. p. 69182M–69182M–10. [3] Smith M, Faraci A, Bello F. Segmentation and generation of patient-speciﬁc 3D models of anatomy for surgical simulation. Studies in Health Technology and Informatics. 2004;98:360–362. [4] Zygote Media Group. 3D Models and 3D Animations of the Male and Female Anatomy; last accessed October 12th, 2010. http://www.3dscience.com. [5] VOXEL-MAN Group. Surgery Simulators and Virtual Body Models; last accessed October 12th, 2010. http://www.voxel-man.de. [6] John NW, Aratow M, Couch J, Evestedt D, Hudson AD, Polys N, Puk RF, Ray A, Victor K, Wang Q. MedX3D: standards enabled desktop medical 3D. Studies in health technology and informatics. 2008;132:189–194. [7] Rosse C, Mejino JLV. A reference ontology for biomedical informatics: the Foundational Model of Anatomy. J Biomed Inform. 2003 Dec;36(6):478–500.

676

S. Ullrich et al. / Quantizing the Void: Extending Web3D for Space-Filling Haptic Meshes

[8] University of Washington, School of Medicine. Foundational Model of Anatomy (FMA); last accessed October 12th, 2010. http://sig.biostr.washington.edu/projects/fm/index.html. [9] Organisation IHTSD. Systematized Nomenclature of Human and Veterinary Medicine (SNOMED); last accessed October 12th, 2010. http://www.ihtsdo.org. [10] Polys NF, Brutzman D, Steed A, Behr J. Future Standards for Immersive VR: Report on the IEEE Virtual Reality 2007 Workshop. IEEE Computer Graphics & Applications. 2008 march;28(2):94–99. [11] John NW, Polys N, Aratow M, Evestedt D. Medical Virtual Environments. In: IEEE Virtual Reality 2010. Waltham, MA, USA: http://www.hpv.cs.bangor.ac.uk/vr10-med/. http://www.hpv.cs.bangor.ac.uk/vr10-med/; 2010. [12] Behr J, Eschler P, Jung Y, Z¨ ollner M. X3DOM: a DOM-based HTML5/X3D integration model. In: Web3D ’09: Proceedings of the 14th International Conference on 3D Web Technology. New York, NY, USA: ACM; 2009. p. 127–135. [13] Si H. TetGen: A Quality Tetrahedral Mesh Generator; last accessed October 12th, 2010. http://tetgen.berlios.de. [14] Sch¨ oberl J. NETGEN - automatic mesh generator; last accessed October 12th, 2010. http://www.hpfem.jku.at/netgen. [15] Open CASCADE. SALOME platform; last accessed October 12th, 2010. http://www.salome-platform.org. [16] Geuzaine C, Remacle JF. Gmsh: A 3-D ﬁnite element mesh generator with built-in preand post-processing facilities. International Journal for Numerical Methods in Engineering. 2009 May;79(11):1309–1331. [17] MPI and NLE-IT. The SimBio-Vgrid mesh generator; last accessed October 12th, 2010. http://www.rheinahrcampus.de/ medsim/vgrid. [18] Berti G. Image-based unstructured 3D mesh generation for medical applications. In: European Congress on Computational Methods in Applied Sciences and Engineering; 2004. [19] Kitware Inc . Chapter 19.3 VTK File Formats. In: The VTK User’s Guide. 11th ed.; 2010. [20] John NW. The Impact of Web3D Technologies on Medical Education and Training. Computers & Education. 2007;49(1):19–31. [21] Moorthy K, Mansoori M, Bello F, Hance J, Undre S, Munz Y, Darzi A. Evaluation of the beneﬁt of VR simulation in a multi-media web-based educational tool. Studies in Health Technology and Informatics. 2004;98:247–252. [22] Bello F, Bulpitt A, Gould DA, Holbrey R, Hunt C, How T, John NW, Johnson S, Phillips R, Sinha A, Vidal FP, Villard PF, Woolnough H, Zhang Y. ImaGINe-S: Imaging Guided Interventional Needle Simulation. In: Eurographics; 2009. p. 5–8. [23] Johnson S, Hunt C, Woolnough H, Crawshaw M, Kilkenny C, Gould D, England A, Sinha A. Assessing Performance on a Virtual Reality Simulated Liver Biopsy Procedure: Validating Imagine-S. In: (Poster presentation) Society of Interventional Radiology (SIR) Annual Scientiﬁc Meeting; 2010. [24] Ullrich S, Grottke O, Fried E, Frommen T, Liao W, Rossaint R, Kuhlen T, Deserno TM. An intersubject variable regional anaesthesia simulator with a virtual patient architecture. International Journal of Computer Assisted Radiology and Surgery. 2009 November;4(6):561–570. [25] Grottke O, Ntouba A, Ullrich S, Liao W, Fried E, Prescher A, Deserno TM, Kuhlen T, Rossaint R. Virtual reality-based simulator for training in regional anaesthesia. British Journal of Anaesthesia. 2009 October;103(4):594–600. [26] Virtual Reality Group RAU. ViSTA VR toolkit; last accessed October 12th, 2010. http://sourceforge.net/projects/vistavrtoolkit/. [27] Allard J, Cotin S, Faure F, Bensoussan PJ, Poyer F, Duriez C, Delingette H, Grisoni L. SOFA an Open Source Framework for Medical Simulation. In: Medicine Meets Virtual Reality (MMVR’15). Long Beach, USA; 2007. p. 13–18. [28] Web3D Consortium. X3D speciﬁcation 2008-07-10; last accessed October 12th, 2010. http://www.web3d.org/x3d/speciﬁcations/ISO-IEC-19775-1.2-X3DAbstractSpeciﬁcation/.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-677

677

Dissecting in Silico: Towards a Taxonomy for Medical Simulators Sebastian ULLRICH a,1 , Thomas KNOTT a and Torsten KUHLEN a a RWTH Aachen University, Germany Abstract. In this paper, we investigated several approaches in literature that classify diﬀerent aspects of medical simulators. We have merged these deﬁnitions to form a structured taxonomy. This new taxonomy should facilitate the design of new medical simulators and allow to analyze and classify existing simulators, algorithms, toolkits and hardware. Keywords. Taxonomy, Medical, Simulation, Hardware, Datasets, Software

1. Introduction & Related Work Taxonomies are a popular device to generate a classiﬁcation. To our knowledge, there is no taxonomy for the design of VR-based medical simulators. To generate the simulator taxonomy, we have analyzed position papers, surveys of existing simulators and other related literature. Satava postulated ﬁve generations of simulators: geometric anatomy, physical dynamics modeling, physiologic characteristics, microscopic anatomy, and biochemical systems [1]. Furthermore, he deﬁned the following requirements for realism in medical simulators: visual ﬁdelity, interactivity between objects, object physical properties, object physiologic properties, and sensory input. Liu et al. [2] discriminate between technical (deformable models, collision detection, visual and haptic displays, and tissue modeling and characterization) and cognitive components (performance and training). Delingette [3] divided simulator components into input devices, surgery simulator (collision detection and processing, geometric modelling, physical modelling, haptic rendering, and visual rendering), and output devices. In a recent overview by John [4], three areas have been deﬁned: input data, processor, and interaction. Here, interaction has been subdivided into haptics, display technologies, other hardware components, and algorithms and software.

2. Taxonomy Merging the deﬁnitions and reports of the related work, we propose a taxonomy (see Figure 1) with three main classes: Datasets, Hardware, and Software. 1 Corresponding

E-Mail Address: [email protected].

678

S. Ullrich et al. / Dissecting in Silico: Towards a Taxonomy for Medical Simulators

Datasets: Synthetic datasets can be Computed (e.g., based on statistic models or heuristics) or Modeled (e.g., produced by digital artists with 3D modeling tools, or sourced from CAD designs of instruments). Usually, these are well-meshed surface geometries with highly detailed textures. Another approach are Subject-speciﬁc datasets. Several medical imaging modalities (e.g., sonography, MRI, CT) allow the reconstruction of volume data, that can either be used directly or segmented for further processing. Furthermore, physiological parameters, tissue properties and other can either be measured In Vivo or Ex Vivo. Hardware: Interaction devices can either be Sensor-based or Props. Sensor-based devices can be commercial oﬀ-the-shelf products, self-constructed prototypes or hybrids and examples range from game console controllers to haptic devices and optical tracking systems. Props can replicate body-parts or instruments, that are either augmented, tracked or simply passive parts of the overall setup. The Processing unit relates to the kind of computing systems that are used for the simulator. This can be Stationary (e.g., single- or multi-core systems, clusters, or servers) or Mobile systems (e.g., handheld devices, or streaming clients). Furthermore, GPUs can be used for parallelization. Finally, the Output can be realized on several modalities, Visual, Haptic and Acoustic being the three most common. The visual component can be further divided into diﬀerent display types: HMD, screen or projection screen with or without stereoscopic rendering. Likewise, haptics is divided into tactile and kinesthetic feedback. Software: The Model is the link between the datasets and the algorithms. It can be regarded from two points of view: Technical (e.g., data structure, LODs, mappings [5], etc.) and Content (e.g., patient, instruments, and environment [6]). One crucial element for the acceptance of a medical simulator is the Interaction with numerous solutions from HCI and 3DUI. Here, we can distinguish between Tasks (navigation, selection, manipulation [7], session management, assessment etc.), Metaphors (direct “natural” interaction, gestures, etc. [8])) and Technical (e.g., GUI elements, OSDs, or annotations). Simulation is divided into diﬀerent levels: Static (e.g., ﬁxed structural anatomy, environment), Dynamic (e.g., physics-based with collision detection and handling [9], rigid body dynamics, or continuum mechanics applied to soft tissue) and Physiological (e.g., functional anatomy [10], or the physiome project). The Rendering is tightly coupled to the results of the simulation. It can be divided into Visual, Haptic or Acoustic algorithms.

3. Conclusions/Discussion First of all, the proposed taxonomy aims to provide a standardized terminology and classiﬁcation. External input is needed to further improve this taxonomy. We believe that this taxonomy can support the design or analysis of simulators. For example, after a task analysis of a medical procedure, the taxonomy could be used to identify which simulator components are necessary and where to prioritize the development. A common pitfall is over-engineering. This can be avoided by careful examination of each component and deﬁnition of the requirements. Here, ﬁdelity is a good measure with many diﬀerent metrics (granularity, interactivity, etc.) and can be applied to any component of a simulator system individually.

S. Ullrich et al. / Dissecting in Silico: Towards a Taxonomy for Medical Simulators

$%

! %

679

!

" "

# # # $ # #

"

&

& $ & '

& #(

# " $ #

)

$

$ & & * +

, $ -

') # & "

-

."# / /()

& #

# #0 1 2 +1#1 ,

#

$

+ # , "+!, +! ,* + !, + ,

Figure 1. Overview of the simulator taxonomy with examples. The selected examples only represent a subset of the rich possibilities in each category.

References [1] Satava RM. Medical virtual reality. The current status of the future. Studies in Health Technology and Informatics. 1996;29:100–106. [2] Liu A, Tendick F, Cleary K, Kaufmann C. A survey of surgical simulation: applications, technology, and education. Presence: Teleoper Virtual Environ. 2003;12(6):599–614. [3] Delingette H, Ayache N. Soft Tissue Modeling for Surgery Simulation. In: Computational Models for the Human Body. Elsevier; 2004. p. 453–550. [4] John NW. Design and implementation of medical training simulators. Virtual Reality. 2008;12(4):269–279. [5] Rosse C, Mejino JLV. A reference ontology for biomedical informatics: the Foundational Model of Anatomy. J Biomed Inform. 2003 Dec;36(6):478–500. [6] Harders M. Surgical Scene Generation for Virtual Reality-Based Training in Medicine. Santa Clara, CA, USA: Springer-Verlag; 2008. [7] Heinrichs WL, Srivastava S, Montgomery K, Dev P. The Fundamental Manipulations of Surgery: A Structured Vocabulary for Designing Surgical Curricula and Simulators. The Journal of the American Associacion of Gynecologic Laparoscopists. 2004;11(4):450–456. [8] Bowman DA, Kruijﬀ E, LaViola JJ, Poupyrev I. 3D User Interfaces: Theory and Practice. Addison-Wesley Professional; 2004. [9] Teschner M, Kimmerle S, Heidelberger B, Zachmann G, Raghupathi L, Fuhrmann A, Cani MP, Faure F, Magnenat-Thalmann N, Strasser W, Volino P. Collision Detection for Deformable Objects. Computer Graphics Forum. 2005 March;24(1):61–81. [10] Ullrich S, Valvoda JT, Prescher A, Kuhlen T. Comprehensive architecture for simulation of the human body based on functional anatomy. In: Proceedings Bildverarbeitung f¨ ur die Medizin 2007. Springer Verlag; 2007. p. 328–332.

680

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-680

Computed Tomography as Ground Truth for Stereo Vision Measurements of Skin Amy M VANBERLOa, Aaron R CAMPBELLb, and Randy E ELLISa,b,c,1 School of Computing, Queen’s University, Kingston, ON, Canada K7L3N6 b Department of Surgery, Queen’s University c Department of Mechanical and Materials Engineering, Queen’s University a

Abstract. Although dysesthesia is a common surgical complication, there is no accepted method for quantitatively tracking its progression. To address this, two types of computer vision technologies were tested in a total of four configurations. Surface regions on plastic models of limbs were delineated with colored tape, imaged, and compared with computed tomography scans. The most accurate system used visually projected texture captured by a binocular stereo camera, capable of measuring areas to within 3.4% of the ground-truth areas. This simple, inexpensive technology shows promise for postoperative monitoring of dysesthesia surrounding surgical scars. Keywords: Dysesthesia, Numbness, Skin Area, Stereoscopic Vision, Computed Tomography

Introduction A common complication of major surgery, such as total knee replacement, is a burning and/or numbness of the skin surrounding the surgical scar. The degree of this condition –dysesthesia– can be clinically evaluated using sensory testing. There is, however, no reliable accepted method for quantitatively tracking the surface area of such a region. The literature on direct dysesthesia measurement of skin areas is relatively sparse. A literature search found a single relevant paper from 1991 by Berg and Mjoberg [1], who made two linear measurements and approximated the curved skin surface as a simple ellipse. This very limited background suggests that there is a clinical need for a way to accurately quantify how a patient’s dysesthesia region changes over time. We propose that stereoscopic computer vision technology can be used to quantify the 2D shape and area of a 3D region of skin that is substantially curved, such as the knee, hip or ankle. To study this we compared four technologies, using computed tomography (CT) scans as ground truth. Regions detected by the vision systems were compared to the CT scans, and the vision systems were evaluated for scanning time and ease of use. A passive stereo system with the addition of visually projected texture was significantly more accurate than the alternatives and was superior in its ease of use. Such a scanning system can easily be installed in an orthopedic clinic and will be a safe noninvasive way to track the progression of this important surgical complication. 1

Corresponding Author: Randy E Ellis, School of Computing, Queen’s University, Kingston, ON, Canada K7L 3N6; E-mail: [email protected]

A.M. Vanberlo et al. / CT as Ground Truth for Stereo Vision Measurements of Skin

681

Development of a real-time, high accuracy 3D surface mapping calculation technology will allow orthopaedic personnel to quickly and accurately quantify and track dysesthesia and other such skin surface complications. Potential applications in other medical specialties, such as plastic surgery and dermatology, are readily imagined.

1. Materials Two plastic models of the lower limb (Pacific Research Laboratories, Bellingham, USA; items 1518 & 1519-2) were used to mimic the human lower extremity. Skin regions were selected to represent relatively small dysesthesia, three on the foot/ankle model and eight on the full limb. Each region was delineated with colored adhesive tape so as to be clearly identifiable visually and radiologically. One model, with the taped regions, is shown in Figure 1. The most likely clinical method would be twopoint discrimination, a method shown to have an accuracy no better than 5mm in adult lower limb testing [2], which restricted regions to the 35mm-100mm range.

Figure 1. A plastic limb (left) and a 3D reconstruction from a CT scan (right). Regions were delineated with colored adhesive tape that aided in automatic visual segmentation. The CT images were manually segmented and the resulting surfaces acted as ground truth for stereo camera testing.

1.1. Technology Two computer vision technologies, in a total of four system configurations, were used: a Desktop 3D active scanner (NextEngine, Santa Monica, USA) that used a monocular camera and structured light to determine point depth, and a Bumblebee2 binocular system (Point Grey Research, Richmond, CA) that used correspondence algorithms to find 3D points. Preliminary work suggested a potential need for visual texture when using the binocular system; this was first omitted, then applied physically to the models and, later, virtually using a compact computer projector (SHARP, Mahwah, USA). Hereafter the laser system will be called “active”, the binocular system with no texture will be “untextured”, binocular with physically applied texture will be “physical”, and the binocular system with projection will be “passive”. Each vision system had software provided for the express purpose of finding 3D point positions. The software included means for previewing the data and calculating point positions. The manufacturer software was used without any modifications. On each model, outlines representing dysesthesia regions were demarcated as convex polygons. For the physical system, texture was randomly applied using transfer decals of lettering (14pt). For the passive system, the same font was projected onto the skin as a virtual texture. The untextured system simply imaged the model skin.

682

A.M. Vanberlo et al. / CT as Ground Truth for Stereo Vision Measurements of Skin

2. Methods Ground truth was established by acquiring CT scans of each model with a Lightspeed mobile-gantry scanner (General Electric, Milwaukee, USA). The scans were taken with 0.625mm slice thickness. The regions were manually segmented using the commercially available Mimics software (Materialise, Leuven, Belgium). The surface area of each region was calculated using the related Magics software. These regions were deemed to be ground truth. A sample region and its surrounding surface is shown in Figure 2; the manually segmented inner surface is shown in Figure 2(a).

(a) (b) (c) (d) (e) Figure 2. A region of interest from each imaging system. (a) the region segmented from a CT scan (b) the active system that used a light plane (c) the binocular untextured system imaged the plastic skin directly (d) the binocular system imaged the physically applied texture as dark blotches (e) the binocular system imaged the visually projected texture,.

The vision systems were then used to acquire each surface region. After a single region was captured, the maximum final acquisition time was noted. The active data were transferred in VRML format, whereas the binocular data were transferred as point clouds. Both data types were converted to a common format, saved, and later imported into Matlab (Mathworks, Natick, USA) for analysis and display. Final automatic segmentation of each region was determined by its quasi-planar r characteristics. First, the barycenter b of a region was calculated; each point was then r translated by - b so that the region had its centroid at the origin. The plane of best fit was next found as the orthogonal linear regression of the points, using principal components analysis [3]. The data were rotated so that the best-fit plane was aligned with the XY plane, then orthogonally down-projected to produce a 2D surface patch. The convex hull [4] of the 2D tape region was calculated then shrunk to reject the 8mm outline tape. The points inside this smaller hull were re-segmented based on color, to reject any points associated with the outline tape. The segmented points were then upprojected to their rotated zero-mean 3D positions. A Delaunay algorithm [4] was used to triangulate the 3D region. Finally, the area of the region was calculated as the sum of the area of the triangles. A sample region, after image processing, is shown in Figure 2(b)-(e) for each camera system.

A.M. Vanberlo et al. / CT as Ground Truth for Stereo Vision Measurements of Skin

683

3. Results Acquisitions time, after setup, of the binocular systems were effectively instantaneous in data capture. The active system, by contrast, required up to 10 seconds to capture the data as the plane of light was scanned across the field of view of its single camera. Regarding area measurements, Table 1 summarizes the statistical comparison of the four visual measurements to the ground-truth CT measurements. Table 1. Systems Compared to CT (ground truth) System Type

Mean Rel. Area

Std. Dev.

t-test p value

f-test p value

Active

-0.088

0.151

0.097

0.000

Untextured

0.424

0.803

0.129

0.459

Physical

-0.112

0.129

0.010

0.000

Passive

0.034

0.214

0.630

0.001

The active system under-estimated areas by about 9%, which was close to being significantly different from ground truth. The untextured system over-estimated areas by over 42%. The physically textured system underestimated areas by 11%, which was significantly different from ground truth. The passive system over-estimated areas by just over 3% and was statistically similar to the CT-based ground truth. We next compared the active, untextured and physically applied systems to the most accurate system, which passively projected visual texture onto the skin. The results are presented in Table 2. The accuracy of each system was its mean relative error, and its precision (also called repeatability) was its standard deviation. Table 2. Systems Compared to Passive Binocular Stereoscopic Vision with Projected Texture System Type

Mean Rel. Area

Std. Dev.

t-test p value

f-test p value

Active

0.003

0.368

0.978

0.020

Untextured

0.810

1.878

0.205

0.000

Physical

0.030

0.293

0.796

0.003

The active system was clearly the next best, significantly similar to the passive system with a probability value of (1-p)<0.03. Neither the untextured system nor the physically applied system was significantly similar or different, although the trends were that the untextured system trended to be different and the physically applied system trended to be the same. All of the systems had significantly different variances. The active system and physically applied systems were more precise, whereas the untextured system was very large compared to the passive system.

684

A.M. Vanberlo et al. / CT as Ground Truth for Stereo Vision Measurements of Skin

4. Conclusions For future clinical applications, the active system was rejected because of the excessive scan time, taking up to 10 seconds to acquire the data. Of the binocular systems, the plain untextured system was rejected because of its poor accuracy and low precision. The passive system, with projected visual texture, and the system with physically applied texture were statistically equivalent. Because applying texture is physically intrusive, and may upset some patients, this was also rejected for future consideration. Thus the binocular system with passive visually projected texture was selected as the superior technology for subsequent research. This study has clear limitations. Only 11 regions from 2 plastic models were used. The study was conducted in a laboratory, rather than in a clinic. The automated segmentation was sensitive to the quality of the setup. Most importantly, this is a preclinical pilot whose purpose was to compare 4 technologies for measuring carefully delineated surface regions. In conclusion, these results suggest that physically projecting visual texture onto skin, then using a simple commercial binocular camera system, can reliably estimate the areas of region despite considerable curvature of the underlying surface being measured. Future work will include a pilot human study and then a clinical study on patients. This preliminary investigation has shown that it is possible to accurately measure skin regions with a simple, fast, semi-automated process that is a candidate technology for monitoring the progression of a common complication of major surgery.

Acknowledgments The active-vision hardware was graciously loaned by Dr. Michael Greenspan. Invaluable assistance was provided by Paul St. John and Mohamed Hefny. This work was supported in part by the Canada Foundation for Innovation, Kingston General Hospital, and the Natural Sciences and Engineering Research Council of Canada.

References [1] [2] [3] [4]

Berg P and Mjober M. “A lateral skin incision reduces peripatellar dysaesthesia after knee surgery”. J Bone Joint Surg [Br] 73-b(3):374-376, 1991. Nolan MF “Limits of two-point discrimination ability in the lower limb in young adult men and women.” Phys Ther. 63(9):1424-8. 1983. Petras I and Podlubny I. “State space description of national economies: The V4 countries”. Comput Stat Data Anal 52(2):1223-1233, 2007. Barber CB, Dobkin DP, and Huhdanpaa H. “The Quickhull algorithm for convex hulls.” ACM Trans Math Software 22(4):469-483, 1996.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-685

685

Towards the Visualization of Spiking Neurons in Virtual Reality Anette VON KAPRI a,b,1 , Tobias RICK a,b , Tobias C. POTJANS c,d , Markus DIESMANN d,e and Torsten KUHLEN a,b a Virtual Reality Group – JARA, RWTH Aachen University b Jülich-Aachen Research Alliance (JARA-HPC) c Institute of Neuroscience and Medicine, INM-6, Research Center Jülich, Germany d RIKEN Computational Science Research Program e RIKEN Brain Science Institute, Wako, Japan Abstract. This paper presents a prototype that addresses the visualization of the microscopic activity structure in the mammalian brain. Our approach displays the spiking behaviour of neurons in multiple layers based on large-scale simulations of the cortical microcircuit. We will apply this visualization to the activity of brainscale simulations by coupling the microscopic structure with the macroscopic level. Thereby, we hope to convey an intuitive understanding of the concise interaction and the activity ﬂow of pairs of distant brain areas. Keywords. Spiking neurons, virtual reality, data visualization

1. Introduction The relationship between structure and function of the human brain is still not understood completely. On the microscopic level the information processing units are neurons that communicate by means of electrical pulses (so-called spikes). A single neuron integrates the information of around 10,000 other neurons, partly located within the local microcircuit and partly from multiple distant areas. Large-scale simulations link the structure of cortical networks to the neuronal activity. However, the complexity of these networks forms an obstacle for the visualization of the simulated data. Previous approaches on the visualization of spiking neuronal network simulations focus on single-scale networks [1]. Neuronal network simulations, however, are currently making a qualitative leap to focus on the multi-scale nature of the brain [2]. Here, we present our approach to accompany these developments with appropriate visualization tools that capture the activity of brain-scale simulations. 2. Method In a ﬁrst step, we address the visualization of a layered cortical network model that represents the local microcircuit below 1 mm2 of cortical surface [3]. The model includes 1 Corresponding

Author: [email protected]

686

A. von Kapri et al. / Towards the Visualization of Spiking Neurons in Virtual Reality

eight cell types, four excitatory and four inhibitory, distributed in the cortical layers 2/3, 4, 5 and 6. Overall, the model consists of 80,000 neurons and 0.3 billion synapses. Based on the cell-type speciﬁc connectivity, the spontaneous and evoked activity of the model reﬂects in vivo recordings. In a second step, we extend our considerations to multi-scale networks that consist of multiple microcircuits, each representing a cortical area, with in total several million neurons. 2.1. Requirements For an interactive visualization of multi-scale spiking neuronal networks we formulate three conceptual requirements. (1) The visualization must reﬂect the neuronal activity, including e.g. the membrane potential and spike events of the neurons over time. (2) The system should allow to adjust the level of detail ranging from the representation of a single cell up to the whole neuronal population. (3) The visualization should convey the inﬂuence of the macroscopic communication (between different brain areas) on the interaction at a microscopic level (between individual neurons). 2.2. Initial Approach Excitatory cells are represented as pyramids and inhibitory cells as spheres (Figure 1 left) to distinguish both neuron types from each other. The layers are outlined and an alternate coloring scheme is used for neurons in consecutive layers (Figure 1). Over time spiking neurons are highlighted and a global overview is provided by activating different layout types (Figure 2). Our system is built on top of a virtual reality application in order to provide direct interaction metaphors and increase the depth perception (Figure 3). See [4].

Figure 1. Left: Inhibitory (sphere) and excitatory cell (pyramid). Middle: Layers are organized above each other. Right: Layers are fanned out.

2.3. Extensions Future extensions integrate statistical information, such as the population ﬁring rates, the irregularity and synchrony of the activity and the cross correlation patterns, to quantify the spiking behavior of the different cell types. Additionally, we visualize connections between ﬁring neurons to understand the propagation of the electric impulses. For the overall pattern of spiking neurons it would be necessary to visualize a larger amount of cells thereby concentrating on the membrane potential.

A. von Kapri et al. / Towards the Visualization of Spiking Neurons in Virtual Reality

687

Figure 2. Spiking behavior in layer 5 over time. Spiking neurons are drawn larger and in a more intense color.

Figure 3. Visualization in the CAVE virtual environment.

Our approach unfolds its potential when we combine the visualization with the activity of multi-scale simulations where an intuitive understanding of the activity patterns is hard to accomplish. Therefore, we couple a macroscopic visualization to the microscopic one. The user may choose pairs of areas for a detailed visualization to shed light on the concise interaction and the activity ﬂow. 3. Conclusion Our interactive visualization of brain-scale neuronal networks does not only provide a measure to cope with the increasing amout of neuron simulation data but furthermore enables the development of an intuitive understanding of the spiking pattern of neurons. References [1] A. Kasi´nski, J. Pawlowski, and F. Ponulak, “SNN3DViewer - 3D visualization tool for spiking neural network analysis,” in ICCVG’09: Proceedings of the International Conference on Computer Vision and Graphics. Berlin, Heidelberg: Springer-Verlag, 2009, pp. 469–476. [2] T. C. Potjans, S. Kunkel, A. Morrison, H. Plesser, R. Kötter, and M. Diesmann, “Brain-scale simulations with NEST: supercomputers as data integration facilities,” in Frontiers in Neuroscience. Conference Abstract: Neuroinformatics, 2010. [3] T. C. Potjans, T. Fukai, and M. Diesmann, “Implications of the speciﬁc cortical circuitry for the network dynamics of a layered cortical network model,” BMC Neuroscience, vol. 10, no. 1, p. 159, 2009. [4] “Youtube video of neuron visualization.” [Online]. Available: http://www.youtube.com/watch?v=UnDtePh0ci8

688

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-688

The Use of Virtual Training to Support Insertion of Advanced Technology at Remote Military Locations Madison I. WALKERa, Robert B. WALKER, MDa, Jeffrey S. MORGAN, MDa, Mary BERNHAGEN, BSb, Nicholas MARKIN, MDb, and Ben H. BOEDEKER, MDb,1 a U.S. Army European Regional Medical Command, Heidelberg, Germany Dept. of Anesthesiology, University of Nebraska Medical Center, Omaha, NE

Abstract. Effective training in advanced medical technologies is essential for military healthcare providers to support the far forward battlefield. The use of modern video communication technologies and novel medical devices can be utilized for meeting this challenge. This study demonstrates the combined use of video conferencing equipment and videolaryngoscopy in the virtual training of a novice in videolaryngoscopy, nasal intubation and airway foreign body removal. Keywords. Intubation training, videolaryngoscopy, video conferencing

Background It is essential that the US military has an effective method for rapid insertion of novel medical technology to far forward locations. An example of such novel technology is advanced airway management tools [1-2]. Airway management problems are estimated to comprise up to 10% of preventable combat deaths in recent conflicts. Airway obstruction contributed to death from civilian major trauma in up to 85% of patients dying before reaching the hospital [3-4]. This demonstration utilized advance video communication technology to train a novice airway manager in videolaryngoscopy.

Methods and Materials A video communications linkage was made between the University of Nebraska Medical Center’s Center for Advanced Technology and Telemedicine (CATT) in Omaha Nebraska and the Heidelberg Army Medical Clinic in Germany. The room equipment at the Omaha site includes a Polycom 7000 with a wall mounted 48” LCD monitor. The connection was via six bonded integrated services digital network (ISDN) lines to create a bandwidth of 384 kbs. A Codian MCU (Multiway Control Unit) was used to link the room’s Polycom system with the ISDN call to Germany.

1

Corresponding Author: Ben H. Boedeker, MD, PhD, Professor, Department of Anesthesiology, Director, Center for Advanced Technology and Telemedicine, University of Nebraska Medical Center, 984455 Nebraska Medical Center, Omaha, NE 68198-4455, USA; E-mail: [email protected]

M.I. Walker et al. / The Use of Virtual Training to Support Insertion of Advanced Technology

689

Located at the Germany site, the trainee (a 16 year old high school student) was instructed in: 1) how to set up the Karl Storz CMAC™ videolaryngoscope (Karl Storz Endoscopy, Tuttlingen, Germany); 2) how to clean the device; 3) how to perform videolaryngoscopy, oral endotracheal intubation, and nasal intubation; and 4) how to remove a foreign body lodged in the glottic opening using videolaryngoscopy and the Boedeker Curved Intubation Forceps. Both the instructor (at the Omaha site) and the trainee performed the laryngoscopy and intubation demonstrations on a Laerdal Difficult Airway Trainer™ (Laerdal Medical Corp, Gatesville, TX).

Results The novice intubator was able to successfully set up the videolaryngoscope, perform an endotracheal intubation using videolaryngoscopy (Figure 1), successfully perform a nasal intubation using videolaryngoscopy assisted by the Boedeker Forceps (Figure 2) and successfully extract a foreign body from the glottis opening using videolaryngoscopy and the Boedeker Forceps (Figures 3-5).

Figure 1. The trainee performing the intubation was located in a medical clinic in Heidelberg, Germany.

Figure 2. The trainee successfully performs a nasal intubation using the Boedeker Forceps and videolaryngoscope.

Figure 3. Using the Boedeker Forceps and the videolaryngoscope, the student is mentored in airway foreign body removal.

690

M.I. Walker et al. / The Use of Virtual Training to Support Insertion of Advanced Technology

Figure 4. View of the foreign body in the glottic opening with the videolaryngoscope.

Figure 5. The student is successful in removing the foreign body.

Conclusions This was the first documented transatlantic virtual training in videolaryngoscopy, the first virtual training of nasal intubation supported by videolaryngoscopy and the first virtual training of foreign body removal using videolaryngoscopy. Video conferencing technology was successfully used to insert a videolaryngoscope system in an OCONUS military installation and conduct complex training tasks. This concept could be deployed to support training required for just-in-time insertion of complex military medical devices to support the far forward medical mission.

References [1] [2] [3] [4]

B.H. Boedeker, S. Hoffman, W.B. Murray. Endotracheal Intubation using virtual images: learning with the mobile telementoring intubating video laryngoscope. Stud Health Technol Inform 125 (2007), 4954. Published by IOS Press. B.H. Boedeker, B.W. Berg, M. Bernhagen, W.B. Murray. Direct versus indirect laryngoscopic visualization in human endotracheal intubation: a tool for virtual anesthesia practice and teleanesthesiology. Stud Health Technol Inform 132 (2008), 31-6. Published by IOS Press. L. Hussain, A. Redmond. Are pre hospital deaths from accidental injury preventable? Br Med J 308 (1994), 1077-80. J. Nicholl, S. Hughes, S. Dixon. The costs and benefits of paramedic skills in pre hospital trauma care. Health Technol Assess 2 (1998), 10-5.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-691

691

Three Dimensional Projection Environment for Molecular Design and Surgical Simulation Eric WICKSTROMa,d, Chang-Po CHENa, Devakumar DEVADHASb, Matthew WAMPOLEa, Yuan-Yuan JINa, Jeffrey M. SANDERSa, John C. KAIRYSc, Martha L. ANKENYe, Rui HUf, Kenneth E. BARNERf, Karl V. STEINERf and Mathew L. THAKURb,d, a Biochemistry & Molecular Biology, bRadiology, cSurgery, dKimmel Cancer Center, e Academic and Instructional Support and Resources, Thomas Jefferson University, Philadelphia PA 19107 f Electrical and Computer Engineering, University of Delaware, Newark DE 19716

Abstract. We are developing agents for positron emission tomography (PET) imaging of cancer gene mRNA expression and software to fuse mRNA PET images with anatomical computerized tomography (CT) images to enable volumetric (3D) haptic (touch-and-feel) simulation of pancreatic cancer and surrounding organs prior to surgery in a particular patient. We have identified a novel ligand specific for epidermal growth factor receptor (EGFR) to direct PET agent uptake specifically into cancer cells, and created a volumetric haptic surgical simulation of human pancreatic cancer reconstructed from patient CT data. Young’s modulus and the Poisson ratio for each tissue will be adjusted to fit the experience of participating surgeons. Keywords. Cancer, haptic, pancreas, SOFA, surgery, tumor, volumetric

Introduction Surgery involves palpating and manipulating tissues in the operating room environment. However, sophisticated radiographic systems present only visual images. The actual assembly of organs of a particular patient must now be imagined by the surgeon before the operation. Complications that were not anticipated, such as bleeding from unusually placed arteries or veins, or unusual lesion geometry, lengthen the procedure, placing extra stress on the patient and the surgeons. One solution is the Simbionix Procedure Rehearsal Studio™ program for the AngioMentor™ simulator that allows surgeons to upload patients’ actual CT angiogram images and then perform a virtual procedure, complete with visual and haptic feedback, on that patient’s anatomy [4]. But the Simbionix system only uses anatomical data. We hypothesize that our fusion of genetic, visual, and tactile information will improve surgeons’ understanding of the extent of disease and will ultimately permit surgeons to better plan operations and to prepare for the actual pathology found.

692

E. Wickstrom et al. / Three Dimensional Projection Environment

1. Methods 1.1. PET Imaging Agents for mRNA We design novel EGFR ligands by molecular dynamics using Amber 10 [5] force fields, followed by haptic sensing of the kinetic pathway of ligand binding to the receptor. The energetically optimal peptide selected above is extended from a solid phase support by solid phase coupling of Fmoc-L-amino acid monomers. The terminal Cys residues are cyclized. A diethyleneglycol spacer is then coupled, followed by the mutant KRAS2 G12D PNA sequence, a second diethyleneglycol spacer, then finally a fluorophore or a DOTA chelator [6]. PNA mismatch and peptide mismatch controls are also prepared. The finished reporter-PNA-peptides are cleaved from the solid support, deprotected, purified by reversed phase liquid chromatography, and analyzed by mass spectroscopy to determine whether the desired sequence was synthesized correctly. 1.2. Volumetric Images We assembled a prototype system for haptic manipulation of volumetric images of pancreatic cancer gene expression and anatomy. Slicer3 [7], Amira 5 (Visage Imaging) SOFA [8] and Phantom Omni manipulators (SensAble) are installed on a 3DBoxx 8570 (Boxx Technologies) with 8 computing nodes and 48 GB of RAM to support an Nvidia 5800 video card with 4 GB GPU. 1.3. Haptic Manipulation A pair of Phantom Omni manipulators is controlled by the end-user to provide translation and rotation input information for surgery instruments in the virtual reality environment. A collision detection algorithm is adopted to detect the penetration between surface meshes of virtual organs and instruments. If a mesh controlled by the haptic device is involved in a collision, the corresponding penetration volume is sampled. The haptic force is calculated using this volume and feedback is provided for the haptic device to give the end-user realistic feedback. To render variations in haptic stiffness properties in different regions of the target tissue, we strategically place key vertices over the surface mesh. Each of these vertices contains an influence radius. As the virtual instrument enters any of these radiuses, the feedback force is appropriately scaled. The proximity of the surgery instrument to the vertex and a predefined base stiffness value control the scale factor, creating distinct regional haptic properties.

2. Results 2.1. Modeling and Testing of EGFR Ligands for mRNA PET Imaging Agents The EGF central loop fragment, amino acids 20-31, CMYIEALDKYAC, with Cys-Cys disulfide bridge between Cys20 and Cys31 (Fig. 1A), was selected empirically from the crystal structure [1]. EGF amino acids 32-48, NCVVGYIGERCQYRDLK, with Cys-Cys disulfide bridge between Cys33 and Cys42 (Fig. 1B), was selected as a second

E. Wickstrom et al. / Three Dimensional Projection Environment

A

B

693

C

Figure 1. Potential EGF ligands, energy-minimized by molecular dynamics in Chem3D. A: cyclized EGF20-31 selected from crystal structure [1]; B: cyclized EGFR-binding EGF32-48 identified by cell binding [2]; C: EGFR-binding peptide identified by phage display [3].

candidate ligand, based on binding to EGFR on cells [2]. GE11, YHWYGYTPQNVI, an EGFR-binding peptide identified by phage display [3] (Fig. 1C), was selected as a third candidate ligand. The molecular dynamics docking pathway of EGF into the ligand binding pocket of EGFR was then calculated with ZDOCK, which performs rigid body docking calculations and determines their relative energies in implicit water with the appropriate dielectric constant, on Salk at the Pittsburgh Supercomputer Center (Fig. 2). Fluorescence microscopy and flow cytometry revealed that AlexaFluor 532EGF20-31 fails to enter cells that overexpress EGFR. On the other hand, AlexaFluor 532-GE11 enters cells whether or not they overexpress EGFR. Fullsize recombinant EGF failed to block uptake of Alexa Fluor 532-GE11. Chinese hamster ovary (CHO) cells, lacking EGFR, also took up AlexaFluor 532-GE11. This result implies that GE11 uptake is nonspecific. However, AlexaFluor 532EGF32-48 was taken up by Figure 2. Graphic image of EGF docking with EGFR from human AsPC1 pancreatic cancer an initial 20 Å out of the ligand binding pocket after 500 ps cells that overexpress EGFR, of molecular dynamics energy minimization, using examined by confocal ZDOCK on the Salk 64-bit high performance computer of fluorescence microscopy to the Pittsburgh Supercomputer Center. evaluate binding. 2.2. Volumetric Simulation with Haptic Feedback Volumetric simulation with haptic feedback manipulation of pancreatic cancer CT data from deidentified patient A (IRB approved) was carried out using SOFA (Fig. 3).

694

E. Wickstrom et al. / Three Dimensional Projection Environment

Figure 3. 3D projection of thorax of patient A. Advanced pancreatic tumor (right) extends from the head of the pancreas (left). The liver and kidneys are also shown, with the scalpel manipulating a kidney (bottom right).

Figure 4. SOFA-based projection of pancreatic tumor from Fig. 3, separated from other organs. Scalpel tool is manipulated for touch-andfeel variation.

We observed that the tissue texture parameters do not yet reflect the actual feel of each organ and lesion. The pancreatic tumor was isolated (Fig. 4). Young’s modulus and the Poisson ratio will be varied in a search for typical tumor texture and feel. PET data from uptake of [18F]fluorodeoxyglucose (FDG) were then overlaid on the CT data for patient A (Fig. 5). Intense FDG concentration was seen in the tumor, in agreement with the aggressive nature of the pancreatic malignancy. Bladder uptake reflects normal excretion of FDG. The next phase of haptic development will utilize co-registered PET/CT data to highlight aggressive tumor tissue for excision.

Figure 5. 3D projection of fused PET/CT data at 5 mm resolution of patient A. CT images of bones and organs are light, PET images are dark.

E. Wickstrom et al. / Three Dimensional Projection Environment

695

3. Discussion 3.1. EGFR Ligands for mRNA PET Imaging Agents EGF32-48 fragment calculations agreed with experimental cellular uptake, as well as non-binding by EGF20-31 fragment. We discovered that the energies of the entire kinetic docking pathway could be calculated in a single run, beginning with EGF translated 20 Å out of EGFR. The ZDOCK force fields were sufficient to draw EGF into the ligand binding pocket. This insight eliminates the need for separate calculations over a series of specific separations. 3.2. Volumetric Simulation with Haptic Feedback We also observed that the haptic manipulators felt heavier than a real scalpel or retractor. This suggests a future need for wireless glove manipulators. The next phase of haptic development will utilize co-registered PET/CT data to highlight aggressive tumor tissue for excision.

Acknowledgments This work is supported by USAMRMC/TATRC grant W81XWH-09-1-0577.

References [1]

[2] [3]

[4] [5]

[6]

[7]

[8]

K.M. Ferguson, M.B. Berger, J.M. Mendrola, H.S. Cho, D.J. Leahy, M.A. Lemmon, EGF activates its receptor by removing interactions that autoinhibit ectodomain dimerization, Mol Cell 11 (2003) 507517. J.S. Lee, M. Blick, Bioactive EGF peptides for promotion of tissue regeneration and cancer therapy, in: U.P. 5183805 (Ed.), Board of Regents, University of Texas, USA, 1993. Z. Li, R. Zhao, X. Wu, Y. Sun, M. Yao, J. Li, Y. Xu, J. Gu, Identification and characterization of a novel peptide ligand of epidermal growth factor receptor for targeted delivery of therapeutics, FASEB J 19 (2005) 1978-1985. J.R. Duncan, C.B. Glaiberman, Analysis of simulated angiographic procedures: part 1--capture and presentation of audio and video recordings, J Vasc Interv Radiol 17 (2006) 1979-1989. D.A. Case, T.E. Cheatham, 3rd, T. Darden, H. Gohlke, R. Luo, K.M. Merz, Jr., A. Onufriev, C. Simmerling, B. Wang, R.J. Woods, The Amber biomolecular simulation programs, J Comput Chem 26 (2005) 1668-1688. X. Tian, M.R. Aruva, K. Zhang, C.A. Cardi, M.L. Thakur, E. Wickstrom, PET imaging of CCND1 mRNA in human MCF7 estrogen receptor-positive breast cancer xenografts with an oncogene-specific [64Cu]DO3A-PNA-peptide radiohybridization probe, Journal of Nuclear Medicine 48 (2007) 16991707. T. Walter, D.W. Shattuck, R. Baldock, M.E. Bastin, A.E. Carpenter, S. Duce, J. Ellenberg, A. Fraser, N. Hamilton, S. Pieper, M.A. Ragan, J.E. Schneider, P. Tomancak, J.K. Heriche, Visualization of image data from cells to organisms, Nat Methods 7 S26-41. L. Soler, J. Marescaux, Patient-specific surgical simulation, World J Surg 32 (2008) 208-212.

696

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-696

Reality Graded Exposure Therapy with Physiological Monitoring for the Treatment of Combat Related Post Traumatic Stress Disorder: A Pilot Study1, 3 a,2

Dennis Patrick WOOD , Jennifer WEBB-MURPHYb, Robert N. MCLAYc, Brenda K. WIEDERHOLDa, James L. SPIRAd, Scott JOHNSTONb, Robert L. KOFFMANe, Mark D. WIEDERHOLDa and Jeff PYNEf a Virtual Reality Medical Center, 9565 Waples Street, San Diego, CA b Combat and Operational Stress Control Center, Naval Medical Center San Diego c Naval Medical Center San Diego, Mental Health Directorate d National Center for PTSD, Department of Veterans Affairs, Honolulu, HI e National Intrepid Center Of Excellence, Bethesda, MD f Center for Mental Healthcare Outcomes Research, Central Arkansas Veterans Healthcare System, Little Rock, AR

Abstract: A high percentage of Operation Iraqi Freedom (OIF) and Operation Enduring Freedom (OEF) combat veterans have been diagnosed with Posttraumatic Stress Disorder (PTSD) during and following their respective combat tours. Virtual Reality (VR) treatment has been documented as an exceptional treatment for anxiety disorders and specifically for PTSD. An Office of Naval Research (ONR) funded pilot study, completed by the Virtual Reality Medical Center and Naval Medical Center San Diego (NMCSD), investigated the use of Virtual Reality Graded Exposure Therapy (VR-GET) study with participants who had been diagnosed with PTSD following their combat deployments. A significant reduction in PTSD symptoms severity was noted. Implications for treatment with VR-GET and future research areas of investigation, including the use of VR-GET with smart phones and the internet, are suggested. KEYWORDS: Virtual Reality Therapy, Virtual Reality Graded Exposure Therapy (VR-GET), physiological monitoring, biofeedback, meditation, War on Terror, Posttraumatic Stress Disorder (PTSD), anxiety disorder, depression, habituation, Office of Naval Research (ONR), Navy Medical Center San Diego (NMCSD), Naval Hospital Camp Pendleton (NHCP) _____________________________________________ (1) The opinions expressed are the private ones of the authors and should not be considered approved or representative of the Navy Medical Department, the Office of Naval Research or the Department of Defense. This study was sponsored by the Office of Naval Research (ONR) Contract (#N00014-05-C-0136) to the Virtual Reality Medical Center San Diego, CA (2)

Corresponding Author: Virtual Reality Medical Center, 9565 Waples Street, San Diego, CA 92121; 858-642-0267; [email protected].

(3)

For information concerning VR-GET, access the VRMC web page (www.vrphobia.com) or http://www.signonsandiego.com/uniontrib/20080909/news_1n9virtual.html.

D.P. Wood et al. / Reality Graded Exposure Therapy with Physiological Monitoring

697

Introduction As of 2010, 2 million men and women have deployed to Iraq and Afghanistan in support of the War on Terror [1]. Several reports have estimated that between 4% and 45% of U.S. troops, that have deployed to Iraq and Afghanistan, have met the criteria for psychiatric disorders including PTSD [2 – 4]. Gahm & Lucenko [4] concluded that 44% of Soldiers that screened positive for mental health difficulties, at Madigan Army Medical Center, were diagnosed with PTSD. Disappointedly, studies have documented that the rate of PTSD is higher among troops who have been to Iraq more than once and PTSD rates would be exacerbated due to combat tour length [3, 5, 6]. Several reports have recommended that the Veterans Administration (VA) and Department of Defense (DoD) should aggressively develop early intervention strategies for preventing and treating PTSD [3, 5, 7, 8]. Recently, both the Veterans Administration and The Institute of Medicine concluded that only exposure therapy was recommended as treatment for PTSD [9, 10]. Virtual Reality Graded Exposure Therapy, with Physiological Monitoring (VR-GET) is a promising exposure therapy that has been documented as an evidenced-based treatment for anxiety disorders and specifically for PTSD [11 – 17]. In order to evaluate the use of VR-GET, with combat veterans who were diagnosed with PTSD, the Office of Naval Research (ONR) funded a VR-GET research study at Naval Medical Center San Diego (NMCSD) and Naval Hospital Camp Pendleton (NHCP). Previously, Wood et al [14] reported that 6 volunteer participants in the ONR funded VR-GET research study, which had successfully completed 10 sessions of VR therapy, experienced measurable reductions in the severity of their depression, PTSD and anxiety symptoms. Recently, an additional 6 volunteer participants, all of whom had also been diagnosed with PTSD secondary to their combat tour or tours, have completed 10 sessions of VR-GET. Following is the report of the outcome of VR-GET pilot study with these 12 participants.

1. Methods 1.1. Participants Twelve male volunteers met the DSM-IV-TR criteria for chronic PTSD [18] and these participants also met the study requirements for enrollment in the pilot phase of the VR-GET study (see Table 1). These 12 participants were originally diagnosed with PTSD, by a Navy psychiatrist, between January 2004 and August 2008. All of these participants were members of the United States Navy. Eleven of our participants were prescribed and were actively taking psychotropic medication prior to and following their enrollment in the pilot VR-GET study. All of our participants consulted with their Navy psychiatrist at least twice during the study period. All of our participants initiated VR-GET since March 2006. All VR-GET was provided at either Naval Medical Center San Diego (NMCSD) or Naval Hospital Camp Pendleton (NHCP).

698

D.P. Wood et al. / Reality Graded Exposure Therapy with Physiological Monitoring

Table 1. Demographics and current status of the Pilot Study participants (N=12): (a) USN: United States Navy; USNR: United States Naval Reserve; (b) mTBI: mild Traumatic Brain Injury; (c) HD: Honorable Discharge; EOS: end of Obligated Service; MCAS: Marine Corps Air Station, San Diego.

RANK (a)

AGE

Combat Tours

mTBI (b)

PSYCH MEDS

MEDICAL BOARD

STATUS (c)

04 – USN

49

multiple

No

Yes

No

E-7 – USN E-7USNR E-7 – USNR E-7 – USN E-6 – USNR E-6 – USNR E-5 – USNR E-5 – USN E-5 – USN E-4 – USN E-4 – USN

41

1

No

Yes

No

45

2

No

Yes

No

HD/EOS/working

44

1

No

Yes

No

HD/EOS/working

28

4

Yes

No

No

47

1 year

No

Yes

Yes

Active Duty/Naval Station HD/working

40

1

No

Yes

Yes

HD/working

37

2

No

Yes

Yes

HD/working

32

1

Yes

Yes

Yes

HD/college

31

1

Yes

Yes

No

HD/EOS/working

25

1

Yes

Yes

No

29

2

Yes

Yes

No

Active Duty/Naval Hospital Active Duty/Deployed: Iraq/ Medical Clinic, MCAS

Active Duty/Deployed: Iraq/Naval Hospital Retired/HD/working

1.2. Assessment Procedure The twelve participants were referred to VR-GET at Naval Medical Center San Diego (NMCSD) or Naval Hospital Camp Pendleton (NHCP) by a Navy psychiatrist, Navy clinical psychologist, Navy general medical officer or they were self-referred. Additional aspects of the Assessment Procedure have been previously described [13 – 15, 19]. 1.3. Clinical Measurement Instruments The Clinical Measurement Instruments have been previously described [13 – 15, 19]. 1.4. Equipment The Equipment has been previously described [13 – 15, 19].

D.P. Wood et al. / Reality Graded Exposure Therapy with Physiological Monitoring

699

1.5. Treatment A description of 10 individual VR-GET treatment sessions follows. Each VR-GET treatment session lasted 90-minutes and was provided once or twice a week during a 10 – 15 week treatment period [19]: Sessions 1 and 2 included meditation training and practice based on a twocomputer disc meditation training program (Jon Cabot Zinn & Andrew Weil, Meditation for Optimum Health, Sounds True, Boulder, CO) that was played for and given to the participants to take home to facilitate daily meditation practice. Meditation training was used to enhance emotional, cognitive, and physical relaxation during and in between treatment sessions. Additionally, during the first two sessions, the participants were asked to discuss their PTSD symptoms and they were asked to, “tell their stories about their sentinel (i.e., most traumatic) events during their combat tour or tours.” PTSD was discussed as a normal response to an abnormal situation. The VRGET treatment goal was to assist the participants gaining control over their intrusive thoughts and feelings and for them to learn to tolerate events or stimuli that currently bothered them (i.e., habituation). Lastly, the participants were taught the principles of attentional re-training. During 20 minutes of the VR-GET sessions 1 and 2, while the participants were immersed in the VR with the assistance of a Head Mounted Display (HMD),the participants’ physiological status (i.e., , heart rate, skin conductance, respiration rate and peripheral skin temperature) was monitored in order to discover what type of combat exposure (c.f., engaged in a fire fight) or positive virtual or mental images (c.f., returning to base during a combat deployment or positive family event back home) helped to increase or decrease the participants’ physiological arousal. When the participants reached maximum arousal, the combat stimuli were minimized and the participants were instructed to “cease fire” and to move to a “non-combat environment” using their skills with meditation and progressive relaxation and it was expected that their physiological arousal would decrease to levels indicating increased control and relaxation. The participants were also instructed in the assessment and reporting of their subjective level of arousal using Subjective Units of Discomfort or SUDs [11]. During VR-GET sessions 3 to 10, the participants applied their skills with meditation, increased physiological control, and attentional refocusing within the VR environment. The 90-minute treatment sessions were divided as follows: 20 - 25 minutes of review of treatment progress during the previous week; 20 minutes of additional training with meditation and attentional refocusing; 20 minutes of VR-GET; and 20 – 25 minutes of debriefing following VR-GET. The 20 minutes in the VR environment, with the HMD, included time to allow the participants to find to safe areas, move to the various areas in the VR combat environment during which arousal elements were increased, by the therapist, with the participants being instructed to utilize their meditation skills and attentional refocusing to calm their mind and body once “cease fire” was called by the therapist. As VR-GET progressed toward sessions 6 - 10, the participants would be able to initiate, on their own, their meditation and attentional refocusing skills to reduce their arousal levels while immersed in a VR combat environment with increasingly arousing VR combat elements. Typically, each VR-GET session included 2 – 3 cycles of low combat arousal, the graded or gradual addition of combat “elements” (i.e., sounds/sights of gunfire, sounds/sights of explosions, launching of RPGs, sounds and/or sights of helicopter(s), vehicle movements, participant returning fire, etc.), the “Exposure” to high arousal elements or environments and “cease fire”. Of note, the low arousal environments

700

D.P. Wood et al. / Reality Graded Exposure Therapy with Physiological Monitoring

would be evidenced by the participants’ low physiological arousal, as assessed by the physiological monitoring, and the report of low SUDs. The high arousal environments would be evidenced by high physiological arousal, as assessed by the physiological monitoring, and the report of high SUDs. During the debriefing period, the participants were asked about their VR experience, feedback was given to the participants about what was observed on the physiological monitoring, and the participants were encouraged to use their skills in the context of their everyday life. The participants were encouraged to practice meditation and attentional refocusing daily. Finally, plans for the subsequent VR-GET session were discussed.

2. Results The severity of PTSD and Depression, in the participants, both significantly reduced following VR-GET (See Figure 1). Additionally, the participants reported a measurable reduction in the severity of their anxiety following VR-GET (See Figure 1). For the main outcome measure, the PTSD Check List, Military Version (PCL-M), scores reduced from a mean 54 prior to treatment to a mean of 43 after the completion of 10 VR-GET sessions.

Figure 1. Results of the Patient Health Questionniare-9 (PHQ-9), PTSD Check List-Military Version (PCLM) and Beck Anxiety Inventory (BAI) at Pre-Treatment and Post-Treatment: PHQ-9 and PCL-M scores significantly reduced Pre-Treatment to Post-Treatment (t = 2.74, p = .02 & t = 3.21, p = .01, respectfully), (Total N = 12).

3. Discussion Participants who completed 10 sessions of Virtual Reality Graded Exposure Therapy (VRGET) showed statistically and clinically significant reductions on self-report measures of Depression and PTSD. Scores of anxiety decreased numerically, but the change was not statistically significant. Although causality cannot be determined from a single-group study, PTSD is a condition in which, after the first three months, a spontaneous recovery is unlikely. Thus it seems likely that VR-GET contributed to significant reductions in PTSD and depression and measurable reductions in the anxiety reported by our participants. Additionally, VR-GET is “safe” and was well tolerated by our participants. Our participants’ reductions in their PHQ-9, PCL-M and BAI scores further documented habituation. During the VR-GET sessions, our participants described becoming “immersed” and engaged in the graded exposures of the VR simulations.

D.P. Wood et al. / Reality Graded Exposure Therapy with Physiological Monitoring

701

Intriguingly, several of the VR-GET participants mentioned that, “I wished I had this training (i.e., meditation, biofeedback, attentional re-training and VR exposure) prior to my first combat deployment or between my combat deployments!” Moreover, these participants noted that, “I don’t think that my PTSD difficulties would have been as bad if I would have had this treatment before or between my combat deployments.” Wiederhold & Wiederhold [12] and Bouchard et al [24] have concluded that specific training (stress exposure training or stress inoculation training) prior to exposure to a stressor (i.e., combat) can help desensitize the individual to the stressful situation, avoiding a panic or “flight or fight” response to the stressful event. Training involving VR-GET and Stress Inoculation Training techniques may act to prevent long-term psychological reactions to stress, such as PTSD. Besides VR-GET being utilized to assist with the successful treatment of Veterans diagnosed with combat-related PTSD, a growing number of researchers and clinicians have been investigating the application of Virtual Reality Therapy to treat or assist with the treatment of various anxiety disorders, including PTSD, through smart phones and the internet [20 - 24]. Of note, the use of smart phones and the internet, to assist VR treatment, might compliment DoD’s and VHA’s mandate to expand PTSD and other treatment access and services for combat veterans [25, 26]. We must caution that there are limitations to the generalizability of our VR-GET results to other PTSD treatment populations at other medical centers, military or civilian. Our pilot study focused on selective psychological questionnaires and physiological assessment, utilized one VR-GET program and also employed a single meditation training stimulus. Another caution is that the participants’ documented habituation may have been due to repeated exposure to the same imaginal stimuli and not entirely to VR. Future research is needed to understand the active ingredients associated with habituation. In summary, VR-GET is a successful virtual reality exposure therapy and clinical protocol for treating military personnel diagnosed with combat-related PTSD. This PCbased system is both clinician and participant-friendly and can be implemented by skilled clinicians with appropriate training. Lastly, VR-GET should be considered part of a comprehensive treatment program for individuals diagnosed with combat-related PTSD.

Acknowledgements The authors thank Joan A. Wood for editorial and administrative assistance in preparing the manuscript.

References [1] C. W. Crane, News Notes: two million troops have deployed since 9/11, ASUS Navy (February 2010) 15. [2] C. W. Hoge, C. A Castro, S. C. Messer, D. McGurk, D. I. Cotting & R. L. Koffman, Combat duty in Iraq and Afghanistan, mental health problems and barriers to care. New England Journal of Medicine 351 (2004), 13 – 22. [3] T. Tanielian & L. H. Jaycox, Invisible wounds of war: psychological and cognitive injuries, their consequences and services to assist recovery, April 2008. Center for Military Health Policy Research, Rand Corporation: Santa Monica, CA. Retrieved from: www.rand.org/pubs/monographs/MG720/ (April 19, 2009).

702 [4]

D.P. Wood et al. / Reality Graded Exposure Therapy with Physiological Monitoring

G. A. Gahm & B. A. Lucenko, Screening soldiers in outpatient care for mental health concerns, Military Medicine 173 (2008), 17 – 24. [5] E. C. Ritchie & M. Owens, Military issues. Psychiatric Clinics of North America 27 (2004), 459 – 471. [6] Y. C. Shen, J. Arkes, B. W. Kwan, L. Y. Tan, T. V. Williams, Effects of Iraq/Afghanistan deployments on PTSD diagnosis for still active personnel in all four services, Military Medicine 175 (2010), 763 – 769. [7] R. Forsten & B. Schneider, Treatment of the stress casualty during Operation Iraqi Freedom One. Psychiatric Quarterly 76 (2005), 343 – 350. [8] M. D. Howard & R. P. Cox, Collaborative intervention: a model for coordinated treatment of mental health issues within a ground combat unit. Military Medicine 173 (2008), 339 – 348. [9] Department of Veterans Affairs Press Release, dated October 18, 2007. VA agrees with key points about PTSD treatment in new Institute of Medicine Report (www.va.gov). [10] A. O. Berg, Chair, Committee on Treatment of Posttraumatic Stress Disorder, treatment of posttraumatic stress disorder: an assessment of the evidence. Institute of Medicine, The National Academies Press, Washington, D.C., 2008. [11] B. K. Wiederhold & M. D. Wiederhold, Virtual reality therapy for anxiety disorders, American Psychological Association Press, Washington, D.C., 2004. [12] B. K. Wiederhold & M. D. Wiederhold, Virtual reality for posttraumatic stress disorder and stress inoculation training, Journal of CyberTherapy & Rehabilitation 1 (2008), 23– 35. [13] B. K. Wiederhold, A. H. Bullinger & M. D. Wiederhold MD, Advanced technologies in military medicine. In MJ Roy (Ed.), Novel Approaches to the Diagnosis and Treatment of Posttraumatic Disorder, 148 – 160. IOS Press, Amsterdam, 2006 [14] D. P. Wood, J. A. Murphy, K. B. Center, C. Russ, R. L. McLay, D. Reeves, J. Pyne, et al., Combatrelated post-traumatic stress disorder: a multiple case report using virtual reality graded exposure therapy with physiological monitoring. In Westwood JD, Haluck RS, Hoffman HM et al (Eds.), 556 – 561. Medicine Meets Virtual Reality 16: Amsterdam: IOS Press, Amsterdam, 2008. [15] D. P. Wood, J. Webb-Murphy, K. B. Center, R. N. McLay, R. L. Koffman, S. Johnston, J. Spira, et al, Combat-related posttraumatic stress disorder: a case report using virtual reality graded exposure therapy with physiological monitoring with a female Seabee. Military Medicine 174 (2009), 1215 – 1222. [16] R. N. McLay, C. McBrien, M. D. Wiederhold & B. K. Wiederhold, Exposure therapy with and without virtual realtiy to treat PTSD while iin the combat theater: a parallel case series, CyberPsychology, Behavior and Social Networking 13 (2010), 37 – 42. [17] R. Twors, S. Szymanska & S. Iinicki, A soldier suffering from PTSD, treated by controlled stress exposition using virtual reality and behavioral training, CyberPsychology, Behavior and Social Networking 13 (2010), 103 – 107. [18] American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, American Psychiatric Association, Washington, D.C., 1994. [19] J. L. Spira, B. K. Wiederhold, J. Pyne & M. D. Wiederhold, Treatment Manual: virtual reality physiological monitored, graded exposure therapy in the treatment of recently developed combatrelated PTSD, Virtual Reality Medical Center, San Diego, CA, 2007. [20] J. L. Mosso, A. Gorini, G. de La Verda, T. Obrador, A. Almazan, D. Mosso, J. J. Nieto & G. Riva, Virtual reality on mobile phones to reduce anxiety in outpatient surgery, In J. D. Westwood, S. W. Westwood, R. S. Haluck, H. M. Hoffman, et al (Eds.), Medicine Meets Virtual Reality 17, 195 – 200, IOS Press, Amsterdam, 2009. [21] V. Germain, A. Marchand, S. Bouchard, S. Guay, & M. S. Drouin, Assessment of the therapeutic alliance in face-to-face or videoconference treatment for posttraumatic stress disorder. Cyberpsychology, Behavior and Social Networking 13 (2010), 29 – 36. [22] G. Riva, S. Raspelli, D. Algeri, F. Pallavicini, A. Gorini, B. K. Wiederhold & A. Gaggiloi, Interreality in practice: bridging virtual and real worlds in the treatment of posttraumatic stress disorders, CyberPsychology, Behavior and Social Networking 13 (2010), 55 – 66. [23] M. D. Wiederhold & B. K. Wiederhold, Virtual reality and interactive simulation for pain distraction, CyberTherpay & Rehabilitation 1 (2010), 14 – 19. [24] S. Bouchard, O. Baus, F. Bernier & D. R. McCreary, Selection of key stressors to develop virtual environments for practicing stress management skills with military personnel prior to deployment, CyberPsychology, Behavior and Social Networking 13 (2010), 83 – 94. [25] J. M. Hames, Army reducing stigma of psychological care, offering telepsychiatry, Army News Service, Washington, D.C., May 7, 2008. Retrieved from http://www.army.mil/-news/2008/05/9013-armyreducing-stigma-of-psychological-care-offering-telepsychiatry (March 6, 2010). [26] Department of Veterans Affairs News Release, Secretary Shinseki Announces $215 million in Projects for Rural Veterans (May 21, 2009). Retrieved from: www.va.gov (October 17, 2010).

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-703

703

Applications of Tactile Feedback in Medicine Christopher WOTTAWAa,b,1, Richard FANa,b, James W. BISLEYa,b,d, Erik P. DUTSONa,c , Martin O. CULJATa,c, and Warren S. GRUNDFESTa,b a UCLA Center for Advanced Surgical and Interventional Technology (CASIT) b Biomedical Engineering IDP, cDepartment of Surgery, dDepartment of Neurobiology

Abstract. A tactile feedback system has been developed in order to provide augmentative sensory feedback for a number of medical applications. The key component to the system is a pneumatic balloon-based tactile display, which can be scaled and adapted for a variety of configurations. The system also features pneumatic and electronic control system components, a commercial force sensor modified to fit the desired application. To date, this technology has been successfully applied to medical robotics, minimally invasive surgery, and rehabilitation medicine. Keywords. Tactile feedback, piezoresistive sensor, pneumatic balloon actuator, robotic surgery, sensory augmentation, rehabilitation

1. Introduction Tactile feedback systems utilize principles of sensory substitution and sensory augmentation to provide previously unavailable or attenuated sensory information,1 and have found applications in surgical and industrial robotics, and virtual reality.2 Tactile feedback systems may also have application to rehabilitation systems for tasks such as restoration of fine motor control in the hands, balance correction, and lower-limb sensory restoration.3-5 1.1. Tactile Sensation Tactile sensations are generated by sensory receptors on the skin that are sensitive to mechanical deformations. Receptors are classified by size of receptive field (type I is small, type II is large) and response characteristics (slowly adapting (SA) vs. fast adapting (FA)).6 Frequency responses for SAI, SAII, FAI, and FAII are 0.4-10 Hz, 0.4100 Hz, 2-40 Hz, and 100-1kHz respectively.7 Attempts have been made to associate FA I with Meissner corpuscles, SA I with the Merkel disks, SA II with Ruffini organs and the FA II with Pacinian corpuscles. Mechanoreceptors in the hairy skin are dominated by type II receptors, namely Pacinian Corpuscles and Ruffini Endings.8 Receptive fields of Pacinian Corpuscles are larger than 20 mm, and those of Ruffini Endings are larger than 10 mm. The spatial resolutions of Pacinian Corpuscles and Ruffini Endings are about 2 cm and 1 cm respectively.9 Using this knowledge of sensory physiology, our team has designed a feedback actuator modality to optimally provide sensations that utilize the body’s own neural architecture.

704

C. Wottawa et al. / Applications of Tactile Feedback in Medicine

2. Methods and Materials A tactile feedback system has been proposed that features tactile actuators and force sensing elements. A control system, consisting of both electrical and pneumatic components, converts the forces measured at the sensor to pressures applied by the actuator. The pneumatic subsystem uses an application-specific arrangement of fastswitching on/off solenoid valves to route pressure streams to actuator elements. The sensor and actuators are described in more detail in the following sections. 2.1. Pneumatic Balloon Actuators Prior actuation mechanisms for haptic feedback include visual, auditory, electrical, and tactile stimulation.10-12 Of these modalities, electrical and tactile stimuli have been the most widely explored.12 Electrical stimulation uses either non-invasive surface electrodes (electrocutaneous or transcutaneous) or more invasive subcutaneous and intraneural implantation methods. While electrical stimulation is advantageous in its ability to provide selective stimulation, biocompatibility and repeatability issues limit practical implementation. 13, 14 Several tactile feedback actuation schemes have been developed by other groups as non-invasive means to provide sensory feedback, including motor driven actuation, vibrotactile displays, piezoelectric actuators, shape memory alloys, rheological fluids, and pneumatically driven actuators. 15 Limitations with these designs can include adaptation effects, low force output, slow response time, or large and bulky mechanical configurations. While vibrotactile actuators can operate with high spatial resolution, stimulation of the fast adapting sensory receptors has a desensitizing effect.12 Motivated by this, a pneumatically driven actuator modality has been adopted by our group to attenuate sensory adaptation effects. This technique has been previously explored in various applications.16, 17 Our actuator design was developed in order to maximize the human perceptual response, achieve high mechanical performance, and to minimize the system’s overall physical footprint. Pneumatic balloon have been developed that provide pressure stimuli using hemispherical silicone balloons, targeting the SA mechanoreceptors through constant deformation. The actuators are composed of a macromolded polydimethylsiloxane (PDMS) substrate housing application-specific pneumatic channels and a thin film silicone membrane (Figure 1).18

Figure 1. Pneumatic balloon actuator with 0% to 100% inflation

Pneumatic balloon actuators have the advantage of large force output, large deflection, rapid response time and low mass on the end factor.18 Fatigue tests were performed and found the balloon actuators to have negligible hysteresis over at least 150,000 inflation-deflation cycles.19

C. Wottawa et al. / Applications of Tactile Feedback in Medicine

705

2.2. Sensors Tactile sensing methods employ a vast array of configurations, including piezoresistive, 20,21 piezoelectric,22 and capacitive23, 24 sensing modalities. Piezoresistive sensors make use of semiconductive materials’ property of changing electrical resistance when under mechanical strain. Piezoelectric sensors generate a voltage directly when under an applied load. However, with the application of a constant load, sensor output quickly decays, most suited towards measuring dynamic force, rather than static force. Capacitive force sensors most often utilize a spring-like dielectric positioned between two conducting plates, so that applied force compresses the dielectric, changing the capacitance. Limitations with capacitive sensors include susceptibility to noise and limits in spatial resolution.25 Additional methods of measuring force include optical sensors,26 inductive sensor technologies,25 and fluid filled elastomers.27 For our tactile feedback system, a commercial single-element piezoresistive force sensor (FlexiForce, Tekscan) was chosen due to its thin-film profile (208 μm), small diameter (10 mm), high linearity within the desired force range (0 to 25 N), low cost, and good static and dynamic responses. Sensors were characterized using an Instron mechanical loading system (Instron 5544), which was programmed to perform 20 loading cycles at the speed of 1 cycle/min over a force range of 0 to 25 N.28 The sensor demonstrated a linear (R2 > 0.99) decrease in resistance with increases in force over this range.

3. Results Prior work in the CASIT labs has focused on the development of the open-ended tactile feedback system architecture. Considerations were given to ease of system expansion to accommodate many elements, simple intuitive hardware and software interfaces and robust construction to facilitate use outside of an engineering lab. The following represent several applications of this tactile feedback technology in fields of clinical interest. 3.1. Robotic Surgery Robotic surgical systems provided surgeons with modifications such as increased degrees of freedom to better mimic natural hand and wrist gestures, stereoscopic video displays, and scaling of surgical gestures to enable precision movements. 29 The complete lack of haptic feedback is often cited as the single greatest technical disadvantage of robotic surgical systems.30 The tactile feedback system was integrated with the da Vinci surgical system, so that proportional forces could be provided directly to the fingertips of the operating surgeon (Figure 2).31 Mechanical tests and human psychophysical tests were performed and determined the most effective actuator architecture for fingertip stimulation.32 Several system evaluations were performed using this configuration, and our group discovered for the first time that tactile feedback produced a significant reduction in grip forces during simple surgical tasks.33

706

C. Wottawa et al. / Applications of Tactile Feedback in Medicine

Figure 2. Robotic surgery tactile feedback system integration, including mounting of sensors and actuators.

3.2. Laparoscopic Surgery Minimally invasive laparoscopic surgery offers advantages over open procedures, such as improved recovery time, decreased trauma, and decreased hospital expenses. Restoring the attenuated tactile sense may allow surgeons to better control grip force. A modified laparoscopic grasper was developed with an integrated tactile feedback system (Figure 3). 34 Preliminary tests have shown successful operation of the system with latency less than 50 ms, high actuation pressures (15 PSI), and high perceptual accuracy of the balloon-based stimuli (> 90%). Ongoing studies will help determine the benefit of tactile feedback in non-robotic laparoscopic surgery.

Figure 3. Laparoscopic Grasper with Tactile Feedback System

3.3. Lower Limb Prosthesis Lower-limb sensory loss as a result of amputation results in sub-optimal movement and an increased incidence of injury. While the adoption of lower-limb prostheses can support movement, the fundamental problem of sensory loss in the prosthetic limb continues to exist. The designed system provides sensory feedback by sensing the prosthetic plantar loads and providing corresponding tactile stimuli to the residual limb. These stimuli allow for the perception of previously unavailable tactile sensory information, potentially improving gait and balance performance during rehabilitation. Initial work has demonstrated the feasibility of the approach in normal subjects and lower-limb amputees.5,35

Figure 4. Amputee subject wearing a tactile feedback system.

C. Wottawa et al. / Applications of Tactile Feedback in Medicine

707

Biomechanical gait studies to quantify walking kinetics, kinematics and temporal spatial parameters are on-going (Figure 4). 3.4. Tactile Glove The stiffness of the pressurized space suit limits the dexterity and flexibility of the astronauts' fingers, attenuating tactile sense of external objects. Providing tactile feedback to the fingertips may potentially improve dexterity. Tactile feedback components were integrated into a two glove system in order to provide users with a remote sense of touch.36 Forces detected on one glove, either worn by a user or mounted on a robotic hand, are translated to discrete pressure levels at the fingers of a remote user. System tests demonstrated that users could accurately identify the correct finger and detect three simultaneous finger stimuli with 99.3% and 90.2% accuracy, respectively.37 The glove-based tactile feedback system may have application to virtual reality, rehabilitation, remote surgery, medical simulation, robotic assembly, and military robotics. 3.5. Torso-worn Balance Feedback Balance disorders result in increased incidence of injury and falls. A tactile feedback vest was designed to provide augmentative sensory feedback to restore knowledge of postural sway to individuals with balance disorders. Pneumatic actuators mounted on a tactile vest provide multi-directional sensory feedback. Accelerometers measure linear and angular acceleration of the upper torso. This work represents a first step in the design of a haptic feedback device for real-time balance correction.38 Future work will focus on working with patients with balance disorders to complete the system development and determine system efficacy. 3.6. Braille Display The refreshable Braille display allows the blind to dynamically interface with standard computer and cell phone displays. Displays have been produced using various modalities including solenoid, piezoelectric and pneumatic mechanisms.39,40 However, many existing Braille display designs are costly or lack robust performance. The pneumatic balloon actuators were modified into a refreshable Braille display. 41 Actuator elements were 1.45 mm in diameter and spaced 0.89 mm apart, in a 4 x 2 display to meet the format of Standard American Braille. Perceptual testing was performed to determine the feasibility of the approach using a single blind human subject. The subject was able to detect randomized Braille letters rapidly generated by the actuator with 100% character detection accuracy.

4. Conclusion A tactile feedback system was integrated with a number of applications in medical robotics, minimally invasive surgery, and rehabilitation medicine. Continuing evaluations of this technology will evaluate benefit and system efficacy through in vivo

708

C. Wottawa et al. / Applications of Tactile Feedback in Medicine

animal trials, biomechanical gait and balance studies, and kinesiological analysis in clinical environments.

5. Acknowledgements The authors would like to thank E. Carmack Holmes and Cheryl Hein at CASIT, along with CASIT alumnus Miguel Franco, Adam Feinman, Adrienne Higa, Aaron King, Ji Son and Steven Wu. The authors most gratefully appreciate funding provided by the Telemedicine and Advanced Technology Research Center (TATRC) / Department of Defense under award number W81XWH-07-1-0672. (1)

References [1]

[2]

[3] [4] [5]

[6]

[7] [8]

[9] [10] [11] [12]

[13]

[14] [15] [16] [17]

B.T. Bethea, A.M. Okamura, M. Kitagawa, T.P. Fitton, S.M. Cattaneo, V.L. Gott, W.A. Baumgartner, D.D. Yuh, "Application of haptic feedback to robotic surgery", Journal of Laproendoscopic & Advanced Surgical Techniques, Vol. 14 No. 3, pp. 191-195, 2004 M.C. Cavusoglu, W. Williams, F. Tendick and S.S. Sastry, “Robotics for Telesurgery: Second Generation Berkeley/UCSF Laparoscopic Telesurgical Workstation and Looking towards the Future Applications.” Industrial Robot: An International Journal, vol. 30, no.1, pp. 22-29, 2003. M. Bouzit, G. Popescu, G. Burdea, R. Boian, "The Rutgers Master II-ND Force Feedback Glove,"10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp.145, 2002. M. Nakamura, L. Jones, “An actuator for the tactile vest – a torso-based haptic device,” Proc. IEEE Symp. Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2003. R.E. Fan, M.O. Culjat, C.H. King, M.L. Franco, R.Boryk, J.W. Bisley, E. Dutson, W.S. Grundfest, "A Haptic Feedback System for Lower-Limb Prostheses," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol.16, no.3, pp.270-277, June 2008 K.O. Johnson and T. Yoshioka, “Neural Mechanisms of Tactile Form and Texture Perception,” The Somatosensory System: Deciphering the Brain’s Own Body Image, R.J. Nelson, ed., CRC Press, 2002, pp. 73-102 R.S. Johansson, A.B. Vallbo, “Tactile sensory coding in the glabrous skin of the human hand," Trends Neuroscience vol. 6, 1983, pp. 27-31. K.S. Hale, K.M. Stanney, "Deriving haptic design guidelines from human physiological, psychophysical, and neurological foundations," Computer Graphics and Applications, IEEE , vol.24, no.2pp. 33- 39, Mar-Apr 2004 R.W. Cholewiak and A.A. Collins, “Sensory and physiological bases of touch,” in The Psychology of Touch, M. A. Heller and W. Schiff, Eds. Hillsdale, NJ: Erlbaum, 1991, ch. 2, pp. 23–60. B. Petzold, M.F. Zaeh, B. Faerber, B. Deml, H. Egermeier, J. Schilp and S. Clarke. “A Study on Visual, Auditory, and Haptic Feedback for Assembly Tasks.” Presence, vol. 13 no. 1, pp. 16-21, 2004. C.Y. Shing, C.P. Fung, T.Y. Chuang, I.W. Penn, J.L. Doong, “The study of auditory and haptic signals in a virtual reality-based hand rehabilitation system.” Robotica, vol. 21 pp. 211–218, 2003. K.A. Kaczmarek, J.G. Webster, P. Bach-y-Rita, W.J. Tompkins. “Electrotactile and vibrotactile displays for sensory substitution systems," IEEE Transactions on Biomedical Engineering, vol.38, no.1 pp.1-16, Jan 1991. X. Navarro, T.B. Krueger, N.Lago, S.Micera, T.Stieglitz, P.Dario, “A critical review of interfaces with the peripheral nervous system for the control of neuroprostheses and hybrid bionic systems” Journal of the Peripheral Nervous System vol. 10, no. 3, pp.229-258, 2005 W.M. Grill, J.T. Mortimer, “Electrical properties of implant encapsulated tissure” Annals of Biomedical Engineering vol. 22, pp. 23-33, 1994. J. Rosen, B. Hannaford, R. Satava. Surgical Robotics – System Applications and Visions, Springer Publishing, 2010 (in press) C. Grosjean, G.B. Lee, W. Hong, Y.C. Tai, and C.M. Ho, “Micro Balloon Actuators for Aerodynamic Control” Proceedings of IEEE Micro Electro Mechanical Systems, pp. 166-171, 1998 G. Yuan; X. Wu; Y.K. Yoon; M.G. Allen. “Kinematically-Stabilized Microbubble Actuator Arrays,” MEMS 2005. 18th IEEE International Conference. p. 411 – 414, 2005.

C. Wottawa et al. / Applications of Tactile Feedback in Medicine

709

[18] C.H. King, M. Franco, M. O. Culjat, J. W. Bisley, E. Dutson, W. S. Grundfest, “Fabrication and characterization of a balloon actuator array for haptic feedback in robotic surgery,” ASME J. Medical Devices, submitted for publication. [19] C.H. King, M.O. Culjat, M.L. Franco, J.W. Bisley, E. Dutson, W.S. Grundfest. “Optimization of pneumatic balloon tactile display for robotic surgery based on human perception.” IEEE Transactions on Biomedical Engineering 55(11):2593-2600, 2008 [20] Tekscan: Tactile Pressure Measurement, Pressure Mapping Systems, and Force Sensors and Measurement Systems. http://www.tekscan.com/ Accessed Sept. 23, 2009. [21] N. Kattavenos, B. Lawrenson, T.G. Frank, M.S. Pridham, R.P. Keatch, A. Cuschieri “Force-sensitive tactile sensor for minimal access surgery.” Minim Invasive Ther Allied Technol 1:42-6, 2004. [22] A. Mirbagheri, J. Dargahi, S. Narajian, F.T. Ghomshe “Design, Fabrication, and Testing of a Membrane Piezoelectric Tactile Sensor with Four Sensing Elements.” American Journal of Applied Sciences 4(9):645-652, 2007. [23] Pressure Profile Systems. http://www.pressureprofile.com/ Accessed Sept. 23 2009 [24] J.N. Palasagaram, R. Ramadoss. “MEMS Capacitive Pressure Sensor Array fabricated Using Printed Circuit Processing Technologies.” 32nd Annual Conference of IEEE Industrial Electronics Society 2005 [25] H.R. Nicholls, M.H. Lee, "A survey of robot tactile sensing technology", Int. J Robotics Research;8(3):3-30, 1989 [26] S. Hirose, K. Yoneda. Development of optical 6-axial force sensor and its signal calibration considering non-linear interference. IEEE Computer Society Press, 1990, p. 46-53. [27] N. Wettels, V.J. Santos, R.S. Johansson, G.E. Loeb. “Biomimetic Tactile Sensor Array.” Advanced Robotics 22; 829-849, 2008 [28] C.H. King, M.O. Culjat, M.L. Franco, J.W. Bisley, G.P. Carman, E.P. Dutson, W.S. Grundfest “A multi-element tactile feedback system for robot-assisted minimally invasive surgery.” IEEE Transactions on Haptics, 2(1):52-56, 2009 [29] B. Jacob, M. Gagner, “Robotics and General Surgery” Surgical Clinics of North America, 83 (6); 1405-19, Dec. 2003 [30] A.R. Lanfranco, A.E. Castellano, J.P. Desai, W.C. Meyers. “Robotic Surgery : a current perspective.” Annals of Surgery, 239 (1): 14-21, 2004 [31] W.S. Grundfest, M.O. Culjat, C.H. King, M.L. Franco, C. Wottawa, C.E. Lewis, J.W. Bisley, E.P. Dutson, "Development and testing of a tactile feedback system for robotic surgery," Proceedings of Medicine Meets Virtual Reality 17: NextMed: Design for the well being. 103-108, Jan. 2009 [32] C.H. King, M.O. Culjat, M.L. Franco, J.W. Bisley, E. Dutson, W.S. Grundfest. “Optimization of pneumatic balloon tactile display for robotic surgery based on human perception.” IEEE Transactions on Biomedical Engineering 55(11):2593-2600, 2008 [33] C.H. King, M.O. Culjat, M.L. Franco, C.E. Lewis, E.P. Dutson, W.S. Grundfest, J.W. Bisley “Tactile feedback induces reduced grasping force in robotic surgery.” IEEE Transactions on Haptics, 2009. [34] C. Wottawa, R.E. Fan, C.E. Lewis, B. Jordan, M.O. Culjat, W.S. Grundfest, E.P. Dutson "Laparoscopic grasper with an integrated tactile feedback system" Proceedings of ICME/IEEE International Conference on Complex Medical Engineering, 9-11 April 2009, Tempe, AZ, 1-5, 2009. [35] R.E. Fan, C. Wottawa, R.J. Boryk, T.C. Sander, M.P. Wyatt, W.S. Grundfest, M.O. Culjat. ”Pilot testing of a tactile feedback rehabilitation system on a lower-limb amputee,” ICME/IEEE International Conference on Complex Medical Engineering, 9-11 April 2009, Tempe, AZ, 2009. [36] M.O. Culjat, J. Son, R.E. Fan, C. Wottawa, J.W. Bisley, W.S. Grundfest, E.P. Dutson. “Remote Tactile Sensing Glove-Based System.” Proceedings EMBC, 2010 [37] J. Son, R.E. Fan, C. Wottawa, W.S. Grundfest, E.P. Dutson, M.O. Culjat, "Tactile sensing glove for extravehicular activity," International Conference on Environmental Systems, 12-16 July 2009, Savannah, GA, 2009-1-2498. [38] S.W. Wu, R.E. Fan, C.R. Wottawa, W.S. Grundfest, M.O. Culjat. ”Torso-based tactile feedback system for patients with balance disorders,” Proceedings 2010 Haptics Symposium, 25-26 March 2010. [39] H. Fischer, B. Neisius, and R. Trapp, “Tactile feedback for endoscopic surgery.” Interactive Technology and the New Paradigm for Health Care, K. Morgan, R. M. Satava, H. B. Seiburg, R. Mattheus, and J. P. Chris, Eds. Washington, DC: IOS, 1995. [40] X. Wu, H. Zhu, S.-H. Kim, M.G. Allen, “A Portable Pneumatically-actuated Refreshable Braille Cell,” Digest Tech. Papers Transducers’07Conference, Lyon, 2007, p. 1409-1512. [41] R.E. Fan, A. Feinman, C. Wottawa, C.H. King, M.L. Franco, E.P. Dutson, W.S. Grundfest, M.O. Culjat, "Characterization of a pneumatic balloon actuator for use in refreshable Braille displays," Proceedings of Medicine Meets Virtual Reality 17: NextMed: Design for/the well being, 2009

710

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-710

Needle Insertion Simulation by Arbitrary Lagrangian-Eulerian Method Satoshi YAMAGUCHIa, 1 , Koji SATAKEb, Shigehiro MORIKAWA c, Yoshiaki SHIRAIb, Hiromi T. TANAKAb a Graduate School of Dentistry, Osaka University, Japan b Department of Human and Computer Intelligence, Ritsumeikan University, Japan c Biomedical MR Science Center, Shiga Medical University, Japan

Abstract. In this paper, we performed needle insertion simulation considering needle tip shape by Arbitrary Lagrangian-Eulerian (ALE) method. ALE method is suitable for the large deformation and a fracture. To evaluate developed model, we compared the needle deflection between experimental results and simulation results. As a result, errors in each needle between both results were less than 3 mm. Keywords. Needle insertion simulation, Arbitrary Lagrangian-Eulerian method, percutaneous minimally invasive therapy

Introduction Needle insertion into soft tissue is an important technique for percutaneous minimally invasive therapy. Surgeons are required to place tip of a needle at a target such as a tumor into liver precisely. However, even for expert surgeons it is not easy because of a needle deflection depend on various needle tip shapes [1]. The aim of our study is to predict the needle deflection considering the needle tip shape for surgical planning. In previous study, we have been performed a needle insertion simulation by Finite Element method (Lagrangian method) and confirmed a puncture force is not much difference compare with experimental results [2]. However, the needle deflection could not be predicted precisely. One of the reasons is to use Lagrangian method which is not much of simulations with large deformation. To solve these problems, we applied Arbitrary Lagrangian-Eulerian (ALE) method which is suitable for the large deformation and a fracture. As an experiment to validate simulation result, we compared needle deflection between experimental results and simulation results.

1. Methods & Materials Simulation model consists of a needle with bevel tip, a soft tissue, and an air. The soft tissue is fixed on a lower surface. A size of the needle, the soft tissue and the air are φ1.6×150, 32×10×125, and 32× 10×5 mm respectively. Angles of bevel needle tip are 30, 45, and 60 deg. Mesh of the needle is generated by rectangular elements as shell. 1

Corresponding Author: Satoshi Yamaguchi, Ph.D., Graduate School of Dentistry, Osaka University, 1-8 Suita, Osaka 565-0871, Japan; E-mail: [email protected]

S. Yamaguchi et al. / Needle Insertion Simulation by Arbitrary Lagrangian-Eulerian Method

711

And mesh of the soft tissue and the air are generated by hexahedral elements as solid. A number of meshes are totally 297,014 elements. The most small mesh size is 0.375 mm. Physical parameters of the needle are represented by Young’s modulus E, Poisson ratio ν, and density ρ. Physical parameters of the soft tissue are represented by shear modulus μ, exponent α, Poisson ratio ν, and density ρ. E of the needle is 80000 MPa. ν of the needle and the soft tissue are 0.3 and 0.495. ρ of the needle and the soft tissue are 8.5e-8 and 1.244e-8 ton/mm2. μ1, μ2, and μ3 of the soft tissue are 0.05, 6.4, 0.01 MPa. α1, α2, and α3 of the soft tissue are 0.01, 0.003, and 0.01. To evaluate this model, we performed an experiment to measure needle deflection by a needle insertion device. As a material of the needle and the soft tissue, we use a copper and a 3% agar (Agar powder, NaN3) respectively. The needle is inserted into the agar at 2 mm/sec. A penetration depth is 100 mm. After insertion, Photos from lateral side is taken by a digital camera to measure the needle deflection. Finally, we compare the simulation results with the experimental results.

2. Results

Figure 1. Comparison of needle deflections

Figure 1 shows a comparison of needle deflections between the experimental results and the simulation results. Errors in each needle between both results were 2.762, 2.106, and 1.661 mm respectively. Considering a size of tumor 2.5 cm [3], these errors were within the allowable limits. Figure 2 shows shear stress distribution in a cross section surface along x-z plane. The smaller the angles of bevel tip the greater the needle deflection. As shown Figure 3, shear stress in each model were difference distributions correspond to the needle tip shape.

712

S. Yamaguchi et al. / Needle Insertion Simulation by Arbitrary Lagrangian-Eulerian Method

Figure 2. Comparison of needle deflection and maximum shear stress distribution (Left: 30[deg], Middle: 45[deg], Right: 60[deg]) at 100 mm.

Figure 3. Close up of maximum shear stress distribution around a needle tip (Left: 30[deg], Middle: 45[deg], Right: 60[deg]) at 7mm.

3. Conclusions We performed needle insertion simulation by Arbitrary Lagrangian-Eulerian method. For an effective surgical planning, we will perform a simulation by using clinical needle.

References [1] A.M. Okamura, et al., Force modeling for needle insertion into soft tissue, IEEE Transactions on Biomedical Engineering, (2004) , 1707-1716. [2] S. Yamaguchi, et al., 3-Dimensional finite element method based needle insertion simulation considering needle tip shape, In Proc. of Int. J CARS, 4 (2009), S294-295. [3] H. Hur, et al., Comparative study of resection and radiofrequency ablation in the treatment of solitary colorectal liver metastases, Am J Surg. 197 (2009), 737-739.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-713

713

Clinical Performance of Dental Fiberscope Image Guided System for Endodontic Treatment Yasushi YAMAZAKI a, Takumi OGAWA b, Yuko SHIGETA b, Tomoko IKAWA b, Shintaro KASAMA b, Asaki HATTORI d, Naoki SUZUKI d, Takatsugu YAMAMOTO c, Toshiko OZAWA a, Takashi ARAI a a Department of Periodontics and Endodontics, b Department of Fixed Prosthodontics , c Department of Operative Dentistry, Tsurumi University School of Dental Medicine d Institute for High Dimensional Medical Imaging, Jikei University School of Medicine

Abstract. We developed a dental fiberscope that can be navigated. As a result we are able to better grasp the device position relative to the teeth, aiming at the lesion more precisely. However, the device position and the precise target setting were difficult to consistently ascertain. The aim of this study is to navigate the position of tip of the dental fiberscope fiber in the root canal with our navigation system. A 3D tooth model was made from the raw dental CT data. In addition, the optical position of the measurement device, OPTOTRAK system was used for registration of the 3D model and actual teeth position and to chase the scope movement. We developed exclusive software to unify information. We were subsequently able to precisely indicate the relation of the position between the device and the teeth on the 3D model in the monitor. This allowed us to aim at the lesion more precisely, as the revised endoscopic image matched the 3D model. The application of this endoscopic navigation system could increase the success rate for root canal treatments with recalcitrant lesion. Keywords. Navigation system, Fiberscope, Laser

Introduction We previously reported the development of a dental fiberscope system having channels for laser and flooding, in addition to a channel for the image fiber1). The system enabled us to treat the lesion inside the root canal while simultaneously observing it through the dental fiberscope. However, due the influence of focus depth and the angle changes of the scope, the comprehension of the device position and the precise target setting was difficult to consistently ascertain. In order to overcome these shortcomings, we developed a dental fiberscope system that could be navigated2-4). With the navigation system, we are able to better grasp the device position relative to the teeth, aiming at the lesion more precisely. The aim of this study was to evaluate the clinical performance of the dental fiberscope navigation system.

714

Y. Yamazaki et al. / Clinical Performance of Dental Fiberscope Image Guided System

Materials and Methods Fig.1 shows an overview of our navigation system for the dental fiberscope. The fiberscope consists of image fibers (6,000 pixels, focus depth: 5 mm, field corner: 70º) together with light guides and working channels for other dental instruments and flooding. The outer diameter of the fiberscope needle is 1.1 mm. We observed the image through the dental fiberscope with a color video monitor using an imaging unit (Fig.2).

Figure 1. Over view of navigation system for dental fiberscope

Figure 2. Endoscope system design

The objective tooth in this study was a maxillary right central incisor under root canal treatment. A 3D tooth model was constructed from the raw dental CT data obtained using a dental cone-beam CT (PSR-9000N, ASAHI ROENTGEN IND., Kyoto, Japan). The image parameters were as follows: matrix size 512 X 512 pixel, FOV ø41 mm X 40 mm (height), pixel size 0.1 mm, and slice thickness 0.1 mm. The CT data were segmented and the 3D model was made using an image analysis (Fig.3). In addition, an optical measurement device (OPTOTRAK system, NDI Inc., Canada) was used for registration of the actual positions of tooth and fiberscope with the corresponding 3D models. This enabled us to track the movements of the tooth and the scope. Fig.4 shows the tracking tools attached with the patient’s dentition. This tool was constructed by 4 markers of OPTOTRAK which were arranged in 3D (not on the same plane).

Figure 3. The tracking tools for the dentition

Figure 4. The tracking tools for the dentition

Y. Yamazaki et al. / Clinical Performance of Dental Fiberscope Image Guided System

715

An exclusive software was developed to unify information. The optical distortion in the fiberscopic image was numerically rendered to the CT image.

Results and Discussion

Figure 5. System configuration of navigation system

Figure 6. Work station console of navigation system

Fig 5 shows the system configuration during the navigation. We were subsequently able to precisely indicate the device position relative to the teeth on the 3D model in the monitor (Fig.6). This allowed us to aim at the lesion more precisely during the laser irradiation, as the revised fiberscopic image matched the 3D model. The form of the lesion could be observed more precisely and better operational views could be obtained during the laser irradiation. Thus, we were better able to grasp the correct position of the fiberscope, allowing us to perform laser irradiation on the lesion with a higher degree of safety and precision. To conduct more accurate treatments with surgical navigation, we are developing software to reduce more distortion in the real-time fiberscope image and to improve the accuracy of the registration.

References T. Ozawa, M. Tsuchida, Y. Yamazaki, T. Arai, J. Nakamura, Clinical application of a fiberscope for periodontal lesions: case reports, Quintessence Int. 30 (1999) , 615-22. [2] Y. Otake, K. Hagio, N. Suzuki, A. Hattori, N. Sugano, K. Yonenobu, T. Ochi, Four-dimensional Lower Extremity Model of the Patient after Total Hip Arthroplasty, Journal of Biomechanics 38 (2005), 397-405. [3] Y. Otake, N. Suzuki, A. Hattori, H. Miki, M. Yamamura, N. Nakamura, N. Sugano, K. Yonenobu, T. Ochi, Estimation of Dislocation after Total Hip Arthroplasty by 4-DimensionalHipMotion Analysis, Studies In Health Technology and Informatics 111 (2005), 372-7. [4] Y. Yamazaki, T. Ozawa, T. Ogawa, Y. Shigeta, T. Ikawa, S. Fukushima, T. Arai, A. Mishima, K. Kobayashi, Y. Otake, A. Hattori, N. Suzuki.: Dental fiberscope with navigation system for endodontic treatments, Studies In Health Technology and Informatics 132 (2008), 562-4. [1]

716

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-716

A Novel Virtual Reality Environment for Preoperative Planning and Simulation of Image Guided Intracardiac Surgeries with Robotic Manipulators Erol YENIARAS a,1, Zhigang DENG b, Mushabbar A. SYED c, Mark G. DAVIES d and Nikolaos V. TSEKOS a a Medical Robotics Lab., Dept. of Computer Science, University of Houston, USA b Computer Graphics and Interactive Media Lab., Dept. of Computer Science, U. of H. c Division of Cardiovascular Medicine, Loyola University Medical Center d Department of Cardiovascular Surgery, The Methodist Hospital System

Abstract. The evolution of image-guided and robot-assisted procedures can be beneficial to intracardiac interventions. This paper proposes a novel approach and a virtual reality system for preoperative planning and intraoperative guidance of cardiac procedures, and for investigating the kinematics and control of a virtual robotic manipulator, based on MRI CINE images. The system incorporates dedicated software modules for processing MR images, generating dynamic trajectories in the continuously changing environment of a beating heart, controlling a specific generic virtual manipulator along those trajectories, and a virtual reality interface that fuses all those information. The proposed system is applied for the simulation of accessing the aortic valve annulus via a small incision on the apex by maneuvering a robotic manipulator through an access corridor that safely transverses the left ventricle (LV) of the beating heart. Keywords. MRI, medical robotics, cardiac surgery, virtual reality, image guidance

Introduction The advent of real-time image guidance (RTIG), especially combined with controlled manipulators, offers new opportunities for volumetric manipulation and assessment of the tissue function before, during and after a procedure [1]. Potential benefits of RTIG for intracardiac procedures on the beating heart include: reduction of side effects associated with cardiopulmonary bypass, the option for assessing the results of the procedure at the natural beating condition of the heart, and the faster patient recovery [2]. Currently, three-dimensional (3D) ultrasound is the most common modality due to its real-time volumetric data collection, and lack of ionizing radiation [3, 4]. In recent years, magnetic resonance imaging (MRI) has emerged as a meritorious modality for RTIG due to the following features: (1) a plethora of contrast mechanisms for the morphological and functional assessment of heart pathology, as well as imaging 1

Corresponding Author: Nikolaos V. Tsekos, Medical Robotics Laboratory, Department of Computer Science, Univ. of Houston, Houston, TX 77204, USA; E-mail: [email protected]

E. Yeniaras et al. / A Novel Virtual Reality Environment

717

protocols for visualizing interventional tools and robotic manipulators, (2) 3D and multislice imaging, (3) on-the-fly adjustment of the acquisition parameters directly from the robotic control core, and (4) no ionizing radiation. MRI guided interventions have evolved significantly in recent years [5] and lately MRI guidance has been demonstrated to be successful in transapical aortic valve interventions [6, 7]. In this paper, we present a novel computational methodology, and its integration into a system, for performing and simulating preoperative planning and intraoperative guidance of MRI-guided and robot-assisted intracardiac procedures on the beating heart via transapical access. In particular, the herein described system introduces: • A dynamic virtual reality environment that provides an information-rich, intuitive and comprehensive perception of the area of procedure (AoP) to the operator based on pre- and intra-operative MRI. • On-the-fly generation of patient-specific dynamic access corridors inside the left ventricle (LV), for the safe maneuvering of a robotic manipulator [8]. • A generic robot control based on dynamic trajectories extracted from MRI.

1. Methods For setting the design direction of the proposed system it is instructive to first assess the specific topography related to a procedure that entails a transapical access to the aortic valve. Figure 1A shows a long axis MR image of the heart with the identified entrance point at the apex and targeted aortic annulus. As illustrated in Figure 1B and 1C, in a generalized transapical approach, the manipulator enters the heart at an apical point (PA) and deploys inside the LV toward the targeted center of aortic annulus (PT). Analysis of multislice MR images indicates that the deployment path of the manipulator from PA to PT may need to follow a non-straight path, including a deflection point (PD) near the base of the LV. From the kinematic and path planning points of view, the major challenge is to determine the transient positions of PA, PD and PT points (guiding-points) and the dynamic safe access corridor (SC) relative to the inherent, absolute coordinate system of the MR scanner. Considering the current simulation and potential future in vivo applications, this work is based on the following assumptions. First; the base of the manipulator is anchored at the apex (PA), which is determined from CINE images, but in an in vivo

Figure 1. (A) Topography of the area of interest. (B) Representation of the deployment task. (C) A model of the virtual robot for the simulated paradigm of transapical access to the aortic valve.

718

E. Yeniaras et al. / A Novel Virtual Reality Environment

scenario, miniature RF coil beacons can be attached to the base of the robot for tracking [9]. Second; SC is both extracted and updated using multislice CINE MRI. In practice a complete CINE set will be used only preoperatively to define SC for a complete heartcycle, whereas a limited number of intra-oblique slices will be used to update the SC in real-time during the procedure [6, 7]. The coordinates of PT will also be extracted and tracked using the same limited number of slices with fast tissue tracking algorithm[10]. Figure 2 outlines the core system architecture, which will be connected to a commercial clinical MR scanner (e.g., Siemens MAGNETOM) for real-time access to the raw MR data and on-the-fly adjustment of imaging parameters[11]. 1.1. MR Image Processing The task of image processing includes the detailed segmentation of the endocardium and the identification of the three guiding-points: the apical (PA), the mid-LV (PD) and the targeted center of aortic annulus (PT). Preoperative planning and the simulation are based on the datasets that were collected on healthy volunteers (n=10) with a true fast imaging, steady-state precession (TrueFISP) pulse sequence with the acquisition parameters: TR = 2.3 ms, TE = 1.4 ms, Alpha = 80o, slice thickness = 6 mm, interslice distance = 6 mm, and acquisition matrix = 224x256. Each dataset included nineteen short and five long axes slices depicting 25 frames for a heart cycle. The CINE datasets of short axis (SA) and long axis (LA) images were segmented to extract the LV and the aortic annulus by tracing the corresponding endocardial boundary using an in-house developed software gadget endowed with Insight Toolkit (ITK) routines. To realistically model the access corridor in LV, the segmentation contours included the papillary muscles as depicted in Figure 3A, i.e., treating them as parts of the endocardial wall since they are also obstacles to the progress of the robotic manipulator. This is in contrast to the current simplified practice in the literature that segments the LV ignoring the details of the papillary muscles. The access corridor SC was determined from the segmentation contours as a surface inside the LV that does not contact the endocardial wall in any time frame. In other words, “when inside this time-dependent safe corridor, the robotic manipulator should touch neither the endocardium nor the papillary muscles”. Using a conservative

Figure 2. Flowchart of the system: the timing of the robot deployment and movement is synchronized with that of the acquisition of the CINE MR images.

E. Yeniaras et al. / A Novel Virtual Reality Environment

719

approach for akinetic and dyskinetic areas, our algorithm performed the following steps for every single time frame (∀t: t=1 to 25): 1. Determine all SA slices which have a visible LV blood pool by simply checking the inside surface areas of the LV endocardial boundary contours from the apex to the base of the heart (e.g., Figure 3A and 3B). Obviously, if an area is non-zero (or bigger than a preset threshold value), the slice shows the blood pool. 2. Project LV endocardial boundary contours of these SA slices onto a single virtual plane along their common orthogonal axis to find their common area (SB (t)) by a 2D polygon intersection algorithm. This projection is based on the fact that SA slices are parallel to each other and collected with the same field of view. Figure 3A and 3B depict all the endocardial contours and the common areas in two extreme heart phases, namely, diastole (relaxation) and systole (contraction). 3. Extend this common area, from the apex to the base of the heart, through all short axis slices to construct a straight access corridor (Figure 3C). Since a unique 3D access corridor is generated for every single heart phase (time frame), SC is a 4D dynamic entity for a complete heart cycle, i.e., ∃SC (t), ∀t: t=1 to 25. Further analysis of SC showed that its average base area for 10 patients takes the maximum value of 430 mm2 in relaxation and minimum value of 97 mm2 in contraction which is wide enough for the safe access of a robotic catheter. The coordinates of the apical point, PA (t), for any time frame (t) were determined by identifying the extreme point of the apical curvature on the central long axis view, which was further verified by assessing whether it belongs to a short axis that does not depict any blood pool (i.e., it is only myocardium) for the time frame. The coordinates of the targeted point, PT (t), were determined from the segmentation contours of two LA and one SA slices that included the aortic valve annulus at the level of the aortic valve leaflets, as the midline aortic annulus. In real life surgery, the operator can define the exact target point according to personal surgical approach to the aortic valve intervention. The deflection point for any time frame, PD (t), was then assigned as the intersection of the aortic annulus midline (from the previous step) with the SB (t) of the first-to-cross short axis slice. To the end, the three extracted guiding points PA (t), PD (t) and PT (t) were sent to the control module while the access corridor SC (t) was sent to the visualization module.

Figure 3. Methodology is depicted step-by-step. A sample short axis slice shows (A) diastole and (B) systole phases and their ITK segmentation contours. The inside of a contour is bloodpool (i.e., inside of LV). All such contours of a particular time frame are projected to a common plane and SB (t) is found as their intersection. (C) SB (t) is extended to define the corridor.

720

E. Yeniaras et al. / A Novel Virtual Reality Environment

1.2. Control and Visualization Modules Control module performs the inverse kinematics (IK) calculation and determines the values of six degree-of-freedom (DOF) parameters (refer to Table 1) for deploying the virtual robot from the apical point (PA) to the targeted point (PT) via deflection point PD, as well as for holding its position at any intermediate or terminal arrangement specified by the operator. The inputs to this module are the dynamic coordinates of the aforementioned guiding-points PA (t), PD (t) and PT (t) and the initial conditions specified by the operator; i.e., the time frame when the robot initiates its maneuvering, and whether and for how long it may hold a certain position along its path. Table 1. IK analysis of the virtual robot (Figure 1C) at the coordinate system of MR scanner. DOF parameter

IK equation

α: first Euler angle at Apical point PA

α = cos ((zD- zA)/ L1)

β: second Euler angle at Apical point PA L1: length of the first prismatic link γ: first Euler angle at Mid-LV point PD θ: second Euler angle at Mid-LV point PD L2: length of the second prismatic link

-1

-1

β = cos ((xD- xA)/(L1*sinα)) L1 = sqrt((xD- xA) 2 + (yD- yA) 2 + (zD- zA) 2) -1

γ = cos ((zT- zD)/ L2) -1

θ = cos ((xT- xD)/(L2*sin γ)) L2 = sqrt((xT- xD) 2 + (yT- yD) 2 + (zT- zD) 2)

The purpose of the visualization module is to generate and update a virtual reality environment that simulates the AoP which was implemented with OpenGL. The visualization module can display any combination of the following objects: MR images, segmentation contours, guiding-points, 3D access corridor, and virtual robot. The AoP is built relative to the inherent coordinate system of the MR scanner, which offers a natural space of visualizing 3D geometric structures. The update rate of the AoP is the same as that of the collected MR images. In its basic form, the AoP can display and refresh the positions of the three guiding points as shown in Figure 4A. The access corridor may also be included with or without the manipulator. Different scenarios can be simulated to assess the beginning of a procedure, the idling of the manipulator at any instance, or the whole duration of the heart cycle. Motion of the virtual robot entails the following three steps: deployment of the first link from the apex to the top of the LV point PD (Figure 4A (1), (2)), extension of the second link toward the targeted point PT, and the holding of the position (Figure 4A (3), (4)). During the maneuvering process, the control module supplies the values of the

Figure 4. Four selected frames: (A) showing the access corridor and the three guiding-points which are calculated with respect to the MR scanner coordinate system and updated dynamically, (B) top view of step-by-step robot deployment within the corridor from apex to aortic annulus.

E. Yeniaras et al. / A Novel Virtual Reality Environment

721

updated DOFs for each time instance. For the purpose of visualization, the intersections of PA to PD line with all the slices are calculated and depicted.

2. Results and Discussion The system was successfully tested for several different combinations of: DOF actuation speeds, diameters of robotic links, numbers of simulated heart cycles, heart phases when deployment was initialized, and holding of the robot at any particular configuration. Figure 5 illustrates the actuated DOFs at a representative example of a simulated procedure. After an initial user-selected short idling period of 250 ms, the graphs reflect two phases of robot maneuvering. The Phase I is the extension of the first link (L1) from the apical point PA toward the deflection point PD. In this phase, the rotational DOFs alpha and beta are actuated to maintain the deployed link L1 inside the access corridor SC and along the PA-to-PD line. After the distal end of the first link reaches the base point PD, the Phase II starts immediately and entails the extension of the second link (L2) to reach the targeted point PT at the entrance of the aortic valve annulus. Concurrently, the rotational DOFs gamma and theta are also actuated to maintain the second link along the midline of the annulus. Once the annulus is reached, the robot maneuvering is performed to hold the position: the base of the robot at the apical point, the distal end of the first link at PD, and the distal end of the second link at P T. In this work, preoperative planning utilized CINE MRI of several slices over a complete heart cycle to determine the diameter of the aortic annulus, the anatomy of the coronary ostia, the apical entrance point and a dynamic access corridor. Once combined with an automated segmentation algorithm[12], which is beyond the scope of this study, the system can function with a direct connection to an MR scanner and thus be able to receive MR data and control the image acquisition parameters on-the-fly. The implemented virtual robotic manipulator was selected to offer the highest possible flexibility for performing a free-beating heart procedure. In addition to assessing the operation of the described system, this virtual tool offers a versatile kinematic structure for further investigating imaging-based path planning within dynamically changing environments, as well as approaches for automated compensation and position-holding. While many other different mechanical designs

Figure 5. (A) Linear, (B) rotational DOFs of the robotic device calculated by the control module. Time (x25): 0-10: Idling period; 10-35: Phase I; 35-65: Phase II; 65- : Holding position.

722

E. Yeniaras et al. / A Novel Virtual Reality Environment

may be envisioned to be more appropriate, including experimental and commercially available steerable snake-like devices such as the Hansen catheters, the proposed approach is not limited to particular technical specifications. The path planning algorithm can be adapted to any given kinematic structure. Our preliminary studies underscore the capabilities of imaging for dynamically changing non-straight access procedures beyond the realm of endoscopy [13]. Image guidance, combined with a robotic device, may also enhance future directions in the evolution of procedures such as the Natural Orifice Transluminal Endoscopic (NOTES) and Single Port Access (SPA) surgeries [14]. Our future work includes the physical prototyping of the virtual robot presented herein and implementation of real-time MRI with mutually-oblique slices.

Acknowledgments This work was supported by the NSF award CNS-0932272. All opinions, findings, conclusions or recommendations expressed in this work are those of the authors and do not necessarily reflect the views of our sponsors.

References F.A. Jolesz, Future perspectives for intraoperative MRI, Neurosurg Clin N Am 16 (2005), 201-213. P. Atluri, E.D. Kozin, W. Hiesinger, Y.J. Woo, Off-pump, minimally invasive and robotic coronary revascularization yield improved outcomes over traditional on-pump CABG, Int J Med Robot 5 (2009) 1-12. [3] R. Mebarki, A. Krupa, C. Collewet, Automatic guidance of an ultrasound probe by visual servoing based on B-mode image moments, Proceedings of MICCAI 11 (2008), 339-346. [4] S.G. Yuen, S.B. Kesner, N.V. Vasilyev, P.J. Del Nido, R.D. Howe, 3D ultrasound-guided motion compensation system for beating heart mitral valve repair, Proceedings of MICCAI 11 (2008), 711-719. [5] N.V. Tsekos, A. Khanicheh, E. Christoforou, C. Mavroidis, Magnetic resonance-compatible robotic and mechatronics systems for image-guided interventions and rehabilitation: a review study, Annu Rev Biomed Eng 9 (2007), 351-387. [6] E.R. McVeigh, M.A. Guttman, R.J. Lederman, M. Li, O. Kocaturk, T. Hunt, S. Kozlov, K.A. Horvath, Real-time interactive MRI-guided cardiac surgery: aortic valve replacement using a direct apical approach, Magn Reson Med 56 (2006), 958-964. [7] M. Li, D. Mazilu, K.A. Horvath, Robotic system for transapical aortic valve replacement with MRI guidance, Proceedings of MICCAI 11 (2008), 476-484. [8] E. Yeniaras, N. Navkar, M.A. Syed, N.V. Tsekos, A Computational System for Performing Robotassisted Cardiac Surgeries with MRI Guidance, Proceedings of SDPS 15 (2010), 1-6. [9] D.R. Elgort, E.Y. Wong, C.M. Hillenbrand, F.K. Wacker, J.S. Lewin, J.L. Duerk, Real-time catheter tracking and adaptive imaging, J Magn Reson Imaging 18 (2003), 621-626. [10] Y. Zhou, E. Yeniaras, P. Tsiamyrtzis, N.V. Tsekos, I. Pavlidis, Collaborative Tracking for MRI-Guided Robotic Intervention on the Beating Heart, Proceedings of MICCAI 13 (2010), 1001-1009. [11] E. Christoforou, E. Akbudak, A. Ozcan, M. Karanikolas, N.V. Tsekos, Performance of interventions with manipulator-driven real-time MR guidance: implementation and initial in vitro tests, Magn Reson Imaging 25 (2007), 69-77. [12] M. Fradkin, C. Ciofolo, B. Mory, G. Hautvast, M. Breeuwer, Comprehensive segmentation of cine cardiac MR images, Proceedings of MICCAI 11 (2008), 178-185. [13] C.A. Linte, J. Moore, C. Wedlake, D. Bainbridge, G.M. Guiraudon, D.L. Jones, T.M. Peters, Inside the beating heart: an in vivo feasibility study on fusing pre- and intra-operative imaging for minimally invasive therapy, Int J Comput Assist Radiol Surg 4 (2009), 113-123. [14] J. Kobiela, T. Stefaniak, M. Mackowiak, A.J. Lachinski, Z. Sledzinski, NOTES-third generation surgery: Vain hopes or the reality of tomorrow, Langenbecks Arch Surg 393 (2008), 405-411. [1] [2]

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-723

723

Enabling Surgeons to Create Simulation-Based Teaching Modules Young In YEO a Saleh DINDAR a George SAROSI b J¨org PETERS a a Dept CISE, University of Florida b Dept Surgery, University of Florida Abstract. To broaden the use of simulation for teaching, in particular of new procedures and of low-volume procedures, we propose an environment and workﬂow that allows surgeon-educators create teaching modules. Our challenge is to make the simulation tools accessible, modiﬁable and sharable by users with moderate computer and VR experience. Our contribution is a workﬂow that promotes consistency between instructional material and measured criteria and makes the authoring process efﬁcient, both for the surgeon, and for computer scientists supporting the simulation environment. Keywords. simulation, training, minimally invasive surgery, infrastructure

1. Introduction Increasing deployment of commercial simulators in surgical education attests to the general perception that virtual training is a valuable experience over conventional dry-lab training. The rationale for training surgeons with the help of 3D virtual scenarios is particularly strong for minimally invasive surgical techniques (see e.g. [5,6]). By allowing trainees to practice decision making and laparoscopic execution prior to an operating room experience, such simulators can make costly trainee time in OR more valuable, increase the safety to patients and reduce the need for in vivo animal practice. And with the compression of surgical training due to work-hour rules uncoupling of instruction from place and time is ever more important. While commercial surgical trainers are becoming widespread and ﬁnd their way into the curriculum, they provide a limited set of training modules. The cost structure favors the most common and the most basic surgical procedures. The corresponding teaching modules are programmed by engineers and modiﬁcations will always depend on development cycles and time to market. This means that in particular new procedures and low-volume procedures currently do not beneﬁt from the trend towards simulation for training. Moreover, ﬁxed-function simulations can not convey the individual variations in technique that surgeon-educators consider an important component of the masterapprentice relationship in traditional surgical education. Our long-term goal is therefore not just to generate new teaching modules, but to craft a software framework within which surgeons themselves deﬁne and adjust (author) interactive simulation teaching modules. The expectation is that, just as with desktop publishing, removing intermediaries will speed transmission of individual expertise and

724

Y.I. Yeo et al. / Enabling Surgeons to Create Simulation-Based Teaching Modules

Figure 1. The teaching module setup. The surgeon-author runs through the module, setting the acceptable performance ranges. The monitor on the left, in the background, displays the web-based instructions, questionnaires, video, and the ﬁnal feedback statistics. The other monitor, here a laptop, displays the 3D virtual scenario (run on the laptop). The interface consists of two Omni devices (robotic arms).

quickly document novel surgical techniques and variants by example. That is, we aim to broaden access to simulation for teaching and to help scientiﬁc discourse with the help of simulation: to quickly document, teach and assess knowledge of variants and innovations in technique, including extreme scenarios. The challenge is two-fold: technology and interface. It is amply documented that developing and combining the necessary state-of-the-art graphics, numerical and interface techniques underlying interactive 3D virtual scenario is highly non-trivial. Fortunately, computational simulation techniques in the biomedical, computer and mechanical engineering have made great strides (see e.g. [3,11,4,2,10]) and increased GPU processing power, accessed via shaders, CUDA or OpenCL, reduces even complex algorithms to a real-time experience. Our focus is on making simulation tools accessible, modiﬁable and sharable by users with moderate computer and VR experience. A key issue here is the work ﬂow for the surgeon-educator when creating a teaching module. In this paper, we present an approach for making such an authoring process more efﬁcient – both for the surgeon-educator acting as author or as instructor and for the computer scientists supporting the simulation. Leveraging advanced but standard web and database technology as well as an existing simulation platform, a key feature of our approach is to let the surgeon-educator determine the content while at the same time promoting consistency between instructional material and measured criteria.

2. Tools and Methods Approach. We are developing an open-source framework to enable surgeons-educators to apply simulation within their laparoscopy curriculum. Speciﬁcally, we leverage the TIPS (Toolkit for Illustration of Procedures in Surgery) infrastructure [7] for the 3D virtual scenario. To present a simple work ﬂow to the surgeon-educator creating a teaching module, a web-based central form Fig. 2 guides the surgeon in decomposing the procedure into steps with their tasks, what to do and how, and safety issues, what not to do and how to identify mistakes, as is typical in traditional teaching. From this decomposition, our

Y.I. Yeo et al. / Enabling Surgeons to Create Simulation-Based Teaching Modules

Figure 2.: The central form supports a hierarchical decomposition of the procedure and speciﬁcation of relevant details.

725

726

Y.I. Yeo et al. / Enabling Surgeons to Create Simulation-Based Teaching Modules

simulator data

enable central form

3D virtual scenario (5) (6)

(1) Pre-questions on Simulation teaching module

Instructions

(7)

data for feedback Post-questions on teaching moduleSimulation

1 2

3

(4)

(2)(3)

(4)

Figure 3. Auto-generation of instructions and questionnaires (bottom row of web-pages) based on the central form (see Section 3 for the numbers in parentheses).

software pre-ﬁlls templates of web-pages for instruction, a list of surgical tools and 3D anatomy (organs, vessels) and properties thereof including haptic and reporting properties for the built-in assessment. The point in generating templates is two-fold: to simplify authoring and to promote consistency between stated and measured trainee performance criteria. Tools. As shown in Fig. 1, the physical setup of the teaching modules includes two monitors showing respectively the 3D virtual scenario on the local PC and the instructions on a web-browser. Two Phantom Omni devices provide six degrees of freedom movement measurement and three degrees of force-feedback. Including a PC with a high-end graphics card, the total cost is ca $5,000. For the 3D virtual scenario, we leverage the TIPS simulation environment which allows the surgeon-author to create, place and size 3D anatomy (organs, vessels, etc.) occluding fatty tissue [9], covering peritoneum [8], select tools and to interactively adjust properties, such as the haptic feedback force. For the web-based component, we leverage dynamic web-application software based on open-source technologies to provide a collaborative authoring tool that automates generation of reports and templates and provides rich multimedia teaching modules to end-users. Methods. The key to streamlining the workﬂow for both developer and author is a web-based central form Fig. 2. It is the basis for auto-generating drafts for instruction pages, evaluation forms and questionnaires; serves as a todo-list for the author; and it conveys essential details about the surgical procedure to technicians who may preassemble the 3D virtual scenario and alert developers to missing objects or features in the database, such as a new surgical tool. Fig. 3 summarizes the basic structure: The teaching modules are built around the steps of the procedure. Drafts of questionnaires are generated from a transcription of the atomic tasks and errors and added as instruction pages for revision by the author. Also an outline of evaluation form is created to be ﬁlled with upper and lower bounds data by the surgeon-educator’s performance in the simulation environment.

Y.I. Yeo et al. / Enabling Surgeons to Create Simulation-Based Teaching Modules

727

Surgeon−Author

Technician

Estimated Time (hours)

0.5

Deconstruct procedure into steps

0.5

Collect operative videos

1.0

Fill out central form

No Request to add items into DB and TIPS

0.5

Create 3D mesh for the requested items

0.5

Add operations into TIPS

1.0

Check list of tools, organs, and operations against existing DB Yes

0.5

Assemble/modify scene

0.5

Set ranges (run TIPS module)

(a) Authoring effort. Flowchart and timeline for authoring the appendectomy module.

(b) Feedback form. The trainee’s performance is compared to the instructor-set min-max range. Figure 4.

Author View. For the surgeon, authoring a teaching module involves (i) ﬁlling in the central form which guides through deconstructing the procedure into its main steps deﬁned by surgical tasks and safety concerns, (ii) design the virtual scenario by assembling pre-existing organs, vessels, tissues and surgical tools, customize appearance such as color and shape, and set behavior, such as stiffness, interactively using GUI tools. Then the author (iii) performs the key points of the procedure in the 3D virtual scenario and ﬁnally (iv) attaches to the module’s web pages any of video, text book illustrations, sketches and/or audio as available and appropriate. The essence of the teaching modules are tasks and safety instructions (errors). Every task (what to do and how) must be measurable within the 3D virtual scenario. For example, the task ‘incision of the peritoneum’ should be made speciﬁc by stating ‘for the full craniocaudal extent of the gland, from the diaphragmatic level to the renal vein’ and this is made measurable by having the surgeon-educator perform the task inside the 3D virtual scenario. Every error, has to have its reporting function enabled. Finally, the author may choose to add comments that give immediate feedback (e.g. bleeding when the force exceeds a threshold) or context (video of surgery) but that do not themselves inﬂuence assessment or outcome. The software layer generates from the central form: templates for the instructions, templates for pre-questions and post-questions, a list of object reporting values and the catalogue of feedback items (Fig. 4(b)). These templates not only help the author with the deconstruction and are convenient but promote consistency of tasks and measurements in the teaching module. Trainee View. A trainee can view the instruction pages via a standard web-interface that guides through: a pre-questionnaire, the surgical procedure overview, tasks and safety information assorted in steps, an illustration of each step from video and a video of the simulated procedure, a post-questionnaire and the evaluation form (see Fig. 3). After each illustration of a step, the trainee is asked to replicate it within the 3D virtual scenario. At the end of a training session, the results are compared to the instructor’s. Developer View. Central form entries can alert developers to missing objects, or object functionality, for example the multi-stapler in the appendectomy module. The surgery video clip attached to the central form and the function description in the task

728

Y.I. Yeo et al. / Enabling Surgeons to Create Simulation-Based Teaching Modules

and error speciﬁcation help to give context and allows for a distributed, asynchronous, collaborative work ﬂow. Built-in assessment via performance ranges. teaching modules are built around surgeon-set assessment criteria expressed as tasks and errors. The ﬁnal detailed feedback compares trainee performance relative to The instructor-established performance range of these criteria. After instantiating and customizing the 3D virtual scenario, the author turns on reporting of relevant measurements as provided by the developers. For example, number and force of probing, length and placement of incision or cautery in proximity to the vena cava can be reported. The author then performs the procedure several times to create acceptable upper and lower values for enabled values. Evaluating data collected from the trainee yields an assessment report that shows any deviation from the acceptable range. (Separate questionnaires assess the perceived quality of the teaching module.) Altogether, this yields a simple, adaptable feedback mechanism that focuses the teaching modules on the outcome. 3. Results To date, teaching modules of variants of three laparoscopic procedures have been created: removal of the adrenal gland, the appendix and the spleen. Each module combines web-based instructions containing text, audio, video both of the surgical procedure and the TIPS simulation, and feedback based on the trainee questionnaires and on the performance in the adjustable 3D virtual scenario. Every teaching module is structured according to the holistic view recommended e.g. in [5]: (cf. Fig. 3 for the numbering) after prior (1) teaching of relevant knowledge i.e., anatomy, pathology, physiology; (2) instruction on the steps and task of the procedure: ‘what to do and how’; (3) deﬁning and illustrating common errors ‘what not to do and how to identify mistakes’; (4) pre- and post test of (13) to later gauge knowledge improvement; (5) technical skills training on the simulator; (6) (blood, color change) provide immediate (proximate) feedback when an error occurs; (7) summative at the completion of a trial. Iterating a teaching module allows to chart progress, varying a teaching module allows implementing teaching techniques such as shaping (progressively increasing the difﬁculty), fading (removing clues), or backward chaining (increasing the number of tasks starting from the back). To date, only the arenalectomy module has undergone peer review by 6 surgeons and initial assessment with 8 residents at two research hospitals. Formal testing, in a uniform environment with a sufﬁciently large pool of trainees is scheduled for 2011. In any case, the focus of the present paper is on how well our approach supports surgeon-educators in creating teaching modules. To this end, we measured amount of time spent for authoring a new laparoscopic appendectomy module. An appendectomy module (see Fig. 4(a)) was authored within an active surgeon’s time plan in two sessions over a two week period. This was the author’s ﬁrst module. While we were physically present during the ﬁrst authoring session to deliver the Omni feedback devices, For the description of the authoring tasks, we used a short instruction video on our web-site and the prompts of the central-form to simulate as more asynchronous, distributed development. The appendectomy teaching module was then created from scratch requiring 3 hours of the author’s time over two sessions. Fig. 4(a) gives a detailed time-line. The two hours of support time were not caused by explaining the authoring interface but needed to create a newly requested multi-stapler and corresponding organ-functionality in the database.

Y.I. Yeo et al. / Enabling Surgeons to Create Simulation-Based Teaching Modules

729

4. Discussion Creating simulations in 3D virtual scenario is challenging and a source of hard, computational problems for the community for some time to come. By making the central-form rather than the 3D virtual scenario the cornerstone of our development, we make a conscious decision to allow the individual surgeon-educator drive the use of simulation in the hope that this broadens the use of simulation and points us to features that promote learning – even at the cost of the highest-ﬁdelity experience. This point of view is also shared by some experts [5,1]. That is, we assume that realism of the 3D virtual scenario of the teaching modules is important only in so as it inﬂuences quality of teaching. In particular, the 3D virtual scenario has to satisfy the surgeon-author’s expectations instead of the bio-engineer’s. A surgeon-author may well choose to color arteries a solid red and veins a solid blue in violation of accurate physical modeling of a wet-lab scenario; or to artiﬁcially overemphasize the stiffness of a vessel or brittleness of an organ to highlight possible complications; or the author may suppress detail. Empowering each surgeon-author to teach their approach could potentially reduce quality and consistency versus commercial modules. For example, it is possible that an author or instructor sets performance ranges too tight (although we have not observed specialist surgeons making this mistake), leave out a task or a safety issue deemed obvious. As with standard publication, peer review rather than the authoring environment has to take care of quality issues. The teaching modules can be disseminated and shared via the internet. The long-term hope then is that criteria will be vetted by the surgical community and evolve into standards. This research was supported by NIH R21 EB005765-01A2.

References [1] AAMC. Effective use of educational technology in medical education: Colloquium on educational technology: recommendations and guidelines for medical educators, 2007. [2] J´er´emie Allard, Maud Marchal, and St´ephane Cotin. Fiber-based fracture model for simulating soft tissue tearing. In Proc 17th MMVR, pages 13–18, Jan 2009. [3] M. Cheng, Z. Taylor, and S. Ourselin. Towards anatomical modelling of multiple organs interaction using real time gpu based nonlinear elasticity. Stud health tech and informatics, 132:77, 2008. [4] Dhanannjay Deo and Suvranu De. A machine learning-based scalable approach for real-time surgery simulation. In Proc 17th MMVR, pages 71–76, Jan 2009. [5] A.G. Gallagher, E.M. Ritter, H. Champion, G. Higgins, M.P. Fried, G. Moses, C.D. Smith, and R.M. Satava. Virtual reality simulation for the operating room: proﬁciency-based training as a paradigm shift in surgical skills training. Annals of surgery, 241(2):364–372, 2005. [6] KS. Gurusamy, R. Aggarwal, L. Palanivelu, and BR. Davidson. Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database of Systematic Reviews, 2009. [7] Minho Kim, Tianyun Li, Juan Cendan, Sergei Kurenov, and J¨org Peters. A haptic-enabled toolkit for illustration of procedures in surgery (TIPS). In Proc 15th MMVR, pages 209–214, Feb 2007. [8] Ashish Myles, Young In Yeo, Minho Kim, Juan Cendan, Sergei Kurenov, and J¨org Peters. Interactive peritoneum in a haptic surgery illustration environment. In Proc 17th MMVR, pages 221–223, Jan 2009. [9] Sukitti Punak, Minho Kim, Ashish Myles, Juan Cendan, Sergei Kurenov, and J¨org Peters. Fatty tissue in a haptic illustration environment. In Proc 16th MMVR, pages 384–386, Feb 2008. [10] ZA. Taylor, S. Crozier, and S Ourselin. Real-time surgical simulation using reduced order ﬁnite element analysis. In 13th MICCAI, 2010. [11] Xiangmin Zhou, Nan Zhang, Desong Sha, Yunhe Shen, Kumar K. Tamma, and Robert Sweet. A discrete mechanics framework for real time virtual surgical simulations with application to virtual laparoscopic nephrectomy. In Proc 17th MMVR, pages 459–464, Jan 2009.

730

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-730

Using A Virtual Integration Environment in Treating Phantom Limb Pain Michael J. ZEHERa, Robert S. ARMIGERa, James M. BURCKa, Courtney MORANa, Janid Blanco KIELYb, Sharon R. WEEKSb, Jack W. TSAOb, Paul F. PASQUINAb, R. DAVOODIc, G. LOEBc, a Johns Hopkins Applied Physics Laboratory (JHU/APL) b Walter Reed Army Medical Center (WRAMC) c University of Southern California (USC)

Abstract. The Revolutionizing Prosthetics 2009 program conducted by the Defense Advanced Research Projects Agency (DARPA) has resulted in a Virtual Integration Environment (VIE) that provides a common development platform for researchers and clinicians that design, model and build prosthetic limbs and then integrate and test them with patients. One clinical need that arose during the VIE development was a feature to easily create and model animations that represent patient activities of daily living (ADLs) and simultaneously capture real-time surface EMG activity from the residual limb corresponding to the ADLs. An application of this feature is being made by the Walter Reed Military Amputee Research Program (MARP) where they are utilizing the VIE to investigate methods of reducing upper extremity amputee phantom limb pain (PLP). Keywords. DARPA, Revolutionizing Prosthetics 2009, activities of daily living (ADL), Virtual Integration Environment (VIE), Walter Reed Military Amputee Research Program (MARP), phantom limb pain (PLP), Musculoskeletal Modeling Software (MSMS)

Introduction The Defense Advanced Research Projects Agency (DARPA) Defense Sciences Office (DSO) Revolutionizing Prosthetics 2009 Program developed an advanced upperextremity prosthesis with the potential to restore full motor and sensory capability to upper extremity amputee patients. The Modular Prosthetic Limb (MPL) produced at the end of the four year program provided a leap forward in capability of upper extremity prostheses. The MPL is controlled and recognized by neural interfaces and perceived as natural with respect to function, weight, durability and comfort. The final phase of the program will be a prosthetic device ready for human clinical trials, including the completed submission of an Investigational Device Exemption (IDE) application to the U.S. Food and Drug Administration (FDA). The Johns Hopkins Applied Physics Laboratory (JHU/APL) was the Systems Integrator for phases one and two of the Revolutionizing Prosthetics 2009 program conducted by DARPA. One of the outcomes of this program is a Virtual Integration

M.J. Zeher et al. / Using a Virtual Integration Environment in Treating Phantom Limb Pain

731

Environment (VIE) that provides a common development platform for researchers and clinicians that design, model and build prosthetic limbs and then integrate and test them with patient[1]-[3]. One aspect of the VIE was the need for clinicians to easily create and model real-time animations that represent patient activities of daily living (ADL). The VIE has been integrated with the Musculoskeletal Modeling Software (MSMS) developed by USC on the RP2009 program. Animations created with MSMS, using Microsoft PowerPoint™ as an editing tool, can be synchronized with external activities in real-time using the VIE.

1. The Virtual Integration Environment (VIE) The general VIE capabilities include a complete limb system simulation environment that allows for mechanical design evaluation, engineering development with a virtual prosthetic limb, a GUI interface for patient training and therapeutic applications, neural signal acquisition, neural decoding algorithm development, and system performance validation and design compliance. For end-to-end interactive simulations the VIE acquires control signals (myoelectric, mechanical, neural, and others), performs signal analysis by interpreting intention, and control by translating intention into movement of a virtual limb[2]. This feature allows the user to interact with objects using feedback (haptics). In the research environment the VIE provides experimental control for researchers that is modular and configurable. Various limb models and control algorithms are supported. This configuration is an excellent engineering test bed for improving neural and control algorithms and evaluating patient interfaces for control signal extraction and sensory feedback. Typical VIE users are biomedical engineers that analyze and design prosthetic limbs, controllers, and algorithms and biomechanical engineers that perform desktop prototyping of model designs in real time for final analysis. Neural Researchers use the VIE to analyze and design signal analysis algorithms, perform real time open and closed loop control experiments, collect experimental data from a variety of sources including EMG, peripheral, and cortical signals, as well as motion tracking data[2]. Clinicians use the VIE to fit real prosthetic limbs, train patient to use control sites with a virtual prosthetic limb prior to the real prosthetic limb being available, and configure real and virtual prosthetic limbs parameters. Patients use the VIE to operate a virtual prosthetic limb in task oriented environment and train control sites in a “fun” way. The VIE has been integrated with the Wii™ game system to provide patients with an interface to Guitar Hero and other games for training fun[7]. Figure 1 shows a conceptual view of the VIE.

732

M.J. Zeher et al. / Using a Virtual Integration Environment in Treating Phantom Limb Pain

Figure 1. The VIE Components.

2. Visualization with MSMS The MSMS provides general tools for building models of human limbs, prosthetic limbs, and rehabilitation tasks. It enables engineers to perform simulations to predict movement under different neural control strategies. It enables clinicians to perform real-time subject-in-the-loop simulations to train patients. The MSMS architecture has been designed for flexibility, expandability, and ease of maintenance[4]. It provides a GUI interface for modeling and simulation environments that are supported by an underlying database. The MSMS modeling language is Java and 3D Java. The Simulation language is Matlab Simulink™ and SimMechanics™, and C code. MSMS models can be built interactively through the GUI graphics tools. Models can be built using XML and can also be imported from SIMM and SolidWorks™. Existing prosthetic arm and world models can be combined to quickly build the desired virtual training environments. The MSMS GUI provides tools to navigate, view, and edit model components. Any number of cameras and lights can be added to view and properly illuminate the virtual environment. MSMS supports 2D and variety of 3D stereoscopic displays for Virtual Reality (VR) experiments. Models can be animated using recorded data or motion data streamed in real-time from variety of live sources. Models can be converted to Simulink for kinematic or dynamic simulation. The MathWorks™ xPC Target can be used to run Simulink models in real-time for neural prosthesis simulations with the subject in the loop. Applications include virtual prototyping of neural prostheses, training patients to operate their neural prostheses, and virtual rehabilitation[5]. Figure 2 shows the model editing feature.

M.J. Zeher et al. / Using a Virtual Integration Environment in Treating Phantom Limb Pain

733

Figure 2. MSMS Model Editing.

3. The Clinical Interface Functional assessment for Activities of Daily Living (ADLs) was determined to be a requirement for satisfying the Revolutionizing Prosthetics 2009 objectives. This necessitates developing baseline ADL assessments that will allow the patient and clinician to properly configure limb profiles. The profile consists of a functional capacity evaluation performed by Physical Therapists and Occupational Therapists, to include grip strength, push/pull, overhead capacity, weighted transfers, and others. After measurements of performance establish the specific Basic Activities of Daily Living (BADLs) patients can progress to more advanced Instrumental Activities of Daily Living (IADLs). The patient’s ability to adapt to changing environments and willingness to accept new challenges is documented. The patient can then progress to the Enhanced Activities of Daily Living (EADLs) where a record of physical capacity and functional performance is maintained and an appropriate maintenance schedule is defined. It can then be determined whether supplemental training or additional strategies are needed for successful integration. This functional assessment allows for upgrades as technologies progress. Functional Assessment for the Warfighter enables patients who wish to remain in service, qualify for a reclassified position. Maintaining and reviewing history of baseline measurement and performance assessments (specific to reclassified occupation) would allow the patient and clinician to determine proper training and performance strategies to meet task requirements for job qualification process. The Clinical VIE with MSMS provides a clinical configuration “Neural Toolkit” that supports a selection of neural interface options. Medical staff can develop an approach for the socket, neural interface, and limb from toolkit modules. Rehabilitation Physicians and Prosthetists can configure the limb and adapt it to the patient using the Clinical VIE. The Clinical VIE is used for simulation and training using gaming.

734

M.J. Zeher et al. / Using a Virtual Integration Environment in Treating Phantom Limb Pain

Prosthetists can establishes a baseline limb “profile” with the appropriate dynamic or static socket. The need for functional assessment of patient ADLs demanded an interface for creating ADL animations that was intuitive and easy to use. Leveraging the MSMS XML interface for model inputs we decided to use Microsoft’s Office 2007 version of PowerPoint™, which includes XML support, to create ADL animation sequences. A library of pre-prepared motion elements is created and the user combines the motion elements by putting the .JPEG files into a PowerPoint™ sequence. A motion element consists of a .JPEG image that represents a specific motion and a corresponding motion file containing joint angle sequences. The presentation is saved as a PowerPoint XML file that can be used by the VIE and MSMS to display the ADL motion sequence to the patient and synchronously record sEMG activity with the ADL motion sequence. A view of the animation is shown in Figure 3.

Figure 3. MSMS ADL Animation View

4. Studying Phantom Limb Pain The Walter Reed Military Amputee Research Program (MARP) is testing an application of this system, utilizing the VIE and MSMS software to generate arm and hand animations for upper extremity amputees suffering from phantom limb pain (PLP). PLP is clinically understood as intense, exteroceptive sensations localized in, or originating from, an amputated body part [8]. Anecdotally, some of Walter Reed’s amputee patients experience PLP episodes that recall traumatic, pre-amputation limb positions (e.g. grenade clutching, rifle trigger pressing, and hyperextension). Duration,

M.J. Zeher et al. / Using a Virtual Integration Environment in Treating Phantom Limb Pain

735

frequency, intensity, and qualitative description characterize PLP episodes [9]. Patients have reported that the majority of painful sensations originate in the distal part of the amputated limb. Both Central and Peripheral Nervous System processes are believed to cause PLP, underscoring PLP’s complex etiology and its treatment challenges. Cortical plasticity, specifically the deafferented limb limb’s decreased cortical representation in primary motor cortex (M1) and somatosensory cortex (S1) as well as increased cortical representation of neighboring cortical areas into the deafferented area, has been implicated as one of PLP’s causative mechanisms in multiple fMRI-based neuroimaging studies [10]. Nearly all traumatic limb amputee patients experience phantom limb cognizance at some point during their rehabilitation, perceiving a vivid impression that their lost limb is not only present, but also in many cases, painful. Burning, itching, and shooting pain in addition to unnatural limb contortion are common phantom limb sensations among military amputees. Acting as a computerized “Mirror Therapy” parallel, The VIE’s ability to simulate real limb motion and create an interactive therapeutic environment lend it a unique ability to treat PLP, a condition with few treatment options. Among PLP’s few treatment options, Mirror Therapy has been shown to provide pain relief for many amputee patients [11, 12]. Mirror therapy involves patients observing the reflection of their moving, intact limb while attempting to produce the same movements with their phantom limb (hidden from view on the opposite side of the mirror). Visual feedback of motor commands, achieved by the patient’s viewing his or her intact limb’s reflection in the phantom limb’s location, is likely to be the key component of phantom pain relief [11,12].

5. Conclusion As a result, we are now exploring whether MSMS-generated computer animation files of hand and arm movements are effective pain reducing therapies for upper extremity amputees. While the patient views the computer animation of his or her phantom arm movement, the patient will attempt to control and maneuver his or her phantom limb in the same manner. Using an array of electrodes placed around the patient’s residual limb at their site of amputation, the VIE monitors and records surface electromyography (sEMG) signals on the patient’s residual limb. Walter Reed hopes to correlate these acquired neuromuscular depolarization signals that indicate muscle fiber activity with changes in phantom pain levels; we would like to determine whether increased control of the phantom limb (more consistency among sEMG signals, less variance between trials of the same arm movement) is associated with decreased PLP. Furthermore, reliable phantom limb muscular activation may indicate a subject’s potential to control a myoelectric prosthetic arm. Using the VIE’s interactive user interface to train acute phantom limb control, patients can increase phantom limb dexterity by both tagging virtual objects (touching or grasping an animated ball) and eventually participating in advanced games that require fine motor control. Because clinicians can tailor VIE animations to a patient’s rehabilitative needs at each timepoint during the patient’s recovery, patients can rebuild motor skills in measured, incremented steps. A rehabilitation and prosthetic limb training regimen that clinicians can tailor to the appropriate level of difficulty during a patient’s recovery trajectory may serve as a useful clinical option in addition to occupational therapy sessions. The

736

M.J. Zeher et al. / Using a Virtual Integration Environment in Treating Phantom Limb Pain

VIE’s capacity to generate increasingly challenging patient interaction scenarios and animations may prove a useful tool in bridging the dexterity gap between an immobilized amputee who experiences PLP to a fully functional amputee operating an advanced prosthetic. With improving the medical management of the war fighter as our primary goal, the critical link between MSMS-generated visual animation files and residual limb/prosthetic biomechanics is an important component of current military amputee research.

References [1] W. Bishop, R. Armiger, J. Burke, M. Bridges, J. Beaty, R. J. Vogelstein, and S. Harshbarger, “A realtime virtual integration environment for the development of neural prosthetics,” in Proceedings of the IEEE Conference of the Engineering in Medicine and Biology Society, 2008, pp. 615–619. [2] W. Bishop, B. Yu, G. Santhanam, A. Afshar, S. Ryu, K. V. Shenoy,R. Vogelstein, J. Beaty, and S. Harshbarger, “The use of a virtual integration environment for the real-time implementation of neural decode algorithms,” in Proc. IEEE Engineering in Medicine and Biology Society Conference, 2008. [3] F. Tenore, R. Armiger, R. Vogelstein, D. Wenstrand, K. Englehart, and S. Harshbarger, “An embedded controller for a 7-degree of freedom prosthetic arm,” in Proceedings of the IEEE Engineering in Medicine and Biology Society Conference, 2007. [4] R. Davoodi and G. Loeb, “A software tool for faster development of complex models of musculoskeletal systems and sensorimotor controllers in Simulink,” Journal of Applied Biomechanics, vol. 18, pp. 357–365, 2002. [5] M. Hauschild, R. Davoodi, and G. Loeb, “A virtual reality environment for designing and fitting neural prosthetic limbs,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 15, no. 1, pp. 9–15,March 2007. [6] [Online]. Available: http://www.delta3d.org [7] R. Armiger and R. Vogelstein, “Air-guitar hero: A real-time video game interface for training and evaluation of dexterous upper-extremity neuroprosthetic control algorithms,” in Proc. IEEE Biomedical Circuits and System, 2008, pp. 121–124. [8] Weinstein SM. “Phantom Pain.” Oncology (Williston Park). 1994 Mar;8(3):65-70; discussion 70, 73-4. Review. [9] Sherman RA. “Phantom Pain: Mechanism Based Management.” Clin Podiatr Med Surg. 1994 Jan;11(1):85-106. Review. [10] Diersa M, Christmanna C, Koeppa C, Rufb M, Flora H. Mirrored, imagined and executed movements differentially activate sensorimotor cortex in amputees with and without phantom limb pain. Pain. 2010 May;149(2):171-2. [11] Chan BL, Witt R, Charrow AP, Magee A, Howard R, Pasquina PF, Heilman KM, Tsao JW. “Mirror therapy for phantom limb pain.” N Engl J Med. 2007 Nov 22;357(21):2206-7. [12] Weeks SR, Anderson-Barnes VC, Tsao JW. “Phantom Limb Pain: Theories and Therapies.” Neurologist. 2010 Sep;16(5):277-86.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-737

737

Validation of a Virtual Preoperative Evaluation Clinic: A Pilot Study Corey V. ZETTERMAN, MDa,b , Bobbie J. SWEITZER, MDc, Brad WEBB, MPASa , Mary A. BARAK-BERNHAGEN, BSa and Ben H. BOEDEKER, MD a.b,1 a University of Nebraska Medical Center, Omaha, NE b Omaha VA Medical Center, Omaha, NE c University of Chicago Medical Center, Chicago, IL

Abstract. Patients scheduled for surgery at the Omaha VA Medical Center were evaluated preoperatively via telemedicine. Following the examination, patients filled out a 15 item, 5 point Likert scale questionnaire regarding their opinion of preoperative evaluation in a VTC format. Evaluations were performed under the direction of nationally recognized guidelines and recommendations of experts in the field of perioperative medicine and were overseen by a staff anesthesiologist from the Omaha VA Medical Center. No significant difficulties were encountered by the patient or the evaluator regarding the quality of the audio/visual capabilities of the VTC link and its ability to facilitate preoperative evaluation. 87.5% of patients felt that virtual evaluation would save them travel time; 87.5% felt virtual evaluation could save them money; 7.3 % felt uncomfortable using the VTC link; 12.2 % felt the virtual evaluation took longer than expected; 70.7 % preferred to be evaluated via VTC link; 21.9 % were undecided; 9.7% felt they would rather be evaluated face-to-face with 26.8 % undecided; 85.0 % felt that teleconsultation was as good as being seen at the Omaha surgical evaluation unit; 7.5 % were undecided. Our study has shown that effective preoperative evaluation can be performed using a virtual preoperative evaluation clinic; patients are receptive to the VTC format and, in the majority of cases, prefer it to face-to-face evaluation. Keywords. Preoperative evaluation, telemedicine, virtual preop, perioperative medicine

Introduction Telemedicine has the potential to offer patients timely, evidenced based care in a costeffective format, and may provide an avenue for the healthcare industry to save patients and healthcare providers both time and money. In addition, telemedicine also has the potential to improve access to underserved populations especially in rural areas. Preoperative evaluation is an essential component in the efficiency of operating room performance as well as being vital to patient safety. The authors contend that effective preoperative evaluation can be performed using a telemedicine-based format, and that patients will have a positive perception of the virtual preoperative evaluation.

1

Corresponding Author: Ben H. Boedeker, MD, PhD, Professor, Department of Anesthesiology, Director, Center for Advanced Technology & Telemedicine, University of Nebraska Medical Center, 984455 Nebraska Medical Center, Omaha, NE 68198-4455, USA; E-mail: [email protected]

738

C.V. Zetterman et al. / Validation of a Virtual Preoperative Evaluation Clinic: A Pilot Study

Tools and Methods Following IRB approval, preoperative evaluations were performed using a VTC link to the provider at the Omaha VA Medical Center in Omaha, Nebraska. Two clinics were established for evaluation. One within a Urology clinic at the Omaha VA Medical Center and another at an Ophthalmology clinic at the Lincoln VA located 50 miles from Omaha. An anesthesia research fellow located at the Omaha VA performed the preoperative evaluations. The evaluation process adhered to established national guidelines and recommendations of experts in the field of perioperative medicine, and was overseen by a staff anesthesiologist from the Omaha VA Medical Center. The evaluations consisted of a health history interview, review of health records, and review of any diagnostic testing ordered prior to surgery. Recommendations were made regarding further testing or consultation and patient education regarding the risks and benefits of the anesthesia as well as recommendations for preparation for their surgery were discussed Following evaluation, the patients completed a 15-item, 5-point Likert scale questionnaire regarding their perceptions of virtual preoperative evaluation. Five questions pertained to the video and audio quality of the VTC link. Four questions asked for the patients to appraise the potential benefits of virtual preops. Of the remaining questions, three assessed the patient’s comfort level with the VTC interface and three asked the patients about their preferences regarding being seen virtually versus being seen face-to-face for preoperative evaluation. The patient’s course was tracked to ascertain whether further delays or case cancellations occurred. In the event of cancellations, the records were reviewed to find if there was anything that could have been done during the preoperative evaluation that might have prevented the cancellation.

Results Forty-one patients were evaluated at the Urology clinic in Omaha, NE. Their responses to the questionnaire are shown in Table 1. Table 1: Patient perception responses to virtual preoperative evaluation I could talk freely to the examiner during the teleconsultation I could hear everything that was being said I could see the pictures on the screen clearly. The examiner was able to ask me questions. The picture quality on the screen was as good as on my TV at home It is an advantage to be seen at the Pre-op clinic to prevent travel to Omaha A teleconsultation could reduce stress on patients by preventing travel A teleconsultation could save me time A teleconsultation could save me money The TV camera made me feel uncomfortable I was embarrassed using the link to speak to the examiner The appointment took longer than expected

Agree 41/41

Don’t Know

40/41 40/41 41/41 40/41

Disagree

1/41 1/41 1/41

37/39

2/39

40/41

1/41

35/40 35/40 1/41 3/41

4/40 5/40

5/41

3/41

1/40 40/41 38/41 33/41

C.V. Zetterman et al. / Validation of a Virtual Preoperative Evaluation Clinic: A Pilot Study

I would prefer a teleconsultation I would prefer to see the examiner face to face in Omaha A teleconsultation is as good as going to Omaha Anesthesia Pre-op Clinic

29/41 4/41

9/41 11/41

3/41 26/41

34/40

3/40

3/40

739

Patient outcomes were tracked to follow cancellations and delays that could have been prevented during the virtual preoperative evaluations. To date none of the patients seen have had their surgeries cancelled on the day of surgery for insufficient preoperative evaluation. One patient awaits further medical evaluation. Seven await their scheduled surgeries.

Discussion This study confirms findings of previous teleanesthesiology pilot program studies [1]. It highlights that patients not only are positive about the experience of virtual evaluation, that the majority of them prefer it. Some of these patients were undergoing their first anesthesia evaluation and thus, may not have been able to form an informed opinion of whether virtual evaluation was superior to face-to-face evaluation. In addition, we have shown that proper evaluation can be performed in that to date none of the patients seen in our virtual clinic have had a day of surgery cancellation due to incomplete preoperative work up. This demonstrates another area of healthcare where evidenced-based medicine can be performed using telemedicine [2]. Some of the patients seemed unsure of possible time or money that could be saved by virtual preoperative evaluation. Several studies have examined the cost-benefit analysis of telemedicine [3, 4]. Teleanesthesia has not been specifically studied; however, benefits noted in other studies regarding travel and transport could easily be superimposed on virtual preoperative evaluation.

Conclusion This study demonstrates that effective preoperative evaluations can be performed using a VTC format which provides good quality sound and picture, is easy and comfortable for the patients to use, and patients feel that it saves them time and money and in the majority of cases the patients preferred to be evaluated via VTC link.

References [1] [2] [3] [4]

B.H. Boedeker, W.B. Murray, B.W. Berg. Patient perceptions of pre-operative anaesthesia assessment at a distance, J Telemed Telecare 13 (2003), 22-24. S. Shea. The informatics for diabetes and education telemedicine (IDEATEL) project, Trans Am Clin Climatol Assoc 118 (2007), 289-304. K.M. Jackson, K.E. Scott, J.G. Zivin, D.A. Bateman, et al. Cost-utility analysis of telemedicine and ophthalmoscopy for retinopathy of prematurity management, Arch Opthalmol 126 (2008), 493-499. The value of provider-to-provider telehealth technologies. Center for Information Technology Leadership, Healthcare Information and Management System Society, Chicago, IL, 2007.

740

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-740

Multifunction Robotic Platform for Natural Orifice Surgery Xiaoli ZHANGa,1, Wei Jian CHINa, Chi Min SEOWa, Akiko NAKAMURAa, Michael HEADa, Shane FARRITORa, c, Dmitry OLEYNIKOVb,c, and Carl NELSON a,b,c a Dept. of Mechanical Engineering, University of Nebraska-Lincoln b Dept. of Surgery, University of Nebraska Medical Center (UNMC) c Center for Advanced Surgical Technology, UNMC

Abstract. A new robotic platform for natural orifice surgery is described. The robot is designed to carry multiple tool tips in a single end-effector arm. Design and experimental validation are presented. Although the design is still being improved, results suggest that the new robotic tool will enable dexterous abdominal surgery with improved force transmission capability. Keywords. Multifunction robotic platform, natural orifice surgery

Introduction Natural orifice transluminal endoscopic surgery (NOTES) is a newly emerging paradigm to reduce the invasiveness of surgery by avoiding incisions in patients’ skin. With this technique, faster recovery, improved cosmetics and skin infection avoidance can be achieved. As a result, patient advantages are significant compared to minimally invasive surgery [1]. This concept, although widely accepted among surgeons, has yet to be successfully implemented due to a lack of enabling tools and technology [2]. Available commercial products include Olympus’ R Scope and USGI Medical’s Transport and Cobra [3]. Typical approaches involve the working channel paradigm, whereby tool tips are actuated via wire cable externally. Common problems include triangulation of the target tissue, size and dexterity of instruments, and poor force transmission and/or precision. Swamstrom et al. [4] suggested that one critical element of development to make NOTES a practical reality is to design more appropriate instrumentation. A stable platform was also recommended because current flexible scopes lack tip stability required to permit exposure and retraction. Lehman et al. originated a dexterous miniature in vivo robot for NOTES incorporating separate arms for triangulation [5]. Micromotors were used to operate the surgical tools directly to provide more force. To increase mobility and stability, the robot was attached to the upper area of the abdominal cavity through an external magnet, allowing easy repositioning. However, due to the lack of number of surgical tools available on the robot, it could only perform basic procedures. The goal of this study is to design and develop a multifunction robotic platform capable of overcoming the issues encountered by the currently available systems. This platform 1

Corresponding Author: Xiaoli Zhang, N104 Scott Engineering Center, Lincoln, NE 68588-0656, USA; E-mail: [email protected]

741

X. Zhang et al. / Multifunction Robotic Platform for Natural Oriﬁce Surgery

enables tool-changing with tool tips controlled directly by micromotors at the surgical site and facilitates stable positioning throughout the abdominal cavity. The aim is to make NOTES feasible in a wide variety of surgeries. Scissor

Clamp

Grasper

Figure 1. The multifunction robotic platform for NOTES.

Figure 2. The multifunction tool head.

Methods and Materials A multifunction robotic platform (Figure 1) has been designed specifically for NOTES. The system architecture consists of (a) a multifunction robotic manipulator, (b) an articulated drive mechanism, and (c) a surgeon control console with haptic interface. The multifunction robotic manipulator (Figure 2) contains a cartridge holding three surgical end effectors (e.g., graspers, scissors, and atraumatic Babcock clamp) and two micromotors. The motors, located at the proximal end of the cartridge, work collaboratively to deploy, engage, operate, and switch between three end effectors. Due to orifice size constraints, all the components including the motors have to fit inside a cylindrical container with a 21mm diameter and 85mm height. The steerable and lockable articulated drive mechanism (Figure 3) functions as a platform for the robot during surgery. It consists of a steerable articulated linkage tube and a motor housing. Each linkage piece has a diameter of 14mm and a length of 32.5mm. The top [bottom] of each piece has a convex [concave] surface, providing a large contact surface area. A central cable acts as a locking mechanism to pull the linkages together and create a stable platform utilizing friction force. The drive mechanism is controlled through 4 additional wire cables. Two directional cables work antagonistically to provide movement in the yaw axis while the other two enable manipulation in the pitch axis. The maximum relative angle achievable in the spherical joint connecting each piece is 30°, which allows the mechanism to safely navigate through a natural orifice.

Figure 3. The articulated drive mechanism with the robotic tool changer.

Figure 4. The articulated mechanism holds in position under joystick control.

742

X. Zhang et al. / Multifunction Robotic Platform for Natural Oriﬁce Surgery

The portable surgeon control console contains the user interface mechanism (Phantom Omni 6-DOF haptic joystick), and a monitor. The joystick allows intuitive operation and enables haptic feedback for communicating workspace constraints. Bench-top tests have been performed to validate the design. The joystick-controlled articulated drive mechanism can move up/down and left/right with good dexterity (Figure 4). It also maintained static equilibrium with an arbitrary 5N load at the distal end. Considering that the articulating pieces will be partially supported from surrounding tissues, the drive mechanism will be able to hold the robotic manipulator in position during surgical procedures. The tool tips are able to provide 16N at the nearly closed posture, which is more than enough to cut a porcine liver [6].

Results The bench-top tests reveal that the system increases the effectiveness and efficiency of surgery and that the motor-driven system improves force transmission as compared to traditional working-channel approaches to tool design. The effectiveness is acquired by providing surgeons appropriate surgical tool functionalities during surgery. As for efficiency, the time and risk involved in manually removing and switching tools outside the patient’s body is reduced. Once the manipulator is maneuvered inside the abdominal cavity, the articulated drive mechanism provides a stable platform for the robot through its shape-lockable property. In the near future, clinical experiments using the multifunction robotic platform will be performed on a porcine model for functional validation and design improvements.

Conclusions We have developed a robotic surgical platform with sufficient strength, dexterity, and multifunctionality to be manipulated by a surgeon through an intuitive interface. It provides intuitive remote control, haptic workspace constraints, sufficient rigidity for surgical tasks and exhibits multifunctionality by integrating multiple tools. This may encourage a more widespread implementation of NOTES in operating rooms.

References [1] A.N. Kalloo, D. Rattner, W. Brugge, C. Gostout, R. Hawes, S. Kantsevoy, M. Marohn, J. Parischa, J. Ponsky, W. Richards, R. Rothstein, N. Soper, L. Swanstrom, and C. Thompson, ASGE/SAGES working group on Natural Orifice Translumenal Endoscopic Surgery (white paper), Gastrointext Endoscopy 62 (2005), 199-203. [2] R.R. Mummadi, and P.J. Pasricha, The eagle or the snake: platforms for NOTES and radical endoscopic therapy, Gastrointest Endoscopy 18 (2008), 279-289. [3] S. Bardaro, and L. Swanstrom, Development of advanced endoscopes for Natural Orifice Translumenal Endoscopic Surgery (NOTES), Minim Invasive Ther Allied Technol 15 (2006), 378-383. [4] L. Swanstrom, M. Whiteford, and Y. Khajanchee, Developing essential tools to enable transgastric surgery, Surg Endosc 22 (2008), 600-604. [5] A. Lehman, J. Dumpert, N. Wood, L. Redden, A. Visty, S. Farritor, B. Varnell, and D. Oleynikov, Natural orifice cholecystectomy using a miniature robot, Surg Endosc 23 (2009), 260-266. [6] M. Rentschler, J. Dumpert, S. Platt, D. Oleynikov, and S. Farritor, Mobile in vivo biopsy robot, IEEE International Conference on Robotics and Automation (2008).

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-743

743

Maintaining Forward View of the Surgical Site for Best Endoscopic Practice Bin ZHENGa,1 Maria A. CASSERA b; Lee L. SWANSTRÖMb; Adam MENEGHETTIa; Neely O. N. PANTONa; and Karim A. QAYUMIa a Department of Surgery, University of British Columbia, CANADA b Legacy Health System, Portland, Oregon, USA

Abstract. Endoscopic surgery performed through patients’ natural orifices (NOTES procedures) often require some degree of retroflexion of the operating system. This can cause a misalignment between the displayed image and the actual work plane, leading to performance difficulties. This study investigated the impact of retroflexion on task performance in a simulated environment. Surgeons were required to perform an aiming and pointing task under two experimental conditions: forward-view vs. retroflexed-view. Results showed that both expert and novice surgeons required significantly longer time for completing the task when the scope was retroflexed, compared to when the scope faced forwards. Results address the importance of careful selection of the surgical approach to avoid image retroflexion. Further analysis revealed that the novices were more vulnerable than experts to image distortion with the retroflexed view. This addresses the necessity for surgeons to go through extensive endoscopic training to overcome the visual-motor challenges before they can perform NOTES procedures safely and effectively. Keywords. Endoscopic surgery, NOTES procedure, Vision-motion alignment, Simulation, Evaluation

Introduction One promising surgical technology that has been developing intensively over the last a few years is constructed on the platform of flexible endoscopy [1-4]. With the recent advances in image display and remote manipulation technologies, surgery on the abdominal cavity is feasible via various natural orifices such as the mouth, anus, urethra, and vagina. This novel surgical paradigm is named Natural Orifice Transluminal Endoscopic Surgery (NOTES). In NOTES, a transvisceral incision is made on a hollow organ to gain access to organs in the peritoneal cavity [5]. By eliminating the need for abdominal incisions, NOTES has the potential to reduce the incidence of surgical site hernias, abdominal adhesions, post-operative incisional pain, and eliminates any visible scarring. While the minimally invasive and cosmetic benefits of NOTES are desirable for patients, NOTES poses new and greater technical challenges for surgeons compared to open or even laparoscopic surgery[3, 6]. From the human factors standpoint, the most 1

Corresponding Author. 3602-910 W. 10th Ave. Vancouver, British Columbia, Canada, V5Z 4E3, Canada; Email: [email protected]

744

B. Zheng et al. / Maintaining Forward View of the Surgical Site for Best Endoscopic Practice

serious challenge of NOTES is the perception-motion integration problem, i.e. how to maintain dexterity in manipulating surgical instruments under an unstable and distorted view of the surgical site. In the most commonly performed NOTES procedure where the gallbladder is removed through the patient’s mouth (Transgastric Cholecystectomy), the tip of the endoscope is twisted backward after entering the abdominal cavity through a transgastric port. In this situation, the distal portion of the scope is flipped over 135 – 175 degrees with respect to the frontal plane [2, 4]. When displayed on the monitor, the images must be interpreted incompletely upside-down by both the surgeon and assistant controlling the instruments. Previous studies outside healthcare have shown that incomplete image distortion creates the worst scenario for mental calibration in remote manipulation, even worse than 180 degree image reversal [7]. Task performance was jeopardized when the image of the working place was distorted. In this study, we intend to quantify the impact of image distortion caused by endoscope retroflexion on performance in a simulated NOTES task. We hypothesize that when the surgical site is viewed in the retroflexed fashion, surgeons who perform the task must readjust their movements appropriately to accommodate the retroflexed image, in turn prolonging the overall task time compared to when the site was viewed in the straight forward view. To test our hypothesis, a drylab simulated NOTES task was performed by a group of NOTES surgeons under two viewing conditions. We expect that our results will help to provide guidelines for setting which surgical approach to take in a given NOTES operation.

1. Methods & Materials 1.1. Apparatus and Task Data collection was conducted in the surgical skills training laboratory. Our simulation model for NOTES was comprised of a wooden plate (14.5 inch in height, 14 inch in width, and 0.5 inch in thickness) and a hollow plastic sphere (5inches in diameter). A small opening placed behind the plate with no direct line of sight to the operator. The plate was fixed to a lab table, and a hole was created on the lower portion of the plate. A single channel therapeutic endoscope (Olympus, Tokyo, Japan) was passed through this hole and then aimed to the entrance of the plastic sphere (Figure 1 A). There were 4 colored dots located around a central point on the inner surface of the sphere. Each dot was labeled with a number 1-5, including the central point. Each participant was required to visually locate these 4 dots and touch the dots in a predefined order using a grasper inserted through the working channel on the endoscope (Figure 1 B).

B. Zheng et al. / Maintaining Forward View of the Surgical Site for Best Endoscopic Practice

745

Figure 1. Experiment setup (A) and sphere used in the pointing task (B)

1.2. Procedure This task was performed under two experimental conditions: 1) Forward view condition: where the entrance of the sphere faces the operator; 2) Retroflexed view condition: the entrance of the sphere is opposite to the operator. Under the second condition the participant was required to retroflex the tip of the scope over 150 degrees to capture the image of the dots (Figure 1). Each participant was allowed to complete a trial once prior to testing in order to verify a correct understanding of the task. No additional task training was given. All participants executed the task five times for each experimental condition. For each trial, the task time was calculated from the moment when the grasper was deployed from the tip of the endoscope and touched the central point, to the moment when the fifth dot was correctly touched. 1.3. Participants A total of 9 participants were recruited for this study, 8 completed required trials over two experimental conditions and data was included for further analysis. Participants include 3 laparoscopic surgeons with extensive experience in flexible endoscopic procedures, and 5 subjects naïve to laparoscopic or endoscopic procedures. A pre-test questionnaire was given to all participants to assess endoscopic and laparoscopic experience, as well as to record demographic data. 1.4. Evaluation and Statistical Analysis All tasks were video-recorded. Task performance score was evaluated by task completion time (in seconds). A 2 x 2 ANOVA (two groups of surgeons for two viewing conditions) was performed using SPSS statistical software (SPSS, SPSS Inc, IL, USA) to compare mean times for each group. P < 0.05 was considered significant. Results are reported as mean ± standard deviation unless otherwise stated.

746

B. Zheng et al. / Maintaining Forward View of the Surgical Site for Best Endoscopic Practice

2. Results Table 1 summarizes the pre-trial characteristics of two groups. Table 1. Pre-trial characteristics of two groups Characteristics Number of participants

Novice 5

Expert 3

Age

26.8 ± 4.2

41.1 ± 7.6

0~3

5~20

0~10

>50

Years of performing laparoscopic surgery Number of performing endoscopy procedure/year

Task performance in the forward-view condition was significantly faster (35 ± 15 sec) than performance in the retroflexed-view condition (51 ± 31 sec, P < .001). On average, the experts finished tasks in shorter time (19 ± 6 sec) than the novices (60 ± 20 sec, P < .001). Secondary analysis of the interaction between view condition and surgeon’s group revealed that experts and novices responded differently to each image viewing condition. The experts performed slightly worse in the retroflexed view (20 ± 6 sec) then the forward view (18 ± 5 sec) condition. In contrast, the novices were much more vulnerable to image distortion; their performance deteriorated significantly in the retroflexed condition (72 ± 21 sec) compared to the forward condition (47 ± 6 sec, P = .002).

3. Discussion Results support our research hypothesis – retroflexed image does impede task performance in NOTES. The reason, we believe, can be attributed to the eye-hand coordination difficulty related to the NOTES procedure. Unlike laparoscopy which requires one level of mental calibration by changing the viewing perspective from the eyes to the scope, NOTES requires additional mental work because the viewing perspective of the endoscope is constantly changing during the procedure [8]. The endoscope must constantly be maneuvered to maintain the horizon and keep track of spatial orientation. The changing perspective of the endoscope also internally changes the configuration of surgical instruments. When performing NOTES procedures with the endoscope retroflexed, another level of mental calibration must be included in the mental adjustment of a surgeon, which can make a surgeon easily lose orientation and dexterity. Loss of orientation and dexterity brings up significant safety concerns [3, 8]. We argue that when possible, the surgical approach needs to be chosen carefully to avoid using the retroflexed view during any NOTES procedure. Currently both forward and retroflexed view approaches are commonly available for a given NOTES procedure. For example, removing a patient’s gall bladder (cholecystectomy) has been achieved through both transgastric (retroflexed) and transvaginal/transcolon (forward view) approach. Now that we have demonstrated that retroflexion has negative impact on surgical task performance, we argue that surgeons should consider a transvaginal/transcolon approach for cholecystectomy.

B. Zheng et al. / Maintaining Forward View of the Surgical Site for Best Endoscopic Practice

747

It is interesting to observe that experienced surgeons were able to perform tasks in the retroflexed condition with minimal delay in comparison to the novice group. This is mainly due to the fact that experts are already experienced having performed large volume of endoscopic procedures on a daily basis. Extensive endoscopic experience allows experts to develop sophisticated cognitive strategies to deal with misalignment between perception and movement as presented by NOTES procedures [9]. Evidence presented in this study indicates that extensive training is required for a novice surgeon to overcome the difficult vision-motion coordination before they can perform NOTES effectively and safely. There are a number of limitations related to this study. The first limitation was that successful performance in a true endoscopic surgery requires skills much more complicated than those needed for the aiming and pointing task used in this study. The second limitation was that only a single surgeon was required to perform the pointing task, unlike the more commonly practiced surgical scenario that requires at least two surgeons work in a team for a NOTES procedure. Recently, we have incorporated bimanual coordination tasks into a new NOTES simulation model which was constructed on a double channel endoscopic platform. Two surgeons are allowed to work side-by-side, one to control the scope, the other to manipulate instruments on the surgical site. Replication of the current study with this new model will help to improve the generalization of our findings to a clinical setting. The third limitation was in the measurement used in the study. We used time to completion to describe the observable impact of visual-motion misalignment on the task performance. In any goal-direct movement such as the task we incorporated in this study, before the observable action, there is a period of cognitive process where environmental information is processed and an appropriate movement is planned[10]. This cognitive process is more sensitive to visual-motion alignment condition, rather than execution of the chosen movement plan. A superior measurement for the cognitive process would be the reaction time, defined as the time from the moment where visual information is presented to an operator, to the moment a movement is performed [11]. Future studies on the human factors of NOTES procedures will integrate the reaction time to measures, to give a comprehensive description of the impact of visual-motion misalignment on surgeons’ performance. In conclusion, the retroflexed view condition in NOTES procedure built on an endoscopic platform has a negative impact on surgeon’s performance. Careful planning is required for selecting an appropriate approach to avoid retroflexion and subsequent image distortion. To ensure safe performance of NOTES procedure, extensive endoscopic training is recommended for general surgeon before they can perform NOTES effectively and safely.

4. Acknowledgments This project has been funded by NOSCAR (Natural Orifice Surgery Consortium for Assessment and Research) research grant in 2007. The authors wish to thank the Boston Scientific Corporation for providing experimental devices of this study.

748

B. Zheng et al. / Maintaining Forward View of the Surgical Site for Best Endoscopic Practice

References [1] [2]

Kavic MS. "Natural orifice translumenal endoscopic surgery: "NOTES"",JSLS, 10(2),133-4, 2006. Bardaro SJ, Swanström L. "Development of advanced endoscopes for Natural Orifice Transluminal Endoscopic Surgery (NOTES)", Minim Invasive Ther Allied Technol, 15(6),378-83, 2006. [3] Volckmann ET, Hungness ES, Soper NJ, Swanstrom LL. "Surgeon Perceptions of Natural Orifice Translumenal Endoscopic Surgery (NOTES)", J Gastrointest Surg, 2009. [4] Swanstrom L, Swain P, Denk P. "Development and validation of a new generation of flexible endoscope for NOTES", Surg Innov, 16(2),104-10, 2009. [5] Sclabas GM, Swain P, Swanstrom LL. "Endoluminal methods for gastrotomy closure in natural orifice transenteric surgery (NOTES)", Surg Innov, 13(1),23-30, 2006. [6] Swanstrom LL, Volckmann E, Hungness E, Soper NJ. "Patient attitudes and expectations regarding natural orifice translumenal endoscopic surgery", Surg Endosc, 23(7),1519-25, 2009. [7] Kim W, Tendick F, Stark L. "Visual enhancements in pick-and-place tasks: Human operators controlling a simulated cylindrical manipulator", IEEE Robot Autom Mag, 3(5),418 - 425, 1987. [8] Swanstrom L, Zheng B. "Spatial Orientation and Off-Axis Challenges for NOTES", Gastrointest Endosc Clin N Am, 18(2),315-24, 2008. [9] Thompson CC, Ryou M, Soper NJ, Hungess ES, Rothstein RI, Swanstrom LL. "Evaluation of a manually driven, multitasking platform for complex endoluminal and natural orifice transluminal endoscopic surgery applications (with video)", Gastrointest Endosc, 2009. [10] MacKenzie CL, Iberall T. The Grasping Hand. Amsterdam; New York: North-Holland; 1994. [11] Martenuik RG, MacKenzie CL. "Methods in the study of motor programming: is it just a matter of simple vs. choice reaction time? a comment on klapp et al. (1979)", J Mot Behav, 13(4),313-9., 1981.

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved. doi:10.3233/978-1-60750-706-2-749

749

Phenomenological Model of Laser-Tissue Interaction with Application to Benign Prostatic Hyperplasia (BPH) Simulation Xiangmin ZHOUa,1, Nan Zhang a, Yunhe SHENa, Dan BURKEa, Vamsi KONCHADAa, Robert SWEETa a Center for Research in Education and Simulation Technologies (CREST), University of Minnesota, Minneapolis, MN

Abstract. Laser-tissue interaction is a multi-physics phenomenon not yet mathematically describable and computationally predictable. It is a challenge to model the laser-tissue interaction for real time laser Benign Prostatic Hyperplasia (BPH) simulation which requires the laser-tissue interaction model to be computationally efficient and accurate. Under the consideration and enforcement of the thermodynamic first law and treating the laser-tissue interaction as a graybox, utilizing the sensitivity analysis of some key parameters that will affect the laser intensity on the tissue surface with respect to the tissue vaporization rate, a phenomenological model of laser-tissue interaction is developed. The developed laser-tissue interaction model has been implemented for a laser BPH simulator and achieves real time performance (more than 30 frames per second). The model agrees well with the available experimental data. Keywords. Laser tissue interaction, Phenomenological Model

Introduction Benign prostatic hyperplasia (BPH) or "enlarged prostate" is a non-cancerous increase in the size and number of cells that make up the prostate. As the prostate enlarges, it impinges the flow of urine through the urethra. BPH is a common problem suffered by the majority of aged men. It causes a number of urinary symptoms, such as frequent urinating, urgent urinating, nocturia, and hesitancy. Traditionally, Transurethral Resection of the Prostate (TURP) is the long-standing BPH treatment of choice, where surgeons use an electrical loop to cut tissues piece by piece and seal blood vessels. Recently, laser Photoselective Vaporization of the Prostate (PVP) has emerged as a safe, less invasive and effective alternative to the "gold standard'' of TURP procedure. In laser PVP, surgeons perform the high-energy laser therapy, a form of heat therapy which vaporizes the overgrown prostate tissue, to provide swift symptom relief. Compared to TURP, laser therapy generally causes less bleeding and shorter hospital stay. It also does not cause impotence or prolonged incontinence.

1

Corresponding Author. E-mail: [email protected]

750

X. Zhou et al. / Phenomenological Model of Laser-Tissue Interaction

During the surgical procedure, a specially designed fiber optic delivery system is used, which is non-contact and side-firing. The fiber optic device is inserted into the urethra of the patient through a standard cystoscope, which is a tube-like instrument used to view the interior of the bladder. The laser light pulses Figure 1: Illustration of laser PVP procedure. are then directed toward the enlarged prostate tissue. The surgeon slowly drags the tip of the laser fiber along the prostatic urethra from the bladder neck to the level of the verumontanum to create a furrow of evaporated tissue. This process is shown in Figure 1. The surgeon repeats this process to create multiple furrows along the prostatic urethra, until a patent bladder outlet is resulted in. The average operative time is typically less than sixty minutes. Once the procedure has been completed, most patients experience immediate symptom relief and a dramatic improvement in symptoms such as urinary flow rate and bladder emptying. Inherently, every surgery is invasive, which may cause unwanted damage to a patient. For example, in laser PVP, the sphincter (the valve that turns the urine flow on or off) may be mistakenly vaporized off. The unique interface style of laser PVP requires the surgeon to acquire different skills than conventional open or laparoscopic surgery. Therefore, to overcome the learning curve in this therapy, new surgical teaching methods have to be developed. The development of simulators can facilitate the transfer of surgical skills to novice surgeons. For example, simulators allow the naive surgeon to develop skills and pass the learning curve without the medico–legal implications of surgical training, limitations in trainee working hours, and ethical considerations of learning basic skills on humans. Furthermore, simulators allow a trainee to gain experiences without increasing the risk to patients’ safety, e.g. making errors which are not allowed in real surgery. In building the laser BPH simulator, an appropriate laser-tissue interaction model is crucial. It is required that (a) the laser-tissue interaction can accurately predict the vaporization volume with respect to the system setting such as power and operating parameters such as working distance and treatment speed; (b) the computational complexity of the laser-tissue interaction model need to be suitable for the real-time simulation. The second requirement eliminates the feasibility of modeling the lasertissue interaction via the physical based approach. And the only feasible approach to model the laser-tissue interaction is the phenomenological approach. However, based on the limited experimental data available in the literature, it is insufficient to build a phenomenological model.

X. Zhou et al. / Phenomenological Model of Laser-Tissue Interaction

751

1. Laser-tissue interaction modeling In order to build the phenomenological model of laser-tissue interaction, we treat the laser-tissue interaction as gray-box. And we choose the inputs as the operating power, sweep speed, and working distance for the laser beam. The output of the model is chosen as the tissue vaporization rate. Form the physical phenomenon of laser-tissue interaction, we have the following assumption: a) at any given time, the tissue vaporization rate is limited by the operating power; b) there is a threshold limit for the linear dependence between the laser intensity and the tissue vaporization rate (volume per time); c) beyond the threshold limit, the vaporization rate is directly correlated to the power, d) below the threshold limit, the vaporization rate is linearly with the insufficient data available from the literature, we are able to construct a laser-tissue interaction model that is suitable for the real time surgical simulation. Given the operating environment of the Laser BPH therapy, the laser is interacting with the soft tissue in a fluid environment. The thermal interaction of the laser-tissue interaction involves coagulation and vaporization. Coagulation of the soft tissue occurs when the temperature of the underlying tissue is reaching 60°C and the thermal damage is induced. For the 532 nm wave length KTP laser that we are modeling, the depth of coagulation zone is consistently 0.8 mm regardless of the power setting of the laser beam (within the maximum power of 120W) and the working distance (the distance between the fiber surface and the tissue surface) [1]. Vaporization of the soft tissue occurs when the temperature of the underlying tissue is reaching 100°C and the water contained in the tissue is vaporized. Since the BPH therapy is operating in a fluid environment, and the coagulation zone of the soft tissue is always 0.8 mm depth, this implies that the peak temperature during the laser-tissue interaction is a minimum of 100°C. As a consequence, there exists a vaporization threshold power, Pv, of the laser beam for the occurrence of the vaporization phenomenon. Below the threshold power, the soft tissue cannot reach the vaporization temperature of 100°C, and no vaporization of the tissue should occur. When the power of the laser beam is higher than the vaporization power, the surface temperature of the tissue reaches 100°C and vaporization occurs. If the laser intensity on the tissue surface is increased beyond the vaporization threshold power, the excessive energy will contribute to the vaporization effect and the tissue surface temperature will keep increasing until the columns or slugs boiling effect occurs. This phenomenon causes the vaporization of the tissue reach a saturation state and the excessive energy is lost to the surrounding fluid due to the increased rate of heat transfer for the nucleate boiling. As the consequence, the relation between the tissue vaporization rate and the working distance is not linear. Utilizing the thermodynamic first law, the heat balance locally describing the laser ablation of the soft-tissue processes can be written as the following for a given domain of an open set and with the boundary B. @ d ` \ +g g g g d r r @? Where H is the enthalpy of which the phase change due to ablation is accounted for, k is the thermal conductivity, T is the temperature, ρb is the blood density, cb is the blood heat capacity, wb is the blood perfusion rate, Tb is the blood temperature, Qm is the metabolic heat generation, and Ql is the volumetric laser heat source. The volumetric laser heat source is obtained by

752

X. Zhou et al. / Phenomenological Model of Laser-Tissue Interaction

Q_ Where α is the absorption coefficient, I is the laser intensity, and ω’ is the solid angle. The laser intensity is described by the solution of the following differentialintegral equation of the transport equation. r £!¤ ` r r ¡¢ where k is the scattering coefficient, s is the laser direction, and p is the probability density function for scattering of which the Henyey-Greenstein phase function can be adopted as an approximation. Along with the trivial initial condition of the body temperature and the Neumann boundary conditions to account for the non-reflective solid boundary, the heat lost due to conduction/convection/radiation of the prostate surface, the above equation completely describes the local heat balance. However, to solve the governing equation to determine the soft tissue vaporization is not feasible, especially with respect to the application of surgical simulation. Firstly, some of the parameters and processes associated with the laser tissue interaction such as the soft tissue absorption coefficient, the scattering coefficient, the exact laser beam profile, and the boiling nucleation of soft tissue are not yet clear. Secondly, an appropriate numerical solution of the governing equation to determine the energy balance and transfer locally is impractical for the real time application requirement of the surgical simulation. Thus a phenomenological model of the laser tissue interaction to account for the global energy balance and transfer is more practical and preferable for the application of surgical simulation. In a global sense, with respect to the thermodynamic first law, the energy balance of the laser tissue interaction processes can be described as the following. r r r r ¥ r ¦ Where Qp is the due to blood perfusion, Qc is due to thermal damage and tissue denaturation, Qh is due to heating up the ablated tissue to the boiling temperature, Qa is due to phase change of the ablation, and Qb is the boundary condition. However, consider the fact that the power of the laser beam is adjustable, with respect to the sensitivity of the laser beam power, the global energy balance yields, ¥ K§ ¨ Q Q¥ § ª Q© Q© Q¥ « K§ ¬ f Q© Where P is the power, and Ic is the coagulation intensity, and Is is the saturation intensity. Ic corresponding to the laser intensity that is not enough to heat the soft tissue to the boiling temperature. There are two conditions that I ≤ Ic could happen. The first condition is that the laser beam power is less than 20 Watt, the second condition is that the fiber tip is too far away from the tissue surface and with the fact that the 120 Watt fiber has a 15° of divergent angle. The existent of the saturation intensity is due to physical constraint that there is a speed limit to heat the tissue to the boiling temperature due to conduction. When the laser intensity is greater than the saturation

X. Zhou et al. / Phenomenological Model of Laser-Tissue Interaction

753

intensity, the excesses energy is transferred to the ambient fluid environment due to nucleate boiling. Based on the above qualitative analysis and using the treatment speed TS (speed of the laser beam sweeping across the tissue surface, in mm/s), laser beam power P (in Watt), and working distance WD (in mm) as parameters, a phenomenological model of laser tissue interaction for the vaporization speed function is constructed as, where VS is the vaporization speed in mm3/s, and f(TS), h(P), and g(WD) are effects of the vaporization speed as functions of TS, P, and WD, respectively. The three functions are determined from experimental data. Utilizing the data provided from [1], we have,

where Pv=20W is the vaporization threshold power. Although the parameters of our model are determined using the experimental data, the available information is sufficient only for 80W laser power setting. We extrapolate the model to handle different laser power settings, e.g.; from 20W to 120W continuously. To validate the proposed model, the predicted results are compared with the experimental data from [1]. Comparisons are shown in Figures 2 and 3, which demonstrate that the proposed model can predict the behavior of the laser tissue interaction accurately within experimental errors.

Figure 2: Comparison of the proposed model and the experimental results.

754

X. Zhou et al. / Phenomenological Model of Laser-Tissue Interaction

Figure 3: Comparison of the proposed model and the experimental results.

2. Simulation Results Our simulation system is implemented on a Windows PC platform, which contains Intel Core2 Duo E6600 CPU, 4GB memory, and NVIDIA 8800GTX graphics board. We use an open source graphics package, called OGRE, as the rendering engine. OGRE is used for rendering special effects. Although our system is not designed with multi-threading support, we take advantage of the CPU/GPU parallelism. When all the rendering commands are submitted to the GPU, the main program does not wait for the rendering operations to be finished. Instead, it begins to process user input and perform geometry updates, including CSG difference evaluation, isosurface extraction, topology cleansing to remove tiny pieces, and collision handling. We test our system using several prostate models with different sizes. The largest one has a bounding box size of 61 x 52 x 60 mm and about 100 cm3 volume. In contrast, the laser beam has a diameter of only 0.75 mm. The largest model in our system contains about 344k vertices and 1.95 million tetrahedral elements. For all experiments, the grid cell size is set as approximately the maximum effective range of the laser beam, which is about 8 mm. Unless specified explicitly, we always use the laser power setting of 80 Watt and laser beam slope angle of 8 degrees. Figure 4 shows the simulation results, and Figure 5 shows the comparison with the operation video.

Figure 4: Laser PVP simulation.

X. Zhou et al. / Phenomenological Model of Laser-Tissue Interaction

755

3. User Validation We have asked two internal surgeons who are experienced in laser BPH therapy to validate our prototype implementation. In general, they were satisfied with the realism of the virtual environment created in our training system. From surgeons' point of view, realism in the behavior of the tissue vaporization model is more important than that of the graphical appearance of the prostate model. They reported that the Figure 5: Comparison of the simulation with the operation video. proposed phenomenological laser-tissue interaction model yielded a result that was very close to what the urologists felt in the operating room. Furthermore, we have sent our system to an annual urological conference for evaluation. About 40 urologists have tested the system. Although we were unable to schedule a comparison on both algorithms, in general, the experts were satisfied with the melting effect generated from our algorithm and the melting speed. Encouraged by this success, we are in the stage of planning some nation-wide, more rigorous user study activities. Details of the new updates and the results of validation study will appear in our future report. 4. Conclusion A phenomenological model of laser tissue interaction is proposed based on the qualitative study for the sensitivity of the global energy balance with respect to the laser intensity (power) is proposed. The proposed model not only can capture all the available experimental data points but also suitable for the real time application of the surgical simulation. And the proposed approach is also suitable to characterize the laser tissue interaction for different laser fiber designs if appropriate experimental data is available.

Acknowledgement Funding from American Medical System (AMS) is acknowledged. Computer support from Minnesota Supercomputer Institute (MSI) is gratefully acknowledged.

References [1] H. W. Kang, D. Jebens, R. S. Malek, G. Mitchell, and E. Koullick. Laser vaporization of bovine prostate: A quantitative comparison of potassium-titanyl-phosphate and lithium triborate lasers. The Journal of Urology, 180 (2008), 3675-2680

This page intentionally left blank

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved.

757

Subject Index 3 dimensional models 96 3D gaze calibration 616 3D interaction 372 3D lung dynamics 567 3D muscle 560 3D segmentation 552 3D visual guidance 400 3D visualization 372 3D-cranio-mandibular model 261 accuracy 524 activities of daily living (ADL) 730 adaptive signal processing 60 affective computing 132 aging 510 AISLE 476 anatomy 18, 96, 264, 280, 397 anatomy navigation system 354 anxiety disorder 696 Arbitrary Lagrangian-Eulerian method 710 arrhythmia 57 arthritis 18 arthroscopy 236 artificial neural networks 25 assessment 8, 86, 304, 493, 496 attention 192 augmented reality 336, 408 autism 132 autistic spectrum disorder 132 automatic volumetric segmentation 476 avatar collaboration 372 barriers to care 503 biofeedback 696 biofilms 394 bio-imaging 138 biomanipulation 231 biomechanical model 560 biopsy 242, 623 biosensors 86, 185, 496 bipolar disorders 496 bougie 65 BPH 574

brain anatomy 105 brain dynamics 329 brain neuro-machine interfaces 163 BRDF 105 bronchoscopy simulation 535 CAD/CAM 239 cadaver 397 cancer 691 cardiac 57 cardiac surgery 150, 716 catheter 594 cell microinjection 231 clinical breast exam 408 clinical examination 271 C-MAC 369 cognition 428 collaborative stereo visualization 264 collision detection 555, 560 computational fluid dynamics 567 computed tomography 18, 680 computer aided psychotherapy 44 computer graphics 389 computer vision 581 computer-based assessment 517 Computerized Cognitive Behavior Therapy (CCBT) 86 confirmation 611 connected components 359 containment detection 560 content validity 274 contextualized learning 144 corneas 653 Cosserat rod 466 CPU-GPU balancing scheme 354 curriculum 150 cutting simulation 311 CvhSlicer 354 CyberMed 386 cyberpsychology 44 630 da VinciTM Surgical System DARPA 730 data sets 670 data visualization 685

758

datasets 677 deformable surface models 560 deformation model 645 dental 653 depression 86, 496, 696 direct laryngoscopy 71 disability 510 dissection 397 dysesthesia 680 ecological validity 433 60 ECRTM education 57, 119, 264 educational content 242 EEG 329, 606 elastic object 638, 645 e-learning 202 electrical discharges 297 electrocautery 166, 311 electromagnetic tracking 479 electromagnetics 329 electromyography 630 60 electronic competency recordTM electroporation 297 emergency response 650 emotion sensing 132 emotions 44 Endo Stitch 461 endoscope 594 endoscopic surgery 743 Endoscopic Third Ventriculostomy 1 endotracheal intubation 400, 611 endovascular simulation 317 error analysis 304 ETI 611 eTraining 650 evaluation 743 evaluation/methodology 535, 542 face validity 274 feedback 119 fiberscope 68, 713 fidelity 147, 670 finite element method 415, 663 finite elements 31 flow-volume curve 25 force sensing 408 fractal dimension 606 fuzzy control 39 game-based learning 147, 254

gas phase plasma 297 gastrointestinal endoscopy 199 guide wire 594 guidewire/catheter 317 habituation 696 hand and wrist 397 haptic collaborative virtual environment 638 haptic feedback 224 haptic interface 199 haptic rendering 112, 645 haptics 135, 213, 397, 542, 555, 588, 591, 670, 691 hardware 135, 677 head modeling 329 head-mounted eyegaze tracker 658 hernioplasty 202 hierarchical segmentation 599 hip 18 hip re-surfacing 283 human-centered computing 535, 542 human computer interaction 400, 549, 552 hybrid reality 552 hydration 653 hydrocephalus 1 image analysis 138 image guidance 716 image guided surgery 283, 479 impulse response deformation model 645 indirect laryngoscopy 77 inertial measurement unit 479 infrastructure 723 integration 93 intelligent tutoring systems 60 interactive learning 254 interactive remote visualization 635 interactive simulation framework 213 internet 86 interreality 185 interscalene block 36 intubation 65, 68, 71, 74, 366 intubation training 77, 80, 83, 549, 688 knowledge transfer 147 language design 209 laparoscopic simulators 588, 591 laparoscopic surgery simulator 658

759

laparoscopic surgery 11, 581 laparoscopic training 588, 591 laparoscopy 348 laryngoscopy 74, 400 laser 394, 713 laser tissue interaction 749 layered depth images 224 learning curve 524 learning technologies 535, 542 lesions 359 level set 599 levels of realism 147 liver 348 localization 329 lower extremities 290 lung allometry 476 lung radiotherapy 567 machine vision 11 major incident response 650 Mammacare® 408 mass casualty 650 mass-spring model 317 master slave 524 medical 677 medical artist 397 medical education 173 medical robotics 716 medical simulation 199, 277, 542, 581 medical student education 271 medical training 277, 650 medical training simulator 51 meditation 696 metamorphopsia 336 micromanipulator 524 military healthcare 503 minimally invasive surgery 454, 723 mixed reality 144, 552 mobile eyetracking 616 mobile technologies 86, 496 modeling and simulation 156 modification 135 motion tracking 280 motor learning 119 motor skills 192 motor-neuroprosthetics 163 MRI 552, 716 MRI compatible robot 623 multifunction robotic platform 740

multi-level computer performance systems 638 multiple sclerosis 359 multi-tasking 192 Musculoskeletal Modeling Software (MSMS) 730 myoelectric prostheses 156 natural orifice surgery 740 Naval Hospital Camp Pendleton (NHCP) 696 navigation system 713 Navy Medical Center San Diego (NMCSD) 696 needle insertion 135 needle insertion simulation 710 network 93 neurofeedback 606 Neuropsychological assessment 433 neurorobotics 163 neurosurgery 51, 166 NeuroVR 8, 493 non-contact position sensing 549 NOTES procedure 743 numbness 680 occlusal contacts 261 Office of Naval Research (ONR) 696 Off-Pump Coronary Artery Bypass Surgery 147 open source 493 open surgery 202 operating room 93 ophthalmology 560 optical tracking 403 orthopedic surgery 283, 324 out-of-hospital 80 pain management 606 palpation 408 pancreas 691 parallel FEM 415 Parkinson’s disease 8 particle method 389 patent 351 patient education 96 patient model 524 patient specific surgical simulation 379 patient training and rehabilitation 156

760

patient-specific 447 patient-specific instrument guides 283 patient-specific model 112, 415 pectus excavatum 473 pelvic floor muscle 218 pelvis 280 penetration volume 224 perception 588 perceptual motor learning 428 percutaneous minimally invasive therapy 710 perioperative medicine 737 peripheral nerve block 552 Personal Health Systems 86, 496 phantom limb pain (PLP) 730 phenomenological model 749 physically-based simulation 461, 466 physics simulation 213 physiological monitoring 696 piezoelectric driven injector 231 piezoresistive sensor 703 plasma-medicine 297 pneumatic balloon actuator 703 pneumoperitoneum 348 postoperative 425 Posttraumatic Stress Disorder (PTSD) 696 precision 524 preoperative evaluation 737 preoperative 425 presence 44 prevention 86 probabilistic tractography 486 projective augmented-reality display 549 prostate brachytherapy 623 prosthodontic 422 prototype 351 psychological treatments 44 psychophysiology 433 pulmonary function test 25 PVP 574 real time 594 real-time interaction 236 real-time simulation 31, 213 real-time spatial tracking 400 reconstruction 447

regional anesthesia 36, 119 rehabilitation 163, 290, 703 rehabilitation robotics 39 remote consultant 93 renal surgery training 415 rendering 105 respirometry 25 Revolutionizing Prosthetics 2009 730 robot assisted surgery 274 robotic devices 247 robotic surgery 703 robotic surgical simulation 379 role-playing 173 Second Life 440 segmentation 138, 359 semantics derivation 209 sensor fusion 479 sensors 535, 542 sensory augmentation 703 serious games 147, 254, 606 shockwaves 394 SimCoach 503 SimTools 611 simulation 119, 125, 135, 150, 166, 173, 202, 271, 324, 447, 517, 599, 611, 630, 650, 677, 723, 743 simulation and 3D reconstruction 348 simulation-based training 400 simulation development 531 simulation meshes 670 Simulation Support Systems 535, 542 simulator maintenance 531 simulator 57, 242, 531 skills training 591 skin area 680 skin burns 653 369 SkypeTM sliding 663 SOFA 691 soft tissue grasping 663 software 677 software framework 343 software system 560 spiking neurons 685 stapedotomy 524

761

stereo imaging 454 stereo-endoscopy 1 stereolithography 18, 552 stereoscopic vision 680 stress 86, 185, 496 stress management 185 stress-related disorders 44 stroke 39 SUI 218 surgery 691 surgery simulation 31, 224, 311, 574 surgery simulation development 209 surgery training 11 surgical navigation 479 surgical planner 473 surgical rehearsal 112 surgical robotics 454 surgical robots setup 379 surgical simulation 112, 144, 236, 535 surgical simulator 389, 415 survey 277 suture 461, 466 suturing 31 system design 304 tactile feedback 703 task analysis 277 task decomposition 277 taxonomy 677 teaching 397 teaching curriculum 36 technical skills 517 technology 510 Technology Enhanced Learning 324 technology transfer office 351 telehealth 425 teleimmersion 290 telemedicine 93, 369, 425, 737 telerehabilitation 290 temporal bone surgery 112 terahertz imaging 653 therapy 493, 496 tongue retractor 68 training 125, 304, 447, 630, 723 training simulator 403 trajectory error 247 trauma 650 treatment plannning 422

tumor 691 two-handed interface 372 ultrasound 119, 138, 242, 447 ultrasound guided regional anesthesia 304 ultrasound image simulation 403 upper limbs 247 user interface design 343 user interfaces 549 user models 428 user studies 372 vaginal wall simulation 218 vergence eye movements 616 video conferencing 688 video laryngoscope 65, 74, 77, 80, 83, 366, 369 video laryngoscopy 71, 688 virtual environment 433, 555, 594 virtual humans 503 Virtual Integration Environment (VIE) 730 virtual patient 144, 173, 408, 440, 650 virtual preop 737 Virtual Reality 8, 44, 51, 96, 156, 163, 185, 202, 280, 264, 304, 386, 389, 397, 422, 428, 486, 493, 496, 510, 552, 574, 581, 685, 716 virtual reality articulator 239 Virtual Reality Graded Exposure Therapy (VR-GET) 696 Virtual Reality Therapy 696 virtual simulation 147, 254 virtual training 274 virtual world 125, 173, 440, 650 vision-motion alignment 743 visual impairment 336 Visual Programming 386 visualization 343, 549 VMET 8 Voice over Internet Protocol (VoIP) 83, 369 volume rendering 112, 264, 372, 635 volumetric 691 VTC 369, 425 Walter Reed Military Amputee Research Program (MARP) 730 War on Terror 696

762

web-based web-based visualization web-enabling

96 635 264

wound wound closure X3D

394 461 670

Medicine Meets Virtual Reality 18 J.D. Westwood et al. (Eds.) IOS Press, 2011 © 2011 The authors. All rights reserved.

763

Author Index Abhari, K. Aggarwal, R. Agin, K. Ahn, W. Aisen, M. Akbar, M. Al-Allaq, Y. Alasty, A. Albani, G. Alcañíz, M. Allen, B.F. Allen, P. Amunts, K. Ando, E. Ankeny, M.L. Anstey, J.B. ap Cenydd, L. Arabalibeik, H. Arai, T. Aratow, M. Ardanza, A. Argun, O.B. Arikatla, V.S. Arizmendi, B. Armiger, R.S. Asghari, M. Atkins, M.S. Avis, N. Awad, I.T. Ayres, F. Babaeva, N.Yu. Backstein, D. Baillargeon, E. Baitson, G. Bajcsy, R. Bajd, T. Banerjee, P. Baniasad, M.A. Baños, R. Barak-Bernhagen, M.A. Barner, K.E. Batrick, N. Beenhouwer, D.

1 440, 650 25 213 510 39 119 39 8 44, 348 11 552 486 422 691 18 105 25 713 670 163 280 31, 311 433 730 663 658 304 36, 119 166 297 192, 254 549 304 290 290 510 39 44, 86, 496 77, 351, 737 224, 691 650 394

Beier, F. Bello, F. Bennett, D. Berg, D.R. Bergeron, B. Bernhagen, M.A.

51 202, 317, 599 653 57 60 65, 68, 71, 74, 80, 83, 688 Bhargava, A. 567 Biersdorff, S. 329 Bisley, J.W. 703 Bito, T. 638 Blevins, N.H. 112 Bloj, M. 105 Blossfield Iannitelli, K. 271, 531 Boedeker, B.H. 65, 68, 71, 74, 77, 80, 83, 351, 366, 369, 425, 688, 737 Boedeker, K.A. 74, 77 Boronyak, S. 552 Botella, C. 44, 86, 496 Branstetter, B. 552 Bregman, R. 549 Brown, A. 549 Brown, B. 611 Brown, E.R. 653 Bucholz, R.D. 93 Buckwalter, J.G. 503 Bulpitt, A. 599 Burck, J.M. 730 Burg, K.J.K.L. 588, 591 Burg, T.C. 588, 591 Burke, D. 96, 280, 574, 749 Cabral, A. 166 Caldwell, D.G. 135 Callahan Jr., J.B. 366, 369 Cameron, B.M. 99 Camilo, A. 623 Campbell, A.R. 680 Carelli, L. 8 Carlson, A. 57 Carrasco, E. 163 Carson, C. 574 Caspers, S. 486

764

Cassera, M.A. 743 Chan, S. 112 Chandrasekhar, R. 274 Charnoz, A. 403 Chaya, A. 549 Chen, B. 138 Chen, C.-P. 691 Chen, D. 311 Chen, E.W. 36, 119 Chen, H.Y. 231 Cheung, J.J.H. 36, 119 Chevreau, G. 242 Chien, J.-h. 428, 630 Chiesa, C. 366, 369 Chin, W.J. 740 Chodos, D. 125 Chowriappa, A. 274 Chui, Y.P. 354 Chung, S.Y. 132 Cipresso, P. 185, 493 Cline, A. 60 Cochran, A. 653 Cohen, D. 650 Cole, G.A. 623 Coles, T.R. 135 Constantinou, C.E. 138, 218 Courteille, O. 144 Courtney, C.G. 433 Cowan, B. 147 Cristancho, S. 147, 150, 254, 517 Culjat, M.O. 653, 703 Darzi, A. 440, 650 Davey, C. 329 Davies, M.G. 716 Davoodi, R. 156, 730 Dawson, M. 433 de Hoyos, A.L. 535 De Mauro, A. 163 de Ribaupierre, S. 1, 180 De, S. 31, 209, 213, 311, 555 Dech, F. 264 Delorme, S. 166 Deng, Z. 247, 716 Dev, P. 173 Devadhas, D. 691 Diederich, S. 51 Diesmann, M. 685 Dietz, A. 524 Difede, J. 503

Dillmann, R. Din, N. Dindar, S. Dittrich, E. Divo, E. Domont, Z.B. Donvito, G. Dubrowski, A.

635 599 723 524 567 271 493 36, 119, 147, 150, 192, 254, 517 Dumpert, J. 454 Durfee, W.K. 57 Dutson, E.P. 11, 703 Eagleson, R. 1, 180 Einhäuser, W. 616 Ellis, R.E. 18, 283, 680 Enders, L. 549 Enochsson, L. 144 Erdman, A.G. 280 Erhart, K. 567 Evestedt, D. 670 Faloutsos, P. 11 Fan, R. 703 Farahmand, F. 39, 663 Farritor, S. 454, 740 Felländer-Tsai, L. 144 Ferrari, M. 379 Ferrari, V. 379 Fischer, G.S. 623 Florez, J. 163 Foo, J.L. 343 Forbell, E. 503 Forest, C. 403 Fors, U. 144 Frantz, F.W. 473 Frizera Neto, A. 163 Fukushima, S. 239, 261, 422 Gaggioli, A. 8, 185, 493, 496 Gagliati, A. 493 García-Palacios, A. 44, 86 Gasco, S. 493 Gil, A. 163 Goretsky, M. 473 Gould, D.A. 135, 317, 599 Grassi, A. 185, 493 Grierson, L. 192, 517 Grundfest, W.S. 394, 653, 703 Gu, Y. 199 Gupta, V. 394 Guru, K. 274

765

Haake, D.A. Haase, R. Hald, N. Halic, T. Hammond, D. Hasegawa, S. Hata, N. Hattori, A. Head, M. Hedman, L. Hein, C. Hein, S. Heinrichs, W.L. Hemstreet, G.P. Hemstreet, J.L. Heng, P.A. Hirabayashi, R. Hirai, S. Ho, Y. Hofer, M. Hoge, C. Holmes III, D.R. Hostettler, A. How, T. Hu, R. Huang, H.B. Huang, M.H. Hubschman, J.P. Hughes, C.J. Hurmusiadis, V. Ikawa, T. Ilegbusi, O. Inuiya, T. Jafari, S. Jain, S. Janssoone, T. Jerald, J. Jiang, D. Jin, Y.-Y. John, N.W. Johnsen, K. Johnston, S. Jose, S. Jowlett, N. Juhas, M. Juhnke, B. Kadivar, Z. Kairys, J.C. Kaltofen, T.

394 524 202 209, 213 329 218 623 239, 261, 422, 713 740 144 447, 611 552 173 425 425 354 239 261 403 524 329 99 403 317 224, 691 231 403 653 594 236 239, 261, 422, 713 567 415 25 574 242 372 166 691 105, 135, 594, 670 408 696 343 192 552 343 247 691 560

Kapralos, B. 147, 254 Karmakar, M. 354 Kasama, S. 239, 261, 422, 713 Kaspar, M. 264 Kasper, F. 11 Kassab, A. 567 Katz, N. 8 Kaye, A.R. 271 Kazanzides, P. 476, 479 Kealey, C.P. 653 Kelly, R. 473 Kenny, P. 503 Kerr, K. 650 Kesavadas, T. 274 Khoramnia, R. 336 Kiely, J.B. 730 Kim, J. 503 Kim, Y. 581 King, D. 599 King, S. 125, 180 Kizony, R. 8 Kjellin, A. 144 Knisley, S.B. 473 Knott, T. 277, 677 Koffman, R.L. 696 Kohlbecher, S. 616 Konchada, V. 96, 280, 574, 749 Konety, B. 96 Korenblum, D. 138 Koritnik, T. 290 Kubota, Y. 415 Kuhlen, T. 277, 486, 670, 677, 685 Kunz, M. 283 Kupelian, P.A. 567 Kuper, G.M. 80, 83 Kurenov, S. 461, 466 Kurillo, G. 290 Kushner, M.J. 297 Kwak, H. 581 Kwok, W.H. 354 Lacy, T. 74 Lago, M.A. 348 Lange, B. 503, 510 Lanzl, I. 336 Larcher, A. 324 Laycock, K.A. 93 Lee, D. 581 Lee, D.H. 112 Lee, D.Y. 199

766

Li, K. 329 Li, P. 112 Liebschner, M. 247 Lind, D.S. 408 Lindgren, G. 144 Linnaus, A. 68 Littler, P. 317 Loeb, G.E. 156, 730 Long, X. 567 López-Mir, F. 348 Lövquist, E. 304 Lu, Z. 213, 311, 555 Luboz, V. 317, 599 Luengo, V. 324 Lüth, T. 524 Luzon, M.V. 594 Machado, L.S. 386 MacNeil, W.R. 93 Maier, S. 635 Makiyama, K. 415 Malony, A.D. 329 Männer, R. 51 Marescaux, J. 403 Markin, N. 688 Martinez-Escobar, M. 343 Martínez-Martínez, F. 348 Martin-Gonzalez, A. 336 Marx, S. 616 Matsuo, K. 351 Mauro, A. 8 McCartney, C.J.L. 36, 119 McDurmont, L.L. 93 McKenzie, F.D. 473 McLay, R.N. 696 Meeks, S.L. 567 Melnyk, M. 192 Meneghetti, A. 743 Meng, Q. 354 Merdes, M. 479 Merians, A. 510 Meruvia-Pastor, O. 359 Miljkovic, N. 80, 83, 366, 369 Miller, D.J. 65, 68, 74, 80, 83, 351, 366, 369 Mills, J.K. 231 Min, Y. 567 Mirbagheri, A. 663 Mitchell, R. 567 Mlyniec, P. 372

Moglia, A. 379 Moncayo, C. 150 Monclou, A. 150, 254, 517 Monserrat, C. 348 Moragrega, I. 86 Morais, A.M. 386 Moran, C. 730 Morgan, J.S. 688 Morganti, F. 8 Morikawa, S. 710 Mosca, F. 379 Moussa, F. 147, 150 Mozer, P. 242 Mukai, N. 389 Mukherjee, M. 630 Murray, W.B. 65, 68, 71, 77, 80, 83 Nagasaka, M. 415 Nakagawa, M. 389 Nakamura, A. 740 Nataneli, G. 11 Navab, N. 336 Navarro, A. 394 Needham, C. 397 Neelakkantan, H. 567 Nelson, C. 740 Nelson, D.A. 400, 549, 552 Neumuth, T. 524 Nguyen, M.K. 606 Nicholas IV, T.A. 68, 71 Nicolau, S.A. 403 Niki, K. 389 Nikravan, N. 119 Niles, T. 408 Noon, C. 343 Nuss, D. 473 O’Malley, M. 247 O’Sullivan, O. 304 Odetoyinbo, T. 317 Ogata, M. 415 Ogawa, T. 239, 261, 422, 713 Oh’Ainle, D. 304 Oleynikov, D. 428, 454, 630, 740 Omata, S. 218 Orebaugh, S. 552 Owen, H. 611 Oyarzun, D. 163 Ozawa, T. 713 Pagano, C.C. 588, 591 Paik, J. 428

767

Pallavicini, F. Panton, N.O.N. Papp, N. Park, E.S. Park, S. Park, S.-H. Parsad, N.M. Parsons, T. Pasquina, P.F. Patel, V. Patton, J. Peddicord, J. Peloquin, C. Peniche, A.R. Pérez, L.C. Peters, J. Peters, T. Petrinec, K. Phillips, N. Pignatti, R. Pinto, R. Polys, N.F. Pons, J.L. Pop, S.R. Porte, M. Potjans, T.C. Prabhu, V.V. Priano, L. Psota, E. Pugh, C.M. Punak, S. Pyne, J. Qayumi, K.A. Qin, J. Quero, S. Ragusa, G. Rank, D. Raspelli, S. Rasquinha, B. Rechowicz, K.J. Reger, G. Reihsen, T. Ren, H. Requejo, P. Rettmann, M.E. Rhode, K. Rick, T. Ritter, F.E. Riva, G.

185, 493 743 567 425 581 428, 630 264 433, 503 730 440, 650 510 343 343 535 454 723 1 447 105 8 552 670 163 594 254 685 591 8 454 271, 531, 535, 542 461, 466 696 743 354 44 510 479 8, 185, 493 18 473 503 57 476, 479 510 99 236 486, 685 428 8, 185, 493, 496

Rizzo, A.A. 503, 510 Robb, R.A. 99 Robinson, E. 549 Roehrborn, C. 574 Rojas, D. 517 Rolland, J.P. 567 Rothbaum, B.O. 503 Rotty, V. 96 Rudan, J.F. 18, 283 Ruddy, B.H. 567 Rueda, C. 150, 517 Runge, A. 524 Runge, H.J. 351 Rupérez, M.J. 348 Sabri, H. 147 Sagae, K. 503 Salisbury, J.K. 112 Salman, A. 329 Salud, J.C. 531, 535 Salud, L.H. 271, 531, 535, 542 Samosky, J.T. 400, 549, 552 Sanders, J.M. 691 Sankaranarayanan, G. 31, 213, 555 Santana Sosa, G. 560 Santhanam, A.P. 567 Sarker, S.K. 202 Sarosi, G. 723 Satake, K. 710 Sathyaseelan, G. 274 Savitsky, E. 447 Schaeffter, T. 236 Schmieder, K. 51 Schneider, E. 616 Schrack, R. 630 Schulte, N. 366 Schultheis, U. 372 Seagull, F.J. 372 Seixas-Mikelus, S. 274 Sensen, C.W. 359 Seow, C.M. 740 Sevdalis, N. 650 Shen, Y. 96, 280, 574, 749 Sherman, K. 236 Shigeta, Y. 239, 261, 422, 713 Shin, S. 581 Shirai, Y. 710 Shorten, G. 304 Silverstein, J.C. 264 Sinclair, C. 36

768

Singapogu, R.B. 588, 591 Singh, R.S. 653 Siu, K.-C. 428, 630 Smelko, A. 552 Smith, E.J. 18 Soames, R. 397 Sofia, G. 135 Soh, J. 359 Soler, F. 594 Soler, L. 403 Song, J.E. 329 Song, Y. 599 Sourina, O. 606 Spira, J.L. 696 Sprick, C. 611 Srimathveeravalli, G. 274 Stallkamp, J. 479 Stegemann, A. 274 Steiner, K.V. 224, 691 Stoll, J. 616 Strabala, K. 454 Strauss, G. 524 Strauss, M. 524 Stroulia, E. 125, 180 Su, H. 231, 623 Suh, I.H. 428, 630 Sukits, A.L. 549 Sung, C. 247 Suwelack, S. 635 Suzuki, N. 239, 261, 422, 713 Swanström, L.L. 743 Sweet, R. 96, 574, 749 Sweet, R.M. 57, 280 Sweitzer, B.J. 737 Syed, M.A. 716 Tagawa, K. 638, 645 Takanashi, S. 389 Tanaka, H.T. 638, 645, 710 Taylor, D. 440, 650 Taylor, Z.D. 394, 653 Tempany, C.M. 623 Tewari, P. 653 Thakur, M.L. 691 Thompson, M. 552 Thompson, Z. 247 Tien, G. 658 Tirehdast, M. 663 Toledo, F. 372 Tonetti, J. 324

Torres, J.C. Torricelli, D. Troccaz, J. Tsao, J.W. Tsekos, N.V. Tucker, D. Turini, G. Turovets, S. Ullrich, S. Unterhinninghofen, R. Vadcard, L. Vanberlo, A.M. Vemuri, A. Venkata, S.A. Vigna, C. Villard, P.-F. Volkov, V. von Kapri, A. Walker, M.I. Walker, R.B. Walter, A. Wampole, M. Wang, Q. Weaver, R.A. Webb, B. Webb-Murphy, J. Weeks, S.R. Wei, D. Weinhaus, A. Weiss, P.L. Westwood, J.D. White, S. Wickstrom, E. Wiederhold, B.K. Wiederhold, M.D. Wilding, G. Wilkinson, C. Williams, J. Winer, E. Winstein, C.J. Wong, C. Wood, D.P. Wood, G.C.A. Wottawa, C. Wu, H.S. Xiao, M. Yamaguchi, S. Yamamoto, T. Yamazaki, Y.

594 163 242 730 716 329 379 329 277, 670, 677 635 242 680 403 213 493 202 329 486, 685 688 688 105 691 606 549, 552 369, 737 696 730 218 280 8 v 653 691 185, 496, 696 696 274 397 503 343 510 574 696 283 703 403 359 710 713 713

769

Yeniaras, E. Yeo, Y.I. Yoganandan, A. Yoon, H.J. Yoshida, Y. Youngblood, P. Zeher, M.J. Zetterman, C.V.

716 723 372 132 218 173 730 737

Zhai, J. Zhang, N. Zhang, X. Zheng, B. Zhou, X. Zilles, K. Ziprin, P.

317 574, 749 740 658, 743 96, 574, 749 486 202

This page intentionally left blank

Medical And Care 3 (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Medical and Care Compunetics 2 (Studies in Health Technology and Informatics, Vol. 114) (Studies in Health Technology and Informatics)

Research into Spinal Deformities 5 (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Medicine Meets Engineering: Proceedings of the 2nd Conference on Applied Biomechanics Regensburg - Volume 133 Studies in Health Technology and Informatics

Plaque Imaging: Pixel to Molecular Level (Studies in Health Technology and Informatics, Vol. 113) (Studies in Health Technology and Informatics)

Connecting Medical Informatics and Bio-Informatics (Studies in Health Technology and Informatics)

Designing User Studies in Informatics (Health Informatics)

E-Health: Current Situation And Examples Of Implemented & Beneficial E-Health Applications (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

The Conservative Scoliosis Treatment:1st SOSORT Instructional Course Lectures Book (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Virtual Reality in Healthcare Market

eHealth Beyond the Horizon - Get IT There:Proceedings of MIE2008 (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Medicine Meets Virtual Reality 18: NextMed - Volume 163 Studies in Health Technology and Informatics

Medical And Care 3 (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Global Health Informatics Education (Studies in Health Technology and Informatics)

Advances in Information Technology and Communication in Health - Volume 143 Studies in Health Technology and Informatics

International Perspectives in Health Informatics - Volume 164 Studies in Health Technology and Informatics

Virtual Reality Technology

Medical and Care Compunetics 2 (Studies in Health Technology and Informatics, Vol. 114) (Studies in Health Technology and Informatics)

Research into Spinal Deformities 5 (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Medicine Meets Engineering: Proceedings of the 2nd Conference on Applied Biomechanics Regensburg - Volume 133 Studies in Health Technology and Informatics

Plaque Imaging: Pixel to Molecular Level (Studies in Health Technology and Informatics, Vol. 113) (Studies in Health Technology and Informatics)

Connecting Medical Informatics and Bio-Informatics (Studies in Health Technology and Informatics)

Designing User Studies in Informatics (Health Informatics)

E-Health: Current Situation And Examples Of Implemented & Beneficial E-Health Applications (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Virtual Reality in Geography

Virtual Reality

Biological Petri Nets - Volume 162 Studies in Health Technology and Informatics

Virtual Reality & Augmented Reality in Industry

Virtual Society? Get Real!: Technology, Cyberbole, Reality

Medicine Meets Virtual Reality 14: Accelerating Change in Healthcare: Next Medical Toolkit

Regional Health Economies and ICT Services (Studies in Health Technology and Informatics Shti)

Evaluation Methods in Biomedical Informatics (Health Informatics)

The Conservative Scoliosis Treatment:1st SOSORT Instructional Course Lectures Book (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Virtual Reality in Healthcare Market

eHealth Beyond the Horizon - Get IT There:Proceedings of MIE2008 (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Artificial Life and Virtual Reality

Artificial Life and Virtual Reality

MEDINFO 2007: Proceedings of the 12th World Congress on Health (Medical) Informatics (Studies in Health Technology and Informatics)

Genome Informatics 2007 (Genome Informatics Series, Volume 18)

Surgical Scene Generation for Virtual Reality-Based Training in Medicine

Surgical Scene Generation for Virtual Reality-Based Training in Medicine

Information Technology Solutions for Healthcare (Health Informatics)

Medicine Meets Virtual Reality 18: NextMed - Volume 163 Studies in Health Technology and Informatics

Medical And Care 3 (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Global Health Informatics Education (Studies in Health Technology and Informatics)

Advances in Information Technology and Communication in Health - Volume 143 Studies in Health Technology and Informatics

International Perspectives in Health Informatics - Volume 164 Studies in Health Technology and Informatics

Virtual Reality Technology

Medical and Care Compunetics 2 (Studies in Health Technology and Informatics, Vol. 114) (Studies in Health Technology and Informatics)

Research into Spinal Deformities 5 (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Medicine Meets Engineering: Proceedings of the 2nd Conference on Applied Biomechanics Regensburg - Volume 133 Studies in Health Technology and Informatics

Plaque Imaging: Pixel to Molecular Level (Studies in Health Technology and Informatics, Vol. 113) (Studies in Health Technology and Informatics)

Connecting Medical Informatics and Bio-Informatics (Studies in Health Technology and Informatics)

Designing User Studies in Informatics (Health Informatics)

E-Health: Current Situation And Examples Of Implemented & Beneficial E-Health Applications (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Virtual Reality in Geography

Virtual Reality

Biological Petri Nets - Volume 162 Studies in Health Technology and Informatics

Virtual Reality & Augmented Reality in Industry

Virtual Society? Get Real!: Technology, Cyberbole, Reality

Medicine Meets Virtual Reality 14: Accelerating Change in Healthcare: Next Medical Toolkit

Regional Health Economies and ICT Services (Studies in Health Technology and Informatics Shti)

Evaluation Methods in Biomedical Informatics (Health Informatics)

The Conservative Scoliosis Treatment:1st SOSORT Instructional Course Lectures Book (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Virtual Reality in Healthcare Market

eHealth Beyond the Horizon - Get IT There:Proceedings of MIE2008 (Studies in Health Technology and Informatics) (Studies in Health Technology and Informatics)

Artificial Life and Virtual Reality

Artificial Life and Virtual Reality

MEDINFO 2007: Proceedings of the 12th World Congress on Health (Medical) Informatics (Studies in Health Technology and Informatics)

Genome Informatics 2007 (Genome Informatics Series, Volume 18)

Surgical Scene Generation for Virtual Reality-Based Training in Medicine

Surgical Scene Generation for Virtual Reality-Based Training in Medicine

Information Technology Solutions for Healthcare (Health Informatics)

Recommend Documents