Human-AI interactive and continuous sensemaking : A case study of image classification using scribble attention maps

Conference paper


Shen, Haifeng, Liao, Kewen, Liao, Zhibin, Doornberg, Job, Qiao, Maoying, van den Hengel, Anton and Verjans, Johan W.. (2021). Human-AI interactive and continuous sensemaking : A case study of image classification using scribble attention maps. CHI Conference on Human Factors in Computing Systems. Virtual 08 - 13 May 2021 pp. 1-8 https://doi.org/10.1145/3411763.3451798
AuthorsShen, Haifeng, Liao, Kewen, Liao, Zhibin, Doornberg, Job, Qiao, Maoying, van den Hengel, Anton and Verjans, Johan W.
TypeConference paper
Abstract

Advances in Artificial Intelligence (AI), especially the stunning achievements of Deep Learning (DL) in recent years, have shown AI/DL models possess remarkable understanding towards the logic reasoning behind the solved tasks. However, human understanding towards what knowledge is captured by deep neural networks is still elementary and this has a detrimental effect on human’s trust in the decisions made by AI systems. Explainable AI (XAI) is a hot topic in both AI and HCI communities in order to open up the blackbox to elucidate the reasoning processes of AI algorithms in such a way that makes sense to humans. However, XAI is only half of human-AI interaction and research on the other half - human’s feedback on AI explanations together with AI making sense of the feedback - is generally lacking. Human cognition is also a blackbox to AI and effective human-AI interaction requires unveiling both blackboxes to each other for mutual sensemaking. The main contribution of this paper is a conceptual framework for supporting effective human-AI interaction, referred to as interactive and continuous sensemaking (HAICS). We further implement this framework in an image classification application using deep Convolutional Neural Network (CNN) classifiers as a browser-based tool that displays network attention maps to the human for explainability and collects human’s feedback in the form of scribble annotations overlaid onto the maps. Experimental results using a real-world dataset has shown significant improvement of classification accuracy (the AI performance) with the HAICS framework.

KeywordsHuman centered computing; Collaborative interaction; HCI theory, concepts and models; Heat maps
Year2021
Digital Object Identifier (DOI)https://doi.org/10.1145/3411763.3451798
Scopus EID2-s2.0-85105775455
Publisher's version
License
All rights reserved
File Access Level
Controlled
Journal citationp. Article 290
Page range1-8
Book editorKitamura, Yoshifumi
Quigley, Aaron
Isbister, Katherine
Igarashi, Takeo
Output statusPublished
Publication dates
Online08 May 2021
Publication process dates
Deposited11 Oct 2021
Permalink -

https://acuresearchbank.acu.edu.au/item/8wwy6/human-ai-interactive-and-continuous-sensemaking-a-case-study-of-image-classification-using-scribble-attention-maps

Restricted files

Publisher's version

  • 2
    total views
  • 0
    total downloads
  • 2
    views this month
  • 0
    downloads this month
These values are for the period from 19th October 2020, when this repository was created.

Export as

Related outputs

Quality assessment in systematic literature reviews : A software engineering perspective
Yang, Lanxin, Zhang, He, Shen, Haifeng, Huang, Xin, Zhou, Xin, Rong, Guoping and Shao, Dong. (2021). Quality assessment in systematic literature reviews : A software engineering perspective. Information and Software Technology. 130, p. Article 106397. https://doi.org/10.1016/j.infsof.2020.106397
Processes, challenges and recommendations of Gray Literature Review : An experience report
Zhang, He, Mao, Runfeng, Huang, Huang, Dai, Qiming, Zhou, Xin, Shen, Haifeng and Rong, Guoping. (2021). Processes, challenges and recommendations of Gray Literature Review : An experience report. Information and Software Technology. 137, p. Article 106607. https://doi.org/10.1016/j.infsof.2021.106607
Inferring location types with geo-social-temporal pattern mining
Anwar, Tarique, Liao, Kewen, Goyal, Angelic, Sellis, Timos, Kayes, A. S. M. and Shen, Haifeng. (2020). Inferring location types with geo-social-temporal pattern mining. IEEE Access. 8, pp. 154789-154799. https://doi.org/10.1109/ACCESS.2020.3018997
An experimental evaluation of imbalanced learning and time-series validation in the context of CI/CD prediction
Liu, Bohan, Zhang, He, Yang, Lanxin, Dong, Liming, Shen, Haifeng and Song, Kaiwen. (2020). An experimental evaluation of imbalanced learning and time-series validation in the context of CI/CD prediction. EASE 2020, April 15-17, 2020, Trondheim, Norway. Norway: Association for Computing Machinery. pp. 21 - 30 https://doi.org/10.1145/3383219.3383222
Temporal betweenness centrality in dynamic graphs
Tsalouchidou, Ioanna, Baeza-Yates, Ricardo, Bonchi, Francesco, Liao, Kewen and Sellis, Timos. (2020). Temporal betweenness centrality in dynamic graphs. International Journal of Data Science and Analytics. 9(3), pp. 257 - 272. https://doi.org/10.1007/s41060-019-00189-x
An adaptive differential evolution algorithm to optimal multi-level thresholding for MRI brain image segmentation
Tarkhaneh, Omid and Shen, Haifeng. (2019). An adaptive differential evolution algorithm to optimal multi-level thresholding for MRI brain image segmentation. Expert Systems with Applications. 138, pp. 1 - 18. https://doi.org/10.1016/j.eswa.2019.07.037
Information visualisation methods and techniques: State-of-the-art and future directions
Shen, Haifeng, Bednarz, Tomasz, Nguyen, Huyen, Feng, Frank, Wyeld, Theodor, Hoek, Peter J. and Lo, Edward H.S.. (2019). Information visualisation methods and techniques: State-of-the-art and future directions. Journal of Industrial Information Integration. 16(100102), pp. 1 - 17. https://doi.org/10.1016/j.jii.2019.07.003
Training of feedforward neural networks for data classification using hybrid particle swarm optimization, mantegna levy flight and neighborhood search
Tarkhaneh, Omid and Shen, Haifeng. (2019). Training of feedforward neural networks for data classification using hybrid particle swarm optimization, mantegna levy flight and neighborhood search. Heliyon. 5(4), pp. 1 - 32. https://doi.org/10.1016/j.heliyon.2019.e01275
Contextual community search over large social networks
Chen, Lu, Liu, Chengfei, Liao, Kewen, Li, Jianxin and Zhou, Rui. (2019). Contextual community search over large social networks. 35th International Conference on Data Engineering (ICDE). Macao, China: IEEE Computer Society. pp. 88 - 99 https://doi.org/10.1109/ICDE.2019.00017
SORCER: A decentralised continuous integration platform for service-oriented software systems
Almalki, Jameel and Shen, Haifeng. (2019). SORCER: A decentralised continuous integration platform for service-oriented software systems. IEEE International Conference on Service-Oriented Computing Applications. United States of America: Springer International Publishing. pp. 458 - 464 https://doi.org/10.1007/978-3-030-17642-6_44
Developing cross-organisational service-based software systems through decentralised interface-oriented continuous integration
Almalki, Jameel and Shen, Haifeng. (2018). Developing cross-organisational service-based software systems through decentralised interface-oriented continuous integration. Australian Software Engineering Conference. United States of America: IEEE Computer Society. pp. 191 - 200 https://doi.org/10.1109/ASWEC.2018.00033
The interplay of factors affecting learning of introductory programming: A comparative study of an Australian and an Indian University
Sharma, Ritu and Shen, Haifeng. (2018). The interplay of factors affecting learning of introductory programming: A comparative study of an Australian and an Indian University. IEEE International Conference on Computer Science and Education. United States of America: IEEE Computer Society. pp. 669 - 674 https://doi.org/10.1109/ICCSE.2018.8468768
Integrating localization and energy-awareness: A novel geographic routing protocol for underwater wireless sensor networks
Hao, Kun, Shen, Haifeng, Liu, Yonglei, Wang, Beibei and Du, Xiujuan. (2018). Integrating localization and energy-awareness: A novel geographic routing protocol for underwater wireless sensor networks. Mobile Networks and Applications. 23(5), pp. 1427 - 1435. https://doi.org/10.1007/s11036-018-1093-0
A smartphone-based point-of-care quantitative urinalysis device for chronic kidney disease patients
Akraa, Shaymaa, Tam, Anh Pham Tran, Shen, Haifeng, Tang, Youhong, Tang, Ben Zhong, Li, Jimmy and Walker, Sandy. (2018). A smartphone-based point-of-care quantitative urinalysis device for chronic kidney disease patients. Journal of Network and Computer Applications. 115, pp. 59 - 69. https://doi.org/10.1016/j.jnca.2018.04.012
Extending attention span for children with ADHD using an attentive visual interface
Asiry, Othman, Shen, Haifeng, Balkhy, Soher and Wyeld, Theodor. (2018). Extending attention span for children with ADHD using an attentive visual interface. International Conference Information Visualisation. United States of America: IEEE Computer Society. pp. 188 - 193 https://doi.org/10.1109/iV.2018.00041
A fast algorithm for optimally finding partially disjoint shortest paths
Guo, Longkun, Deng, Yunyun, Liao, Kewen, He, Qiang, Sellis, Timos and Hu, Zheshan. (2018). A fast algorithm for optimally finding partially disjoint shortest paths. Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18), Stockholm, Sweden, July 13-19, 2018. Stockholm, Sweden: International Joint Conferences on Artificial Intelligence. pp. 1456 - 1462 https://doi.org/10.24963/ijcai.2018/202
On the feasibility of a smartphone-based solution to rapid quantitative urinalysis using nanomaterial bioprobes
Akraa, Shaymaa, Guo, Feng, Shen, Haifeng, Tang, Youhong, Li, Jimmy, Lee, Gobert and Tang, Benzhong. (2017). On the feasibility of a smartphone-based solution to rapid quantitative urinalysis using nanomaterial bioprobes. MobiQuitous 2017: The 14th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services. United States of America: Association for Computing Machinery (ACM). pp. 523 - 524 https://doi.org/10.1145/3144457.3144508
Are you a human or a humanoid: Predictive user modelling through behavioural analysis of online gameplay data
Gao, Chen, Jin, Kaiqi, Shen, Haifeng and Babar, Muhammed Ali. (2017). Are you a human or a humanoid: Predictive user modelling through behavioural analysis of online gameplay data. Advanced Engineering Informatics. 33, pp. 410 - 424. https://doi.org/10.1016/j.aei.2017.01.004
Automatic clustering and summarisation of microblogs: A multi-subtopic phrase reinforcement algorithm
Alghamdi, Mahfouth and Shen, Haifeng. (2017). Automatic clustering and summarisation of microblogs: A multi-subtopic phrase reinforcement algorithm. In M. Wagner, X. Li and T. Hendtlass (Ed.). Third Australasian Conference, ACALCI 2017, Geelong, VIC, Australia, January 31 – February 2, 2017, Proceedings. United States of America: Springer International Publishing. pp. 86 - 98 https://doi.org/10.1007/978-3-319-51691-2_8
iLSE: An intelligent web-based system for log structuring and extraction
Serasinghe, Sahan, Shen, Haifeng and Chen, David. (2017). iLSE: An intelligent web-based system for log structuring and extraction. In J. Lv, H. Zhang and M. Hinchey and X. Liu (Ed.). 24th Asia-Pacific Software Engineering Conference: APSEC 2017: 4-8 December 2017, Nanjing, Jiangsu, China. United States of America: IEEE Computer Society. pp. 588 - 593 https://doi.org/10.1109/APSEC.2017.70
A cost model for long-term compressed data retention
Liao, Kewen, Moffat, Alistair, Petri, Matthias and Wirth, Anthony. (2017). A cost model for long-term compressed data retention. 10th International Conference on Web Search and Data Mining. United States of America: Association for Computing Machinery. pp. 241 - 249 https://doi.org/10.1145/3018661.3018738
Edge labeling schemes for graph data
Goonetilleke, Oshini, Koutra, Danai, Sellis, Timos and Liao, Kewen. (2017). Edge labeling schemes for graph data. 29th International Conference on Scientific and Statistical Database Management. United States of America: Association for Computing Machinery. pp. 1 - 12 https://doi.org/10.1145/3085504.3085516
Web of credit: Adaptive personalized trust network inference from online rating data
Mao, Yuqing and Shen, Haifeng. (2016). Web of credit: Adaptive personalized trust network inference from online rating data. IEEE Transactions on Computational Social Systems. 3(4), pp. 176 - 189. https://doi.org/10.1109/TCSS.2016.2639016
Effective construction of relative Lempel-Ziv dictionaries
Liao, Kewen, Petri, Matthias, Moffat, Alistair and Wirth, Anthony. (2016). Effective construction of relative Lempel-Ziv dictionaries. WWW '16: 25th International World Wide Web Conference. Switzerland: Association for Computing Machinery. pp. 807 - 816 https://doi.org/10.1145/2872427.2883042
Sentiment analysis and visualisation in a backchannel system
Jiranantanagorn, Peerumporn and Shen, Haifeng. (2016). Sentiment analysis and visualisation in a backchannel system. OzCHI '16: The 28th Australian Conference on Computer-Human Interaction, Launceston, Tasmania, Australia - November 29 - December 02, 2016. United States of America: Association for Computing Machinery (ACM). pp. 353 - 357 https://doi.org/10.1145/3010915.3010992
Concealing jitter in multi-player online games through predictive behaviour modeling
Gao, Chen, Shen, Haifeng and Babar, Muhammed Ali. (2016). Concealing jitter in multi-player online games through predictive behaviour modeling. In W. Shen, X. Liu and C. Yang, J.-P. Barthès, J. Luo, L. Chen and J. Yong (Ed.). The 2016 IEEE 20th international conference on computer supported cooperative work in design (CSCWD), May 4-6, 2016, Nanching, China. United States of America: IEEE Computer Society. pp. 62 - 67 https://doi.org/10.1109/CSCWD.2016.7565964
Cloud for e-Learning: Determinants of its adoption by university students in a developing country
Almazroi, Abdulwahab Ali, Shen, Haifeng, Teoh, Kung-Keat and Babar, Muhammed Ali. (2016). Cloud for e-Learning: Determinants of its adoption by university students in a developing country. In J. Guo, H. Cai and X. Fei, K.-M. Chao and J.-Y. Chung (Ed.). The thirteenth IEEE international conference on e-business engineering, 4-6 November 2016, Macau, China. United States of America: IEEE Computer Society. pp. 71 - 78 https://doi.org/10.1109/ICEBE.2016.022
NSSSD: A new semantic hierarchical storage for sensor data
Gheisari, Mehdi, Movassagh, Ali Akbar, Qin, Yongrui, Yong, Jianming, Tao, Xiaohui, Zhang, Ji and Shen, Haifeng. (2016). NSSSD: A new semantic hierarchical storage for sensor data. The 2016 IEEE 20th international conference on computer supported cooperative work in design (CSCWD), May 4-6, 2016, Nanching, China. United States of America: IEEE Computer Society. pp. 174 - 179 https://doi.org/10.1109/CSCWD.2016.7565984
Partial selection of agile software requirements
Mougouei, Davoud, Shen, Haifeng and Babar, Muhammad Ali. (2015). Partial selection of agile software requirements. International Journal of Software Engineering and Its Applications. 9(1), pp. 113-126. https://doi.org/10.14257/ijseia.2015.9.1.10
Improved approximation algorithms for constrained fault-tolerant resource allocation
Liao, Kewen, Shen, Hong and Guo, Longkun. (2015). Improved approximation algorithms for constrained fault-tolerant resource allocation. Theoretical Computer Science. 590, pp. 118-128. https://doi.org/10.1016/j.tcs.2015.02.029
Improved approximation algorithms for computing k disjoint paths subject to two constraints
Guo, Longkun, Shen, Hong and Liao, Kewen. (2015). Improved approximation algorithms for computing k disjoint paths subject to two constraints. Journal of Combinatorial Optimization. 29(1), pp. 153-164. https://doi.org/10.1007/s10878-013-9693-x
Brief announcement : Efficient approximation algorithms for computing k disjoint restricted shortest paths
Guo, Longkun, Liao, Kewen, Shen, Hong and Li, Peng. (2015). Brief announcement : Efficient approximation algorithms for computing k disjoint restricted shortest paths. 27th ACM Symposium on Parallelism in Algorithms and Architectures. Portland, Oregon, United States of America 13 - 15 Jun 2015 Association for Computing Machinery. pp. 62–64 https://doi.org/10.1145/2755573.2755608
A lightweight solution to version incompatibility in service-oriented revision control systems
Almalki, Jameel and Shen, Haifeng. (2015). A lightweight solution to version incompatibility in service-oriented revision control systems. ASWEC 2015: 24th Australasian Software Engineering Conference, Adelaide, SA, Australia. United States of America: Association for Computing Machinery (ACM). pp. 59 - 63 https://doi.org/10.1145/2811681.2811693
An efficient and reliable geographic routing protocol based on partial network coding for underwater sensor networks
Hao, Kun, Jin, Zhigang, Shen, Haifeng and Wang, Ying. (2015). An efficient and reliable geographic routing protocol based on partial network coding for underwater sensor networks. Sensors. 15(6), pp. 12720 - 12735. https://doi.org/10.3390/s150612720
Designing a mobile digital backchannel system for monitoring sentiments and emotions in large lectures
Jiranantanagorn, Peerumporn, Bhardwaj, Parveen, Li, Ruilun, Shen, Haifeng, Goodwin, Robert and Teoh, Kung-Keat. (2015). Designing a mobile digital backchannel system for monitoring sentiments and emotions in large lectures. In F.-Ch. Kuo, S. Marshall and H. Shen, M. Stumptner and M. Ali Babar (Ed.). ASWEC 2015: 24th Australasian Software Engineering Conference, Adelaide, SA, Australia. United States of America: Association for Computing Machinery (ACM). pp. 141 - 144 https://doi.org/10.1145/2811681.2824994
Extending attention span of ADHD Children through an eye tracker directed adaptive user interface
Asiry, Othman, Shen, Haifeng and Calder, Paul. (2015). Extending attention span of ADHD Children through an eye tracker directed adaptive user interface. In F.-Ch. Kuo, S. Marshall and H. Shen, M. Stumptner and M. Ali Babar (Ed.). ASWEC 2015: 24th Australasian Software Engineering Conference, Adelaide, SA, Australia. United States of America: Association for Computing Machinery (ACM). pp. 149 - 152 https://doi.org/10.1145/2811681.2824997
Sustaining cognitive diversity in collaborative learning through shared spatially separated virtual workspaces on mobile devices
Reilly, Mark, Shen, Haifeng, Calder, Paul and Duh, Henry. (2015). Sustaining cognitive diversity in collaborative learning through shared spatially separated virtual workspaces on mobile devices. In In Wyeld, T., Calder, P. and Shen, H. (Ed.). Computer-human interaction: Cognitive effects of spatial interaction, learning, and ability pp. 171 - 193 Springer International Publishing. https://doi.org/10.1007/978-3-319-16940-8_9
LP-based approximation algorithms for reliable resource allocation
Liao, Kewen and Shen, Hong. (2014). LP-based approximation algorithms for reliable resource allocation. The Computer Journal. 57(1), pp. 154-164. https://doi.org/10.1093/comjnl/bxs164
On the shallow-light Steiner tree problem
Guo, Longkun, Liao, Kewen and Shen, Hong. (2014). On the shallow-light Steiner tree problem. 15th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT 2014). Hong Kong, China 09 - 14 Dec 2014 IEEE Computer Society. pp. 56-60 https://doi.org/10.1109/PDCAT.2014.17
Effect of off-zenith observations on reducing the impact of precipitation on ground-based microwave radiometer measurement accuracy
Xu, Guirong, Stick Ware, Randolph, Zhang, Wengang, Feng, Guangliu, Liao, Kewen and Liu, Yibing. (2014). Effect of off-zenith observations on reducing the impact of precipitation on ground-based microwave radiometer measurement accuracy. Atmospheric Research. 140-141, pp. 85 - 94. https://doi.org/10.1016/j.atmosres.2014.01.021
Towards a collaborative classroom through shared workspaces on mobile devices
Reilly, Mark, Shen, Haifeng, Calder, Paul and Duh, Henry. (2014). Towards a collaborative classroom through shared workspaces on mobile devices. BCS-HCI '14: The 28th International BCS Human Computer Interaction Conference on HCI 2014 - Sand, Sea and Sky - Holiday HCI. United Kingdom: Electronic Workshops in Computing. pp. 335 - 340
Stimulating high quality social media through knowledge barter-auctioning
Ji, Qijin, Shen, Haifeng, Mao, Yuqing and Zhu, Yanqing. (2014). Stimulating high quality social media through knowledge barter-auctioning. SocialCom '14: The 2014 International Conference on Social Computing, August 04-07 2014, Beijing, China. United States of America: Association for Computing Machinery (ACM). pp. 4 - 11 https://doi.org/10.1145/2639968.2640068
SORC: Service-oriented distributed revision control for collaborative web programming
Bin Sarib, Ahmad Sholehin and Shen, Haifeng. (2014). SORC: Service-oriented distributed revision control for collaborative web programming. 2014 IEEE 18th International Conference on Computer Supported Cooperative Work in Design (CSCWD 2014), Hsinchu, Taiwan, 21-23 May 2014. United States of America: IEEE Computer Society. pp. 638 - 643 https://doi.org/10.1109/CSCWD.2014.6846919