TrinityRCL : Multi-granular and code-level root cause localization using multiple types of telemetry data in microservice systems

Journal article


Gu, Shenghui, Rong, Guoping, Ren, Tian, Zhang, He, Shen, Haifeng, Yu, Yongda, Li, Xian, Ouyang, Jian and Chen, Chunan. (2023). TrinityRCL : Multi-granular and code-level root cause localization using multiple types of telemetry data in microservice systems. IEEE Transactions on Software Engineering. 49(5), pp. 3071-3088. https://doi.org/10.1109/TSE.2023.3241299
AuthorsGu, Shenghui, Rong, Guoping, Ren, Tian, Zhang, He, Shen, Haifeng, Yu, Yongda, Li, Xian, Ouyang, Jian and Chen, Chunan
Abstract

The microservice architecture has been commonly adopted by large scale software systems exemplified by a wide range of online services. Service monitoring through anomaly detection and root cause analysis (RCA) is crucial for these microservice systems to provide stable and continued services. However, compared with monolithic systems, software systems based on the layered microservice architecture are inherently complex and commonly involve entities at different levels of granularity. Therefore, for effective service monitoring, these systems have a special requirement of multi-granular RCA. Furthermore, as a large proportion of anomalies in microservice systems pertain to problematic code, to timely troubleshoot these anomalies, these systems have another special requirement of RCA at the finest code-level. Microservice systems rely on telemetry data to perform service monitoring and RCA of service anomalies. The majority of existing RCA approaches are only based on a single type of telemetry data and as a result can only support uni-granular RCA at either application-level or service-level. Although there are attempts to combine metric and tracing data in RCA, their objective is to improve RCA's efficiency or accuracy rather than to support multi-granular RCA. In this article, we propose a new RCA solution TrinityRCL that is able to localize the root causes of anomalies at multiple levels of granularity including application-level, service-level, host-level, and metric-level, with the unique capability of code-level localization by harnessing all three types of telemetry data to construct a causal graph representing the intricate, dynamic, and nondeterministic relationships among the various entities related to the anomalies. By implementing and deploying TrinityRCL in a real production environment, we evaluate TrinityRCL against two baseline methods and the results show that TrinityRCL has a significant performance advantage in terms of accuracy at the same level of granularity with comparable efficiency and is particularly effective to support large-scale systems with massive telemetry data.

Keywordscomputer network security; graph theory; service oriented architecture; telemetry; application level; code level localization; code level root cause localization; continued services; effective service monitoring; finest code level; host level; large scale software systems; large scale systems; layered microservice architecture; massive telemetry data; metric level; microservice systems; monolithic systems; multigranular rca; online services; rca solution trinity rc lthat; service anomalies; service level; stable services; unigranular rca; microservice architectures; telemetry; measurement; codes; monitoring; location awareness; computer architecture; root cause; telemetry data; microservices
Year2023
JournalIEEE Transactions on Software Engineering
Journal citation49 (5), pp. 3071-3088
PublisherIEEE Computer Society
ISSN0098-5589
Digital Object Identifier (DOI)https://doi.org/10.1109/TSE.2023.3241299
Scopus EID2-s2.0-85148448378
Page range3071-3088
FunderNational Key Research and Development Program of China
Research Council of Norway
Meituan
Key Research and Development Program of Jiangsu Province
National Natural Science Foundation of China (NSFC)
Publisher's version
License
All rights reserved
File Access Level
Controlled
Output statusPublished
Publication dates
Online01 Feb 2023
Publication process dates
Accepted18 Jan 2023
Deposited07 Nov 2023
Grant ID2019YFE0105500
309494
BE2021002-2
62072227
62202219
Permalink -

https://acuresearchbank.acu.edu.au/item/8zy56/trinityrcl-multi-granular-and-code-level-root-cause-localization-using-multiple-types-of-telemetry-data-in-microservice-systems

Restricted files

Publisher's version

  • 54
    total views
  • 0
    total downloads
  • 1
    views this month
  • 0
    downloads this month
These values are for the period from 19th October 2020, when this repository was created.

Export as

Related outputs

Robustness Verification Method for Artificial Intelligence Systems Based on Source Code Processing
Yang, Yan-Jing, Mao, Run-Feng, Tan, Rui, Shen, Haifeng and Rong, Guo-Ping. (2023). Robustness Verification Method for Artificial Intelligence Systems Based on Source Code Processing. Ruanjian Xuebao. 2023(34), pp. 4018-4036. https://doi.org/10.13328/j.cnki.jos.006879
The Why, When, What and How about Predictive Continuous Integration : A Simulation-Based Investigation
Liu, Bohan, Zhang, He, Ma, Weigang, Li, Gongyuan, Li, Shanshan and Shen, Haifeng. (2023). The Why, When, What and How about Predictive Continuous Integration : A Simulation-Based Investigation. IEEE Transactions on Software Engineering. 49(12), pp. 5223-5249. https://doi.org/10.1109/TSE.2023.3330510
Revisit security in the era of DevOps : An evidence-based inquiry into DevSecOps industry
Zhou, Xin, Mao, Runfeng, Zhang, He, Dai, Qiming, Huang, Huang, Shen, Haifeng, Li, Jingyue and Rong, Guoping. (2023). Revisit security in the era of DevOps : An evidence-based inquiry into DevSecOps industry. IET Software. 17(4), pp. 435-454. https://doi.org/10.1049/sfw2.12132
How Do Developers’ Profiles and Experiences Influence their Logging Practices? An Empirical Study of Industrial Practitioners.
Rong, Guoping, Gu, Shenghui, Shen, Haifeng, Zhang, He and Kuang, Hongyu. (2023). How Do Developers’ Profiles and Experiences Influence their Logging Practices? An Empirical Study of Industrial Practitioners. IEEE/ACM 45th International Conference on Software Engineering (ICSE). Melbourne, Australia 14 - 20 May 2023 United States: IEEE Computer Society. pp. 855-867 https://doi.org/10.1109/ICSE48619.2023.00080
Evaluating the efficacy of using a novel gaze-based attentive user interface to extend ADHD children's attention span
Shen, Haifeng, Asiry, Othman, Babar, M. Ali and Bednarz, Tomasz. (2023). Evaluating the efficacy of using a novel gaze-based attentive user interface to extend ADHD children's attention span. International Journal of Human-Computer Studies. 169, p. Article 102927. https://doi.org/10.1016/j.ijhcs.2022.102927
Fed-SC : One-shot federated subspace clustering over high-dimensional data
Xie, Songjie, Wu, Youlong, Liao, Kewen, Chen, Lu, Liu, Chengfei, Shen, Haifeng, Tang, MingJian and Sun, Lu. (2023). Fed-SC : One-shot federated subspace clustering over high-dimensional data. 2023 IEEE 39th International Conference on Data Engineering (ICDE). Anaheim, California, United States of America 03 - 07 Apr 2023 Institute of Electrical and Electronics Engineers Inc.. pp. 2905-2918 https://doi.org/10.1109/ICDE55515.2023.00222
Hamstring strain injury risk factors in Australian Football change over the course of the season
Sim, Aylwin, Timmins, Ryan G., Ruddy, Joshua D., Shen, Haifeng, Liao, Kewen, Maniar, Nirav, Hickey, Jack T., Williams, Morgan D. and Opar, David A.. (2023). Hamstring strain injury risk factors in Australian Football change over the course of the season. Medicine and Science in Sports and Exercise. https://doi.org/10.1249/MSS.0000000000003297
Logging practices in software engineering : A systematic mapping study
Gu, Shenghui, Rong, Guoping, Zhang, He and Shen, Haifeng. (2023). Logging practices in software engineering : A systematic mapping study. IEEE Transactions on Software Engineering. 49(2), pp. 902-923. https://doi.org/10.1109/TSE.2022.3166924
CNN attention guidance for improved orthopedics radiographic fracture classification
Liao, Zhibin, Liao, Kewen, Shen, Haifeng, van Boxel, Marouska F, Prijs, Jasper, Jaarsma, Ruurd L., Doornberg, Job N., van den Hengel, Anton and Verjans, Johan W.. (2022). CNN attention guidance for improved orthopedics radiographic fracture classification. IEEE Journal of Biomedical and Health Informatics. 26(7), pp. 3139-3150. https://doi.org/10.1109/JBHI.2022.3152267
Challenges and solutions when adopting DevSecOps : A systematic review
Rajapakse, Roshan N., Zahedi, Mansooreh, Babar, M. Ali and Shen, Haifeng. (2022). Challenges and solutions when adopting DevSecOps : A systematic review. Information and Software Technology. 141(106700), p. Article 106700. https://doi.org/10.1016/j.infsof.2021.106700
Human-AI interactive and continuous sensemaking : A case study of image classification using scribble attention maps
Shen, Haifeng, Liao, Kewen, Liao, Zhibin, Doornberg, Job, Qiao, Maoying, van den Hengel, Anton and Verjans, Johan W.. (2021). Human-AI interactive and continuous sensemaking : A case study of image classification using scribble attention maps. CHI Conference on Human Factors in Computing Systems. Virtual 08 - 13 May 2021 pp. 1-8 https://doi.org/10.1145/3411763.3451798
Quality assessment in systematic literature reviews : A software engineering perspective
Yang, Lanxin, Zhang, He, Shen, Haifeng, Huang, Xin, Zhou, Xin, Rong, Guoping and Shao, Dong. (2021). Quality assessment in systematic literature reviews : A software engineering perspective. Information and Software Technology. 130, p. Article 106397. https://doi.org/10.1016/j.infsof.2020.106397
Processes, challenges and recommendations of Gray Literature Review : An experience report
Zhang, He, Mao, Runfeng, Huang, Huang, Dai, Qiming, Zhou, Xin, Shen, Haifeng and Rong, Guoping. (2021). Processes, challenges and recommendations of Gray Literature Review : An experience report. Information and Software Technology. 137, p. Article 106607. https://doi.org/10.1016/j.infsof.2021.106607
Preliminary Findings about DevSecOps from Grey Literature
Zhang, He, Mao, Runfeng, Dai, Qiming, Huang, Huang, Rong, Guoping, Shen, Haifeng, Chen, Lianping and Kaixiang Lu, Kaixiang. (2020). Preliminary Findings about DevSecOps from Grey Literature. 2020 IEEE 20th International Conference on Software Quality, Reliability and Security (QRS). Macau, China 11 - 14 Dec 2020 United States: IEEE Computer Society. pp. 450-457
Inferring location types with geo-social-temporal pattern mining
Anwar, Tarique, Liao, Kewen, Goyal, Angelic, Sellis, Timos, Kayes, A. S. M. and Shen, Haifeng. (2020). Inferring location types with geo-social-temporal pattern mining. IEEE Access. 8, pp. 154789-154799. https://doi.org/10.1109/ACCESS.2020.3018997
An experimental evaluation of imbalanced learning and time-series validation in the context of CI/CD prediction
Liu, Bohan, Zhang, He, Yang, Lanxin, Dong, Liming, Shen, Haifeng and Song, Kaiwen. (2020). An experimental evaluation of imbalanced learning and time-series validation in the context of CI/CD prediction. EASE 2020, April 15-17, 2020, Trondheim, Norway. Norway: Association for Computing Machinery. pp. 21 - 30 https://doi.org/10.1145/3383219.3383222
The impact of trust on the adoption of cloud computing services by university students
Almazroi, Abdulwahab Ali, Shen, Haifeng and Mohammed, Fathey. (2019). The impact of trust on the adoption of cloud computing services by university students. 3rd International Conference of Reliable Information and Communication Technology 2018. Kuala Lumpur, Malaysia 23 - 24 Jul 2018 Malaysia: Springer Verlag. pp. 902-911 https://doi.org/10.1007/978-3-319-99007-1_84
An empirical study of the effectiveness of software architecture evaluation meetings
Babar, M. Ali, Shen, Haifeng, Biffl, Stefan and Winkler, Dietmar. (2019). An empirical study of the effectiveness of software architecture evaluation meetings. IEEE Access. 7, pp. 79069-79084. https://doi.org/10.1109/ACCESS.2019.2922265
An adaptive differential evolution algorithm to optimal multi-level thresholding for MRI brain image segmentation
Tarkhaneh, Omid and Shen, Haifeng. (2019). An adaptive differential evolution algorithm to optimal multi-level thresholding for MRI brain image segmentation. Expert Systems with Applications. 138, pp. 1 - 18. https://doi.org/10.1016/j.eswa.2019.07.037
Information visualisation methods and techniques: State-of-the-art and future directions
Shen, Haifeng, Bednarz, Tomasz, Nguyen, Huyen, Feng, Frank, Wyeld, Theodor, Hoek, Peter J. and Lo, Edward H.S.. (2019). Information visualisation methods and techniques: State-of-the-art and future directions. Journal of Industrial Information Integration. 16(100102), pp. 1 - 17. https://doi.org/10.1016/j.jii.2019.07.003
Training of feedforward neural networks for data classification using hybrid particle swarm optimization, mantegna levy flight and neighborhood search
Tarkhaneh, Omid and Shen, Haifeng. (2019). Training of feedforward neural networks for data classification using hybrid particle swarm optimization, mantegna levy flight and neighborhood search. Heliyon. 5(4), pp. 1 - 32. https://doi.org/10.1016/j.heliyon.2019.e01275
SORCER: A decentralised continuous integration platform for service-oriented software systems
Almalki, Jameel and Shen, Haifeng. (2019). SORCER: A decentralised continuous integration platform for service-oriented software systems. IEEE International Conference on Service-Oriented Computing Applications. United States of America: Springer International Publishing. pp. 458 - 464 https://doi.org/10.1007/978-3-030-17642-6_44
Developing cross-organisational service-based software systems through decentralised interface-oriented continuous integration
Almalki, Jameel and Shen, Haifeng. (2018). Developing cross-organisational service-based software systems through decentralised interface-oriented continuous integration. Australian Software Engineering Conference. United States of America: IEEE Computer Society. pp. 191 - 200 https://doi.org/10.1109/ASWEC.2018.00033
The interplay of factors affecting learning of introductory programming: A comparative study of an Australian and an Indian University
Sharma, Ritu and Shen, Haifeng. (2018). The interplay of factors affecting learning of introductory programming: A comparative study of an Australian and an Indian University. IEEE International Conference on Computer Science and Education. United States of America: IEEE Computer Society. pp. 669 - 674 https://doi.org/10.1109/ICCSE.2018.8468768
Integrating localization and energy-awareness: A novel geographic routing protocol for underwater wireless sensor networks
Hao, Kun, Shen, Haifeng, Liu, Yonglei, Wang, Beibei and Du, Xiujuan. (2018). Integrating localization and energy-awareness: A novel geographic routing protocol for underwater wireless sensor networks. Mobile Networks and Applications. 23(5), pp. 1427 - 1435. https://doi.org/10.1007/s11036-018-1093-0
A smartphone-based point-of-care quantitative urinalysis device for chronic kidney disease patients
Akraa, Shaymaa, Tam, Anh Pham Tran, Shen, Haifeng, Tang, Youhong, Tang, Ben Zhong, Li, Jimmy and Walker, Sandy. (2018). A smartphone-based point-of-care quantitative urinalysis device for chronic kidney disease patients. Journal of Network and Computer Applications. 115, pp. 59 - 69. https://doi.org/10.1016/j.jnca.2018.04.012
Extending attention span for children with ADHD using an attentive visual interface
Asiry, Othman, Shen, Haifeng, Balkhy, Soher and Wyeld, Theodor. (2018). Extending attention span for children with ADHD using an attentive visual interface. International Conference Information Visualisation. United States of America: IEEE Computer Society. pp. 188 - 193 https://doi.org/10.1109/iV.2018.00041
On the feasibility of a smartphone-based solution to rapid quantitative urinalysis using nanomaterial bioprobes
Akraa, Shaymaa, Guo, Feng, Shen, Haifeng, Tang, Youhong, Li, Jimmy, Lee, Gobert and Tang, Benzhong. (2017). On the feasibility of a smartphone-based solution to rapid quantitative urinalysis using nanomaterial bioprobes. MobiQuitous 2017: The 14th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services. United States of America: Association for Computing Machinery (ACM). pp. 523 - 524 https://doi.org/10.1145/3144457.3144508
Are you a human or a humanoid: Predictive user modelling through behavioural analysis of online gameplay data
Gao, Chen, Jin, Kaiqi, Shen, Haifeng and Babar, Muhammed Ali. (2017). Are you a human or a humanoid: Predictive user modelling through behavioural analysis of online gameplay data. Advanced Engineering Informatics. 33, pp. 410 - 424. https://doi.org/10.1016/j.aei.2017.01.004
Automatic clustering and summarisation of microblogs: A multi-subtopic phrase reinforcement algorithm
Alghamdi, Mahfouth and Shen, Haifeng. (2017). Automatic clustering and summarisation of microblogs: A multi-subtopic phrase reinforcement algorithm. In M. Wagner, X. Li and T. Hendtlass (Ed.). Third Australasian Conference, ACALCI 2017, Geelong, VIC, Australia, January 31 – February 2, 2017, Proceedings. United States of America: Springer International Publishing. pp. 86 - 98 https://doi.org/10.1007/978-3-319-51691-2_8
iLSE: An intelligent web-based system for log structuring and extraction
Serasinghe, Sahan, Shen, Haifeng and Chen, David. (2017). iLSE: An intelligent web-based system for log structuring and extraction. In J. Lv, H. Zhang and M. Hinchey and X. Liu (Ed.). 24th Asia-Pacific Software Engineering Conference: APSEC 2017: 4-8 December 2017, Nanjing, Jiangsu, China. United States of America: IEEE Computer Society. pp. 588 - 593 https://doi.org/10.1109/APSEC.2017.70
Voluntary participation in discussion forums as an engagement indicator : An empirical study of teaching first-year programming
Sharma, Ritu, Shen, Haifeng and Goodwin, Robert. (2016). Voluntary participation in discussion forums as an engagement indicator : An empirical study of teaching first-year programming. OzCHI '16: The 28th Australian Conference on Computer-Human Interaction, Launceston, Tasmania, Australia - November 29 - December 02, 2016. Association for Computing Machinery (ACM). pp. 489-493 https://doi.org/10.1145/3010915.3010967
Web of credit: Adaptive personalized trust network inference from online rating data
Mao, Yuqing and Shen, Haifeng. (2016). Web of credit: Adaptive personalized trust network inference from online rating data. IEEE Transactions on Computational Social Systems. 3(4), pp. 176 - 189. https://doi.org/10.1109/TCSS.2016.2639016
Sentiment analysis and visualisation in a backchannel system
Jiranantanagorn, Peerumporn and Shen, Haifeng. (2016). Sentiment analysis and visualisation in a backchannel system. OzCHI '16: The 28th Australian Conference on Computer-Human Interaction, Launceston, Tasmania, Australia - November 29 - December 02, 2016. United States of America: Association for Computing Machinery (ACM). pp. 353 - 357 https://doi.org/10.1145/3010915.3010992
Concealing jitter in multi-player online games through predictive behaviour modeling
Gao, Chen, Shen, Haifeng and Babar, Muhammed Ali. (2016). Concealing jitter in multi-player online games through predictive behaviour modeling. In W. Shen, X. Liu and C. Yang, J.-P. Barthès, J. Luo, L. Chen and J. Yong (Ed.). The 2016 IEEE 20th international conference on computer supported cooperative work in design (CSCWD), May 4-6, 2016, Nanching, China. United States of America: IEEE Computer Society. pp. 62 - 67 https://doi.org/10.1109/CSCWD.2016.7565964
Cloud for e-Learning: Determinants of its adoption by university students in a developing country
Almazroi, Abdulwahab Ali, Shen, Haifeng, Teoh, Kung-Keat and Babar, Muhammed Ali. (2016). Cloud for e-Learning: Determinants of its adoption by university students in a developing country. In J. Guo, H. Cai and X. Fei, K.-M. Chao and J.-Y. Chung (Ed.). The thirteenth IEEE international conference on e-business engineering, 4-6 November 2016, Macau, China. United States of America: IEEE Computer Society. pp. 71 - 78 https://doi.org/10.1109/ICEBE.2016.022
NSSSD: A new semantic hierarchical storage for sensor data
Gheisari, Mehdi, Movassagh, Ali Akbar, Qin, Yongrui, Yong, Jianming, Tao, Xiaohui, Zhang, Ji and Shen, Haifeng. (2016). NSSSD: A new semantic hierarchical storage for sensor data. The 2016 IEEE 20th international conference on computer supported cooperative work in design (CSCWD), May 4-6, 2016, Nanching, China. United States of America: IEEE Computer Society. pp. 174 - 179 https://doi.org/10.1109/CSCWD.2016.7565984
ClasSense : A mobile digital backchannel system for monitoring class morale
Jiranantanagorn, Peerumporn, Shen, Haifeng, Goodwin, Robert and Teoh, Kung-Keat. (2015). ClasSense : A mobile digital backchannel system for monitoring class morale. The International Journal of Learning. 1(2), pp. 161-167. https://doi.org/10.18178/IJLT.1.2.161-167
Partial selection of agile software requirements
Mougouei, Davoud, Shen, Haifeng and Babar, Muhammad Ali. (2015). Partial selection of agile software requirements. International Journal of Software Engineering and Its Applications. 9(1), pp. 113-126. https://doi.org/10.14257/ijseia.2015.9.1.10
A lightweight solution to version incompatibility in service-oriented revision control systems
Almalki, Jameel and Shen, Haifeng. (2015). A lightweight solution to version incompatibility in service-oriented revision control systems. ASWEC 2015: 24th Australasian Software Engineering Conference, Adelaide, SA, Australia. United States of America: Association for Computing Machinery (ACM). pp. 59 - 63 https://doi.org/10.1145/2811681.2811693
An efficient and reliable geographic routing protocol based on partial network coding for underwater sensor networks
Hao, Kun, Jin, Zhigang, Shen, Haifeng and Wang, Ying. (2015). An efficient and reliable geographic routing protocol based on partial network coding for underwater sensor networks. Sensors. 15(6), pp. 12720 - 12735. https://doi.org/10.3390/s150612720
Designing a mobile digital backchannel system for monitoring sentiments and emotions in large lectures
Jiranantanagorn, Peerumporn, Bhardwaj, Parveen, Li, Ruilun, Shen, Haifeng, Goodwin, Robert and Teoh, Kung-Keat. (2015). Designing a mobile digital backchannel system for monitoring sentiments and emotions in large lectures. In F.-Ch. Kuo, S. Marshall and H. Shen, M. Stumptner and M. Ali Babar (Ed.). ASWEC 2015: 24th Australasian Software Engineering Conference, Adelaide, SA, Australia. United States of America: Association for Computing Machinery (ACM). pp. 141 - 144 https://doi.org/10.1145/2811681.2824994
Extending attention span of ADHD Children through an eye tracker directed adaptive user interface
Asiry, Othman, Shen, Haifeng and Calder, Paul. (2015). Extending attention span of ADHD Children through an eye tracker directed adaptive user interface. In F.-Ch. Kuo, S. Marshall and H. Shen, M. Stumptner and M. Ali Babar (Ed.). ASWEC 2015: 24th Australasian Software Engineering Conference, Adelaide, SA, Australia. United States of America: Association for Computing Machinery (ACM). pp. 149 - 152 https://doi.org/10.1145/2811681.2824997
Sustaining cognitive diversity in collaborative learning through shared spatially separated virtual workspaces on mobile devices
Reilly, Mark, Shen, Haifeng, Calder, Paul and Duh, Henry. (2015). Sustaining cognitive diversity in collaborative learning through shared spatially separated virtual workspaces on mobile devices. In In Wyeld, T., Calder, P. and Shen, H. (Ed.). Computer-human interaction: Cognitive effects of spatial interaction, learning, and ability pp. 171 - 193 Springer International Publishing. https://doi.org/10.1007/978-3-319-16940-8_9
Towards a collaborative classroom through shared workspaces on mobile devices
Reilly, Mark, Shen, Haifeng, Calder, Paul and Duh, Henry. (2014). Towards a collaborative classroom through shared workspaces on mobile devices. BCS-HCI '14: The 28th International BCS Human Computer Interaction Conference on HCI 2014 - Sand, Sea and Sky - Holiday HCI. United Kingdom: Electronic Workshops in Computing. pp. 335 - 340
Stimulating high quality social media through knowledge barter-auctioning
Ji, Qijin, Shen, Haifeng, Mao, Yuqing and Zhu, Yanqing. (2014). Stimulating high quality social media through knowledge barter-auctioning. SocialCom '14: The 2014 International Conference on Social Computing, August 04-07 2014, Beijing, China. United States of America: Association for Computing Machinery (ACM). pp. 4 - 11 https://doi.org/10.1145/2639968.2640068
SORC: Service-oriented distributed revision control for collaborative web programming
Bin Sarib, Ahmad Sholehin and Shen, Haifeng. (2014). SORC: Service-oriented distributed revision control for collaborative web programming. 2014 IEEE 18th International Conference on Computer Supported Cooperative Work in Design (CSCWD 2014), Hsinchu, Taiwan, 21-23 May 2014. United States of America: IEEE Computer Society. pp. 638 - 643 https://doi.org/10.1109/CSCWD.2014.6846919