Reliability between online raters with varying familiarities of a region: microscale audit of pedestrian streetscapes (MAPS)

Background To test inter-rater reliability of the online Microscale Audit of Pedestrian Streetscapes (MAPS) tool between raters with varying familiarities of Phoenix, Arizona. Methods The online MAPS tool, based on the MAPS in-field audit tool and scoring system, was used for audits. Sixty route pairs, 141 segment pairs, and 92 crossing pairs in Phoenix were included. Each route, segment or crossing was audited by two independent raters: one rater in Phoenix and the other in San Diego, California, respectively. Item, subscale scores, and total scores reliability analyses were computed using Kappa or intra-class correlation coefficient (ICC). Results The route overall score had substantial reliability (ICC: 0.832). Of the route subscale and overall scores, sixteen out of twenty had moderate to substantial reliability (ICC: 0.616–0.906), and the four subscales had fair reliability (ICC: 0.409–0.563). Sixteen out of twenty scores in segment and crossing sections demonstrated fair to substantial reliability (ICC: 0.448–0.897), and the remaining four had slight reliability (ICC: 0.348–0.364). Conclusions Most of the online MAPS items, subscales, and overall scores demonstrated fair to substantial reliability between raters with varied familiarities of the Phoenix area. Results support use of online MAPS to measure microscale elements of the built environment by raters unfamiliar with a region.