To best understand this article in the context of the visual breed identification literature, please see National Canine Research Council's complete analysis here.
Article Citation:
Olson, K. R., Levy, J. K., Norby, B., Crandall, M. M., Broadhurst, J. E., Jacks, S., Barton, R. C., & Zimmerman, M. S. (2015). Inconsistent identification of pit bull-type dogs by shelter staff. The Veterinary Journal, 206, 197-202.


National Canine Research Council Summary and Analysis:

This study explored two aspects of visual breed identification: consistency among experts and validity when compared to DNA analysis. The researchers specifically focused on visual identification of “pit bull-type dogs,” as they are most commonly targeted in breed-specific legislation (BSL). Because the term “pit bull” does not denote a specific breed and there is no agreed-upon definition of “pit bull-type dogs,” for this study the authors accepted a wide range of identifiers on the part of the participants as designated “pit bull” labeling.
A major strength of this study is the sample itself; shelter managers specifically selected the participants due to their routine visual breed-assignment duties, staff who visually identify dogs’ breeds when they come into a shelter and may be called upon to testify to a dog’s breed in BSL disputes. Hence, the decisions made by this survey group regularly have real-world consequences. 
Participants included sixteen raters (three staff members and one veterinarian from four different Florida shelters) who visually identified a total of 120 dogs. Thirty dogs with a wide range of morphology were selected at each shelter. The dogs’ breed labels at intake were noted and later compared to the raters’ responses, but were not visible to the raters during the experimental process. However, individual raters may or may not have had previous knowledge of the dogs in the study, so the identifications were not truly “blind.” 
Raters examined the dogs in their kennels and assigned a primary breed and a secondary breed if they chose. If the rater could not see enough of what he or she considered “defining characteristics” for a primary breed identification, they could indicate “mixed breed.” If a rater identified the dog as American Staffordshire terrier, Staffordshire bull terrier,  “pit bull,” “American Pit Bull Terrier,” or “mix” with any of these labels, they were counted as having identified the dog as a “pit bull-type.” This was considered a DNA match if the test results showed 12.5% or more American Staffordshire terrier or Staffordshire bull terrier. Thus the criteria for a match were extremely broad, including multiple breeds and very low DNA material percentage.
Blood samples from each dog were sent to the Wisdom Panel Professional Canine Genetic Analysis, MARS Veterinary for DNA identification. DNA analysis revealed that 21% of the sample dogs had “pit bull-type” heritage, though none were purebred and none contained more than 50% of the “pit bull-type” breeds, as defined by the study. Median agreement between visual identification and DNA analysis for discriminating between “pit bull-type dogs” and “non pit bull-type dogs” ranged from 67-78%. Twenty of the twenty-five dogs identified as “pit bull-type” by DNA analysis were so identified by at least one shelter staff, while only 12 had been identified as such at the time of intake. The average sensitivity for identifying “pit bull-type dogs” was 50%. Of the remaining 95 dogs who fell into the “not pit bull-type” group by DNA analysis, 38% were nevertheless given the label by at least one breed assessor. That is a significant finding; of the dogs whose DNA did not reveal contributions from Staffordshire bull terrier, or American Staffordshire terrier,  more than 1 out of 3 were inappropriately labeled “pit bull” by at least one staff member.
In sum, the overall validity (specificity and sensitivity results) even with such a broad target for identification, did not reach the level of a coin toss.
Like the studies that preceded it, the current data underscore the inconsistency and unreliability of visual breed identification and thus undermines any attempts to correlate breed with behavior on the part of scientists, media outlets, veterinarians, shelter staff, and policy makers.
Link to Full Text of the Original Article: