In the eye of the beholder? An examination of the inter-rater reliability of the LSI-R and YLS/CMI in a correctional agency

 

 

 

Michael Rocque, Judy Plummer-Beale

Journal of Criminal Justice

October 18, 2014

 

 

http://www.sciencedirect.com/science/article/pii/S0047235214000889

 

Highlights

•Inter-rater reliability is an often overlooked psychometric property of risk assessment instruments in the criminal justice system

•We conduct inter-rater reliability studies on the Level of Service Inventory-Revised (LSI-R) and Youth Level of Service/Case Management Inventory (YLS/CMI) in a correctional agency

•Findings indicate adequate reliability for the LSI-R and excellent reliability for the YLS/CMI


Abstract

Purpose

To examine the inter-rater reliability of two risk assessment tools: The Level of Service Inventory-Revised (LSI-R) and the Youth Level of Service/Case Management Inventory (YLS/CMI).

Methods

Two identical experiments are reported. For both studies, a random sample of 10 offenders were interviewed and videotaped with each tool (totaling 20 offenders). The tapes were then shown to a random selection of 20 raters (for a total of 40 raters) employed at a state agency. The fully-crossed design allowed each of the raters to rate the each of the cases, resulting in 200 total risk score observations for each tool. Inter-rater reliability analyses were then conducted.

Results

The LSI-R demonstrated adequate to fair reliability, with certain domains showing lower reliability. Overall, the LSI-R had an ICC of .65. The YLS/CMI demonstrated higher reliability (ICC of .78). In addition, for the LSI-R study, comparisons were made between staff raters who work in a facility versus those in the community (e.g., probation officers). For the YLS/CMI study, comparisons were made between incarcerated offenders versus probationers. Neither comparison yielded consistent differences.

Conclusions

The YLS/CMI is generally reliable. The LSI-R showed less reliability. However, each study showed certain domains with less than ideal reliability.

 

 

 

 

 

 

 

 

Human Services and Justice Coordinating Committee

www.hsjcc.on.ca/