Estimation and Use of Interrater Agreement and Interrater Reliability Indices in Organizational Research & Practice

James LeBreton, Ph.D.
Pennsylvania State University

• Food included on-site.
• NEW: Email questions for the presenter to vp.ptcmw@gmail.com by 12pm ET the day before.
  

• Crystal City Shops, 2010 Crystal Dr, Arlington, VA 22202 (map)
  
• Metro Access and Parking Available

Abstract: The use of interrater reliability (IRR) and interrater agreement (IRA) indices has increased dramatically during the last 25 years.  This popularity is likely due to the increased role multilevel modeling techniques (e.g., random coefficient regression modeling; multilevel structural equation modeling) play in organizational research and practice.  Specifically, IRR and IRA indices are often used to justify combining lower-level data (e.g., individual climate perceptions) to create a higher-level aggregate variable (e.g., shared psychological climate).  In addition, estimates of IRR (and to a lesser degree, IRA) have a long (and somewhat controversial) history as tools used to evaluate the psychometric quality of criteria used as part of the test validation process (e.g., supervisor ratings of job performance).  The purpose of this workshop is to expose researchers to various issues surrounding commonly used estimates of IRR and IRA.

My workshop has three main objectives:

• To synthesize and integrate various definitional issues concerning the concepts of IRR and IRA. 
• To provide answers to some of the more commonly asked questions associated with these statistics (e.g., what are reasonable cut-offs for IRA, how many raters are needed to estimate IRR). 
• To demonstrate the principles discussed via a hands-on examples. 

Speaker Bio: James M. LeBreton is a Professor of Psychology at Pennsylvania State University. Previously, he was on the faculty at Purdue University and Wayne State University.  He earned his Ph.D. in industrial-organizational psychology from The University of Tennessee.  During the last 15 years he has been involved in the development and validation of several measures designed to assess implicit personality.  His current research involves the measurement of aggressive personality characteristics and how these characteristics can be used to enhance our understanding of important behavior in organizations (e.g., counterproductive work behavior, leadership, team performance).  James also conducts research involving the development and application of new statistics and research methods.  His current projects focus on topics such as 1) assessing the relative importance of predictors in regression models, 2) assessing interrater agreement and reliability, 3) enhancing tests for mediation and moderation, and 4) analyzing longitudinal and multilevel data.  In 2009, James was awarded the Early Career Award from the Academy of Management’s Research Methods Division and the Center for the Advancement of Research Methods and Analysis (CARMA).   Beginning in 2014, James assumed the editorship at Organizational Research Methods where he had been serving as an Associate Editor.  James has also served on the editorial boards at the Journal of Applied Psychology, Journal of Management, and Journal of Business and Psychology.