Simplifying Assessment Development & Validity

By Kimberly R. Leveridge (Brinkmeyer), Ph.D.,

Personality assessment is useful for describing an individual on characteristics which can not be directly observed.  Behaviors are visible to people, but the reasons behind them and the motivations for them are not observable.  Psychological assessment results provide a vocabulary for describing propensities and a view of the “whys” behind the behaviors.  This information sets the stage for more effective employee and manager selection, succession planning, team building, and professional development.

Test Development

During test development, the first question to answer is “what to measure”.  Items are then written to reflect the behaviors associated with the dimension(s) being measured.  For example, extroversion is defined as:  interest in or behavior directed toward others.  Thus, these items might be written to reflect the degree to which a person enjoys (or does not enjoy) social settings, crowded events, and making presentations.  Sample items focusing on extroversion might be “in a group, I enjoy attracting attention to myself” or “I don’t care for large, noisy crowds.” 

Once a pool of test items are written—pool size is determined by the number of dimensions being assessed and the number of items deemed necessary to tap into each dimension—they are pilot tested to determine the degree to which they correlate with each other, they differentiate people (some endorse,  some do not), and they are reliable.  Reliability is determined by test-retest—subjects answer the question the same way on multiple administrations—and by internal consistency—items which are designed to measure extroversion tend to correlate, or hang-together.

Assessment Validation Step 1

The next step is to determine the validity of the measure, which is a bit more complicated to explain.  First, there is what’s known as the test-test validation process which correlates (see attachment for explanation of correlations) scores on our instruments with other instruments.  These test-to-test correlations are conducted with instruments that are hypothesized to have similar or related constructs and with instruments that are hypothesized to be unrelated.  For example, the process of validating the CDR Character Assessment included having subjects take the Character along with the ASVAB, PSI Basic Skills Test (both should be unrelated), Myers-Briggs, SDS, Interpersonal Adjective Scales, BigFive Factor Markers, and the MMPI-2 (all of which should have some relationship to the measures).  These analyses resulted in correlations that confirmed hypothesized relationships.

Assessment Validation Step 2

The next level of validation should include correlations between test scores and relevant non-test indicators—such as actual performance ratings.  This step is taken to validate (confirm or not) whether the instrument accurately measures the predicted behavior and the impact on performance.  For example, those who have a high scores on the CDR Character Assessment “Adjustment” scale and a high CDR Risk Assessment “Egotist” scale will generally have higher self-ratings on 360 performance reviews.  This translates to people who have higher opinions about their own performance in comparison with the perceptions of others.  Thus, the correlations will be higher between these scale scores and the resulting behavior ratings.  The validation process is not simple and it is important to perform statistical analyses using a variety of non-test indicators and performance results.  In addition to performance reviews, other examples of non-test indicators may include:  sales results, customer retention, customer complaints, accidents, turnover, errors, etc.  We can provide summaries of this analysis or actual sample validation studies conducted for clients.

When evaluating personality assessment measures or styles inventories, it is important to determine whether the assessment authors perform only the first level of validity analysis, i.e. test to test, or, also validated the assessment results through correlations with actual performance behaviors.  The test development process determines the applicability of the assessment results to workplace decisions.  As with our assessments, only valid and reliable tools, as determined through the test development process, are valid for selection decisions.  In other words, our measures correlate to actual results.

   

What is a Correlation?

Correlation is a measure of how closely two variables move together through time, or the degree to which two variables are associated.  A positive correlation exists when two variables increase or decrease together.  For example, height and weight are positively correlated, meaning that as height increases, so does weight. More of one means more of the other.  A negative correlation exists when increases in one variable are accompanied by decreases in the other, and vice versa. For example, research might show that self-esteem and depression are negatively correlated, indicating that as self-esteem increases incidence of depression decrease.  More of one means less of the other.

What Does a Correlation Represent?

There is a simple technique for illustrating the real size and importance of correlations.  The “binomial effect size display” (BESD*) allows correlation coefficients to be interpreted in terms of the percentage of correct classifications they represent.

Effect (Impact) Sizes Corresponding to Various Values of Correlations
Correlation	Success Rate Increased	Difference in Success Rates
.10	From .45 to .55	10%
.20	From .40 to .60	20%
.30	From .35 to .65	30%
.40	From .30 to .70	40%
.50	From .25 to .75	50%

---

*  Adapted from “A Simple General Purpose Display of Magnitude of Experimental Effect, by R. Rosenthal and; D. B. Rubin.  In Journal of Educational Psychology, 74, 166-169.

About the Author:  Kimberly Leveridge, Ph.D., is co-founder of CDR Assessment Group, Inc. and past Vice President of Research & Operations.  She serves as CDR's Scientific Advisor and her current career is serving in a variety of talent development leadership roles within industry.

Blogger Note:  Kim wrote this piece some years back -- and it is the most clear and BEST explanation that I have found on how to know whether or not the assessments your are using or considering to use are valid.  We hope this is helpful information to you!  

                                                                                                                                                             Nancy Parsons


© 2004, CDR Assessment Group, Inc.  Tulsa OK, All Rights Reserved.   

Image courtesy of Stuart Miles at FreeDigitalPhotos.net

Worry Less, Lead Better

By Patricia Wheeler, Ph.D.

Managing Partner of The Levin Group

We’ve all heard the statistics….although women make up over half the workforce, less than a quarter of them rise to middle management positions.  And fewer than 10% ever claim a seat on their company’s executive committee.   These numbers haven’t changed in over a decade, despite the diversity efforts of many organizations.  

How do we make sense out of this?  What’s the problem?  How much of women’s lack of upward mobility is due to corporate culture that favors promotion of those who currently match the demographics of their executive chamber?  And what are the parts of the equation over which talented women do wield control and influence? 

As a leadership consultant and experienced executive coach, I have worked with hundreds of senior women executives over the past two decades.  I know that each of the above questions have merit, and advancing more talented women to the senior level will involve addressing both.  And while it’s extremely important that organizations embrace and create a corporate culture that values having diverse teams, our focus here is to look through the lens of what’s under a leader’s own sphere of control and influence.  This is where individuals can make the most immediate impact.  

There’s interesting and important research emerging from CDR Assessment Group.  CDR President Nancy Parsons and her team analyzed assessment results from over 250 male and female leaders in 26 organizations.  For those of you not familiar with CDR, they provide excellent assessments that detail one’s leadership style under “normal” conditions as well as predict leaders’ default styles under pressure. 

So what did they find?  One important finding is that there is no significant difference between the ratings of men and women on measures of overall leadership characteristics.  For example, there’s no overall difference between male and female leaders in drive for results, strategic perspective, or ability to work in teams.  If you’ve worked with as many senior women leaders as I have, this comes as no surprise.  Successful leaders come in all styles.    

The one factor differentiating men from women appeared in the assessment that reflects how a leader tends to behave under stress.  Under pressure, women were more likely than men to respond to pressure with excessive worry.  In the workplace, leaders possessing this characteristic are likely to delay making decisions due to fear of failure or criticism.  

Interestingly enough, women in Parsons’ research did not appear more perfectionistic than men, and as a group they did not try to please others more.  But they did worry more. And what might be the effect for those of us who possess this “Worrier derailer?”  Others may perceive a worrier as indecisive, self-doubting and even lacking courage which are not the sort of behaviors that will lead to a seat on the executive committee, regardless of how smart and technically talented the leader is. 

What’s the best course of action for a woman who possesses the Worrier derailer?  Here are some steps that have helped my “Worrier” clients:

1.  Take a good look in the mirror.  If you have this characteristic, so be it.  Understand that your worry is likely to be in excess of what many situations call for.   If you realize you’re likely to overreact under pressure, you can plan a more adaptive response in advance.  One of my clients does advance planning every time she has a meeting scheduled with her very critical boss.  She asks herself what he’s likely to react to negatively and mentally rehearses her messages to him.  Doing so has diminished his perception that she’s passive and reactive.  

Does this mean we can and should be able to eliminate worry?  Of course not!  Worrisome things do happen.  But knowing that you are likely to over-worry can help you anticipate situations where your “Worrier” may be triggered and plan appropriate coping strategies.  Remember that this tendency need not define you and your leadership.  

2.  Manage your arousal.  When we are under strong pressure, our biology changes.  We move into a “fight or flight” mode; this causes us to create more adrenaline and cortisol.  Think of these as the “unhappy neurochemicals” that help us escape actual danger (our biology hasn’t really changed since we had to outrun saber-toothed tigers) but don’t help us react adaptively to most workplace stresses.  Recognize when you’re under pressure, take a step back, and intentionally breathe more deeply and slowly.  This small step actually helps our body regulate itself out of the excessive worry zone.  Find ways of reminding yourself of your competence and plan in advance how you might better address recurring pressures without withdrawing into worry.  

3.  Embrace the positive.  Scientists are finding increasing evidence that when we make the effort to acknowledge, embrace and resonate with the positive events and emotions in our life, we increase our likelihood of success.  We make better decisions and we become better leaders.  And we diminish the likelihood of being held captive by our derailers and default settings.  

The even better news is that we don’t need to experience intense positive emotions to reap these benefits…..what serves us best is to create and keep the habit of acknowledging what we appreciate and what’s going well in our world.  

These small changes in self-awareness and behavior can help us stay out of “Worrier” and stay in our most effective leadership behaviors.  This, in turn, should help many talented women advance more effectively up the leadership pipeline.   

- Copyright 2015, Leading News

Patricia Wheeler
Acclaimed executive coach ranked in the top twenty consultants in the Leadership 500 and managing partner of The Levin Group.  With more than 25 years of coaching and consulting experience, she works with leaders around the world who must innovate and deliver exceptional business results within an environment of rapid change and increasing complexity.  An expert on senior leader success, she contributed to the AMA Handbook of Leadership and Coaching for Leadership, Third Edition.   Email Patricia today!

 Printed by Assessment Advisor with written permission by the copyright owner on 4/14/15.

Attn Executive Coaches: Are You Seeing Too Much Gray?

Perhaps your leadership coaching and development tools are too drab to identify true talent.

CDR Assessments provide clarity and self-awareness unlike any you may find.  Perhaps it is time to upgrade your toolbox! Join us in Oklahoma City on May 6 & 7, 2015 for a  CDR Coaches' Certification Workshop!     The CDR 3-D Suite, an unmatched assessment tool, along with highly skilled coaches, enables leaders to get beneath the surface.  The Suite peaks self awareness by pinpointing specific character, risk factors that can derail their effectiveness, and drivers and motivational needs.  These insights are essential to formulate and fast forward the best suited development plans.   Your client's will benefit by: accelerating the development process - by hitting the ground running at the first session clarity and self awareness unlike any past feedback no spin feedback - constructive, on point results without sugar coating focusing on their own unique true talent and strengths, gaps, risk factors and motivational needs quickly building accurate action plans for measurable results, and more... Patricia Wheeler, Ph.D., managing partner at The Levin Group, LLC in Atlanta wrote, " "The CDR 3D Suite is my favorite "deep dive" leadership style assessment for the leaders I coach.  It's my go-to assessment when clients are open for me to choose.  The reasons are twofold:  I genuinely like and trust the data and I genuinely like and trust the CDR folks. They're extremely professional and a delight to work with."    The CDR Coaches' Certification Process includes: Pre-requisite - participants take the CDR 3-D Suite and have 2.5 hour coaching feedback session   Two-Day Workshop with expert instructor Materials- actual case studies, manual, templates, leadership development action plans, and supplemental materials Complimentary CDR 3-D Suite - for practice leader coaching session after the workshop Marketing Kit - to help you promote your services to external or internal clients with CDR tools  Post Workshop Review - participants review their practice session preparation with an instructor before "going live" with client coaching session Workshop Location:   Oklahoma City University, Meinders School of Business, 2501 N. Blackwelder, Oklahoma City, OK For a full brochure or to enroll, contact:  cdrinfo@cdrassessmentgroup.com or call 918.488.0722