TAAASS501A Lead and coordinate assessment systems and services

Coordinate assessment validation activities

Figure 10 Validation of assessment system and services

What is validity?

Validity is broadly concerned with the assessment process actually assessing what it is supposed to assess. Since 1999, the National Assessors and Workplace Trainers Body's national Training Package has supported a view that:

A valid assessment assesses what it claims to assess; evidence collected is relevant to the activity and demonstrates that the performance criteria have been met (1999:18).

However, more recently the management of validation has been viewed as:

. . . a process involving assessors working in collaboration to review, compare and evaluate their assessment process and their assessment outcomes in relation to the same unit of competency (see Range Statement for this competency in the TAA04 Training and Assessment Training Package, 2004)

The above definition focusses very much on the validity of the judgment and interpretation made against an assessed outcome. This focus is a shift beyond the more usual focus of instrument and process validity to user validity or validation based on use. This brings into play the necessity to clarify not just what was intended to be assessed, but how different assessors or different contexts affect the intended assessment outcomes.

Validity suggests that unless otherwise stated assessment should not vary in use across different:

• Assessors;
• Components of a complex task being assessed;
• Means to collect evidence;
• Individuals being assessed;
• Assessors' interpretations; or
• Contexts.

Any limitations that assessment may have in relation to the above would indicate potential problems with the ability to consistently collect evidence and make authentic judgments.

Validity is therefore about the robustness of the instruments (tool) and the guiding assessment process. While an assessment instrument and process can be designed to accommodate issues with validity, over time the conditions affecting implementation can change. This is why managing assessment validity is an ongoing exercise.

Validity is closely related to reliability. Reliability relates to how accurate, precise and consistent the assessment instrument or process may be over time. A lack of reliability may suggest problems with the accuracy or consistency of evidence being collected.

Review the assessment system validity policies and procedures

Determining validity in assessment can be completed using a number of approaches. Complete the following reading to identify some current practices and approaches to improving validity in competency-based assessment.

Reading 5

Thomson, P, Saunders, J & Foyster, J (15 August 2001), Improving the validity of competency-based assessment, NCVER: Adelaide . Sourced online June 2003 at http://www.ncver.edu.au/publications/621.html .

Testing the validity of assessment tools

Prior to implementation of an assessment instrument it is usually necessary to validate the tool or to pilot and test the process.

Some basic questions need to be asked when preparing a plan for the trialling of an assessment tool.

Pre-trial setup questions include:

• Can we produce a trial and design brief?
• What is the budget and critical resources profile for the trial?
• Who will manage the trial?
• Who will conduct the trial (internal or external staff, consultants, etc.)?
• What is the project timeline and how long before the trial must report the results?
• How will the results be reported?

The above questions may assist in preparation for the trial but some model questions can be posed for the actual planning of the trial . These include:

• What aspects of the tool or procedures need to be tested?
• What information needs to be sourced?
• What are the parameters of the trial (cost, population, criteria, etc.)?
• What methods need to be used to collect and report information/data?
• What instruments will be used to collect and report information/data?
• What will constitute a reliable sample or valid research report?

Finally the planning phase needs to move into an implementation phase where it is using methods and approaches to research that are consistent with the purpose of the trial.

Piloting and trialling the assessment tools

When confirming the 'fitness for purpose' of any assessment tool it is important to revisit the principles of assessment. The question is not just about the tools measuring what it is supposed to measure on a consistent basis; it is also about whether the tool meets criteria such as reliability, validity, flexibility and fairness. An authentic tool should be able to address all these principles and assist the collection of evidence across a full range of contexts and scenarios. Testing reliability and validity together form the key basis for determining the design integrity of an assessment system, process or tool.

Testing for reliability is about removing concerns that the tool can be subject to random variations. For any measure or set of procedures associated with the assessment instrument and supporting documentation it must be possible to quantify and qualify precisely the likelihood of variances occurring. Reliability measures the chance for standard error or inconsistencies in application occurring, whether controls are sufficient, and the likelihood that chance assessment decisions can be made inaccurately.

Validity must also be considered when trialling or piloting an assessment instrument. Having a reliable assessment tool does not confirm whether it is measuring all aspects of the evidence or whether it is measuring to a degree sufficient to ensure that the required assessment decision is not subject to distortion, inaccuracy, or deficiency. To do this validity must be confirmed.

Tests need to ensure that the distortions possible through criteria (competencies) are not given insufficient or uneven weighting. They also need to remove deficiencies where the tool has omitted coverage of important activities or outcomes. This may occur where assessment omits coverage of aspects of behaviour, knowledge or skills that can underpin a competency. Removal of inaccuracy may relate to how assessment weights certain outcomes, how they are recorded and how the records of evidence are compiled into a final judgment.

A synthesis of all the above problems and the failure to test for the validity of assessment tools are commonly found where the underpinning knowledge within a competency is not assessed as rigorously as the skill components. In some cases greater reliability and validity can be obtained when units of competency are assessed together. Holistic assessment occurs when more than one unit of competency are assembled together to permit the assessor to make a more balanced and valid judgment of individual competence. This approach can also prevent the assessment process from dealing with disaggregated competencies that are never assessed in a manner that relates to real performance in a range of contexts.

Nunnally and Bernstein (1994) suggest the validity of an instrument should be considered or tested from three aspects:

Content validity -are all domains of evidence covered in sufficient depth and breadth?

Criteria-related validity -do the scores and evidence collected relate to specific outcomes where judgments are to be made?

Construct validity -do the tool and instrument measure all aspects of the underlying and demonstrated knowledge and skills sought?

The ultimate aim is also to keep the assessment tool practical and easy to use. Easy-to-source measures and effective tools for providing a benchmark and basis for feedback between trainer and trainee must be encouraged. In some cases this may challenge how well we can design valid and reliable tools, but the aim is to ultimately have an instrument, process or system all individuals will consider reliable and valid.

When focussing on competency-based assessment, the tendency in workplace assessment is to rely on observation and skills tests. It is unlikely that assessment tools and processes can be designed to conform to this mindset alone. Many assessors need to be schooled in how different tools can achieve more reliable and valid outcomes. They also need to realise psychometric and qualitative measures may hold added-value to the competency-based criteria if a more useful assessment decision is made.

It is possible to nominate different approaches to testing/trialling/piloting of an assessment tool and map how these advantage the reliability and validity of the tool.

Table 7 Trialling methods and their impact on the reliability and validity of an assessment tool

Testing Options	Reliability	Validity
Sampling-test the predictability of outcomes across groups, locations, levels, demographic populations, etc.	• •	• • •
Control groups-work from known domains of application or where an assessment tool has proven its success and contrast application and trials elsewhere.	• •	• •
Comparative tests-compare and contrast trials and samples from numerous locations across domain, metric, level of application, population/demographic group, criteria, etc. This may also involve cross-sectional analysis where similar groups across different times/location/levels would trial a tool or a set of measures.	• • •	• • •
Simulations-establish simulations in real, contrived or assessment centres to test application.	•	• • •
Statistical analysis-systematically analyse and report on performance measures or design integrity across domain, metric, level of application, population/demographic group, criteria, cross-criteria (e.g. multiple competencies), etc.	• • •	• •
Longitudinal study-examines measurement of outcomes and application over an extended time or population sample.	• • •	• •
Grounded theory (Strauss & Corbin 1990)-comparison between expected results based on existing studies and literature and data from actual performance improvement determined in a trial.	•	• •
Multiple measurement/criteria trials-using a number of measures or procedures to ascertain evidence that results are collated and compared to derive a composite process and convergence to an agreed set of measures that work across different scenarios, situations and criteria (James, 1973:76).	• • •	• • •

KEY
• • • = Very appropriate
• • = Appropriate
• = Useful

Figure 11 Process for trialling and piloting assessment tools

Complete risk analysis on validation activities

Risk management and analysis is common to all areas of management in the modern organisation. Assessment processes and systems are no less responsible for reducing risk and being conducted within risk management guidelines. Risk management is best defined as:

The chance of an uncertainty happening that will have an impact on objectives. It is measured in terms of a combination of the likelihood of a risk event and its consequences ( Standards Australia's AS/NZS 4360:2004, Risk management and HB436:2004, Risk management guidelines (August 2004) available from http://www.standards.org.au/ ).

Risk analysis and assessment are defined as:

A systematic approach for describing and/or calculating risk. Risk analysis involves the identification of undesired events, and the causes and consequences of these events ( http://www.nr.no/~abie/RiskAnalysis.htm ).

In terms of the assessment process, validation is responsible for reducing:

• The risk of assessment instruments and processes producing the incorrect outcome; and
• The assessment decision placing someone in a position whereby they can cause harm within an operational context.

Given the above validation activities, the assessment of a nuclear plant main valve operator would be considered more critical to get right than, for example, the assessment of a mining drill operator's non-verbal communication skills. While a glib comparison the possibility of risk is self-evident.

A systematic approach for the leader seeking to address risk within an assessment process or system would be to:

• Establish the context . Define and identify the organisation and assessment environments, characteristics, dependencies and stakeholders, their goals and objectives, and the scope and boundaries of the specific risk management process. Develop criteria against which risks are evaluated and identify the structure for risk management.
• Identify and define risks . The assessment risks are best identified through a collaborative approach involving a wide cross-section of stakeholders in the project and recorded in a formal risk register. All conceivable risks, including 'show stoppers', must be considered. Ensure any certainties are identified as problems and addressed in the assessment processes.
• Conduct assessment risk analysis. An analysis of the risks is conducted to determine their causes, and estimate their probability and consequences.
• Conduct assessment risk evaluation . The assessment risks are considered and prioritised according to their potential impact on the business and the project, and each risk is assessed to determine its level of acceptability.
• Develop and implement risk treatments . Risk treatments are developed to cost-effectively reduce, contain and control project risk. Formal risk management reporting mechanisms are also built into reporting of assessed outcomes.
• Monitor, report, update and manage risks . As risks change during the project, the risk profile is continuously monitored, reviewed and updated. New risks may be identified as more information becomes available and existing risks may be eliminated through the effectiveness of the risk treatments.

(Source: OCIT, 2004, http://www.oit.nsw.gov.au/Guidelines/4.3.24.d-Project-RM.asp#a70 )

Coordinate and improve validation process

As with any quality cycle it is vital that the validation process also become a cycle of improvement. As such it is important for the manager or leader of the assessment process and system to:

• Gain participation for suitably qualified and committed personnel to attend validation panels or activities;
• Keep accurate documentation on the validation process and the role of individuals is provided; and
• Conduct reporting and feedback such that it immediately stimulates action to trigger improvements.