Principles of High Quality Assessments l Validity and Reliability

This video is the second part of a lecture on principles of high-quality assessments, focusing on validity and reliability. Validity, derived from 'valid' meaning acceptable or appropriate, refers to whether an assessment instrument measures what it intends to measure, such as learning objectives, student learning, instructional effectiveness, or material effectiveness. It is the most important criterion for a good assessment and should be appropriate for the objectives and the learners' level. Assessments can be for formative, diagnostic, or summative purposes.

The video outlines four types of validity. Face validity refers to the physical appearance of the instrument, including font style, size, indentation, layout, and clarity of copies, which can impact a student's initial impression and emotional response to the test. Content validity, considered the most important, ensures the assessment covers the knowledge, skills, and concepts discussed in class. Teachers should only include items relevant to what was taught and can use a Table of Specifications (TOS) to ensure a balanced distribution of items across topics.

Criterion-related validity involves relating the assessment's criteria to other established exams and is statistically established. It has two purposes: concurrent validity, which estimates current performance based on criteria (e.g., mock exam results), and predictive validity, which forecasts future performance. Construct validity refers to the psychological traits and effects of the instrument, assessing if the test truly measures the intended construct (e.g., emotional maturity). Factors like student readiness, external disturbances, lack of awareness of the exam, unclear copies, or ambiguous instructions can affect construct validity.

Several factors can affect an assessment's validity. Unclear directions can mislead students (e.g., an example where a student wrote 'sum' instead of the numerical answer). Vocabulary and sentence construction should be kept short and simple. Ambiguity of options in multiple-choice questions should be avoided by making options homogeneous and plausible. Inadequate time limits can hinder performance. Over-emphasis on easy or difficult questions can be problematic; a gradual increase in difficulty is recommended. Using inappropriate instruments for objectives and poorly constructed test items (which can become 'bonus' items) also decrease validity. Finally, the length of the test and the readiness of students (who should be informed about upcoming exams) are crucial.

Principle number four is reliability, which means an assessment can be trusted. In assessment, reliability refers to the consistency of scores obtained by the same person or group over time or across different forms. Reliable test results are dependable, consistent, and repeatable.

The video outlines five types of reliability. Test-retest reliability measures stability by administering the same test twice over a period of time (e.g., pre-test and post-test) to measure learning gains. However, this method can be questionable due to practice effects or a lack of seriousness from students during the re-test. Alternate form reliability (also known as equivalent or parallel form) addresses this by using two different sets of test items that cover the same competencies and standards, administered either twice (pre-test and post-test with new items) or once (Set A and Set B).

Split-half reliability measures internal consistency by dividing a single exam into two halves (e.g., odd and even numbered items) and correlating the scores. Kuder-Richardson reliability also measures internal consistency, specifically for binary or dichotomous tests (two-answer items like true/false). Cronbach's alpha, another measure of internal consistency, is used for essays and other tests, estimating reliability using standard deviation per item and of test scores.

Factors affecting reliability include test length; longer assessments generally produce more reliable results. The spread of scores, measured by standard deviation, also impacts reliability; a wider spread often leads to higher reliability. Item difficulty plays a role: very easy or very difficult items can decrease reliability, while items of average difficulty tend to increase it. Item discrimination and time limits are also important considerations for improving reliability.