Standard Setting and Automated Test Assembly
Creating effective assessments requires precision in both defining performance standards and assembling tests that align with intended goals. Our methodologies ensure that tests are both fair and optimally constructed.
Standard Setting
Standard setting is the process of determining the performance levels that define proficiency on a test. Our approach involves both qualitative and quantitative methods to ensure accuracy and fairness.
1. Defining Proficiency Levels
- Establishing clear performance categories (e.g., Basic, Proficient, Advanced).
- Collaborating with subject matter experts to define what constitutes mastery at each level.
2. Methodologies We Use
- Angoff Method: Experts estimate the probability of a minimally competent candidate answering each item correctly. These probabilities are averaged to set cut scores.
- Bookmark Method: Items are ordered by difficulty, and experts determine the point at which a candidate transitions from one proficiency level to another.
- Contrasting Groups Method: Performance data from different groups (e.g., high and low performers) are analyzed to identify cut scores.
3. Validation of Standards
- Conducting pilot tests to ensure the set standards are reasonable and achievable.
- Analyzing the impact of cut scores on population performance to refine them further.
Automated Test Assembly
Automated Test Assembly (ATA) is a technology-driven approach to creating tests that meet specific design and psychometric criteria. This ensures tests are constructed efficiently and consistently.
1. Key Features of ATA
- Item Pool Optimization: Selecting items from a large pool based on pre-defined criteria such as difficulty, content coverage, and statistical properties.
- Customizable Test Blueprints: Ensuring the test aligns with specific content domains, skill areas, and cognitive levels.
2. Steps in Automated Test Assembly
- Defining Constraints: Setting parameters for test length, difficulty range, content distribution, and exposure control.
- Algorithmic Assembly: Using optimization algorithms (e.g., linear programming) to select the best combination of items that meet all constraints.
- Simulation and Testing: Running simulations to validate the assembled test and ensure it performs as intended.
3. Benefits of ATA
- Efficiency: Reduces the time required for manual test construction.
- Consistency: Ensures all versions of a test are equivalent in difficulty and content.
- Scalability: Easily accommodates large-scale testing needs.
