Bottom-Up Scoring (Default Probench Scoring)

Overview

Bottom-up scoring is a method used in surveys to calculate scores for higher-level categories by aggregating scores from lower-level components. This approach is particularly useful when analyzing hierarchical data, such as surveys with sections, subsections, and individual questions, where the results from smaller components contribute to the overall score.

Key Concepts

Hierarchical Structure:

• Surveys are divided into multiple levels, such as questions, subsections, sections, and the overall survey.
• Each lower-level component feeds into the scoring of its parent level.

Weighting:

• Assign weights to questions or subsections to reflect their relative importance.
• Weights should sum up appropriately at each hierarchical level to ensure accurate aggregation.

Normalization:

• Normalize scores to a common scale (e.g., 0-100) for consistency.
• This ensures comparability across different components.

Aggregation Rules:

• Define rules for aggregating scores, such as averaging, summing, or using weighted averages.

Step-by-Step Process

1. Design the Survey

• Define the Structure: Identify sections, subsections, and individual questions.
• Set Weights: Assign weights to each question and subsection.

2. Collect Responses
Gather responses from participants using your survey platform.

3. Calculate Scores at the Lowest Level

• Scoring Questions: For each question, calculate the score based on the response and its scoring criteria.
• Normalize scores to a predefined scale if necessary.

4. Aggregate Scores Upwards

• Subsections: Combine question scores within a subsection using the defined aggregation rules.
• Sections: Combine subsection scores within a section.
• Overall Score: Aggregate section scores to compute the overall survey score.

6. Validate and Analyze

• Check Consistency: Ensure that the calculated scores align with the weights and aggregation rules.
• Analyze Results: Use the scores to identify strengths, weaknesses, and trends.

Example

Survey Structure

• Section 1: Customer Satisfaction (Weight: 0.6)
  • Question 1.1: Rate the quality of service (Weight: 0.5)
  • Question 1.2: Rate the timeliness of service (Weight: 0.5)
• Section 2: Product Feedback (Weight: 0.4)
  • Subsection 2.1: Product Quality (Weight: 0.4)
    • Question 2.1.1: Rate product durability (Weight: 0.7)
    • Question 2.1.2: Rate product design (Weight: 0.3)
  • Subsection 2.2: Product Usability (Weight: 0.6)
    • Question 2.2.1: Rate ease of use (Weight: 1)

Scoring

1. Questions: Calculate scores for each question based on participant responses.
2. Subsections: Weighted average of scores for questions within a subsection.
3. Sections: Weighted average of subsection scores within a section.
4. Overall: Weighted average of section scores for the overall survey.

Individual Question Scores

• Section 1: (50x0.5) + (50x0.5) = 50%
  • Question 1.1 : 50%
  • Question 1.2 : 50%
• Section 2: (85x0.4) + (100x0.6) = 94%
  • Subsection 2.1: (100x0.7) + (50*0.3) = 85%
    • Question 2.1.1: 100%
    • Question 2.1.2: 50%
  • Subsection 2.2: 100%
    • Question 2.2.1: 100%

Final Scores

Section 1: 50
Section 2: 94
Overall: (50x0.6) + (94x0.4) = 67.6

Benefits of Bottom-Up Scoring

• Provides a detailed breakdown of performance at every level.
• Highlights specific areas of strength and weakness.
• Offers flexibility to tailor weights and aggregation methods.

Tips for Implementation

• Use a survey platform that supports hierarchical data structures and customizable scoring.
• Test the scoring method with sample data to ensure accuracy.
• Clearly communicate scoring criteria to participants and stakeholders.

Conclusion

Bottom-up scoring is an effective method for analyzing hierarchical survey data. By aggregating scores systematically from the bottom up, you can gain detailed insights while ensuring an accurate overall assessment.