How Open Response Assessment Problem Type Functions?

Overview

In open response assessments (ORA), learners submit essay responses and then go through a series of assessment steps (such as peer assessment and self assessment) to complete the assignment.

Tip: Make sure you have logged in to the Open edX Studio before proceeding with the article.

Note: Open response assessments that are visible to all learners do not respect cohorts. In other words, it is possible for learners in one cohort to be asked to grade responses for learners in another cohort. If you want to make an open response assessment divided by cohort, you must create that assessment in a course component that is defined as cohort-specific.

Elements of an Open Response Assessment

When you create an open response assessment assignment, you include several elements.

One or more prompts, or questions, that learners answer.
- A prompt is the question that you want your learners to answer. You can add more than one prompt in an ORA assignment. In addition to requiring a written response, you can require or allow learners to upload an image or other type of file to accompany their written response.
- Within each prompt, you can include helpful information for your learners, such as the approximate number of words or sentences that their responses should have, the types of files that they can upload, or what they can expect after they submit their responses.
- In the learner view of the assignment, each prompt appears above the field where learners enter their responses.

One rubric is used to grade all the prompts in the assessment. A rubric consists of several criteria and a set of options for each criterion.
- Criteria. Each criterion describes characteristics that a response should have. Examples are concepts that a response should cover, or the amount of supporting information that a response must include. Each criterion has a name and a prompt.
  - The criterion name is a one or two word summary of the criterion, such as “Content” or “Organization”. This name must be unique within the assignment and cannot be changed after you release the assignment.
  - The criterion prompt describes how to evaluate a response based on this criterion.
- Options. Each criterion has a set of options, usually a range of ratings, which describe how well each response satisfies the criterion. For example a set of options might be “Fair”, “Good”, or “Excellent”. Each option has a name, an explanation, and a point value.
  - The option name is a one or two word summary of the rating.
  - The option explanation consists of details that help the person performing the assessment to decide whether the response matches the rating. Make sure the explanation for each option is as specific as possible.
  - The option point value is the number of grade points given for this option.

Note: Different criteria in the same assignment can have different numbers of options.
You can also include criteria that do not have options, but that do include a field where learners or staff can enter feedback.

One or more assessment steps. Assignments can include a learner training step, a peer assessment step, a self assessment step, and a staff assessment step.

Assessment Steps

In your assignment, you also specify the assessment steps. You can set the assignment to include some combination of the following steps.

Learner Training Step
Peer Assessment Step
Self Assessment Step
Staff Assessment Step

Note: If you include a learner training step, you must also include a peer assessment step. The learner training step must come before peer or self assessment steps. If you include both peer and self assessment steps, edX recommends that you place the peer assessment before the self assessment. If you include a staff assessment step, it should be the final step in the assignment.

Learner Training Step

Learner training steps teach learners to perform their own assessments. A learner training assessment contains one or more sample responses that you write, together with the scores that you would give the sample responses. Learners review these responses and try to score them the way that you scored them.

Note: If you include a learner training step, you must also include a peer assessment step. The learner training step must come before any peer and self assessment steps.

In a learner training assessment, the Learn to Assess Responses step opens immediately after a learner submits a response. The learner sees one of the sample responses that you created, along with the rubric. The scores that you gave the response do not appear. The learner also sees the number of sample responses that he or she will assess.

The learner selects an option for each of the assignment’s criteria, and then selects Compare your selections with the instructor’s selections. If all of the learner’s selections match the selections defined by the course team, the next sample response opens automatically.

If any of the learner’s selections differ from those specified by the course team, the learner sees the response again, with a message indicating that his assessment differs from the instructor’s assessment.

The learner continues to try scoring the sample response until his scoring for all criteria matches the scoring defined by the course team.

Peer Assessment Step

In the peer assessment step, learners review the responses of other learners in the course. For each response, they select an option for each criterion in your rubric based on the response. Learners can also provide text feedback, or comments, on each response.

If you include both peer and self assessment steps, edX recommends that you place the peer assessment before the self assessment.

Self Assessment Step

In self assessment steps, the learner sees her own response followed by the rubric. As with peer assessments, the learner evaluates the response using the rubric, selecting an option for each criterion.

If you include both peer and self assessments, edX recommends that you include the peer assessment before the self assessment.

Staff Assessment Step

In staff assessment steps, a member of the course team performs an evaluation of the learner’s response. Course team members grade the response using the problem’s rubric, in the same way that self and peer assessments are done, and can include comments in their assessment.

Note: If a staff assessment step is included in an assignment, learners do not receive final grades until the staff assessment step has been completed. The scores that you give learners in staff assessment steps override scores from any other assessment type in the assignment, including peer assessments that are completed after the staff assessment.

Including a staff assessment step in an ORA assignment is best for courses with smaller groups of learners. For example, in a course with cohorts, you might create an ORA assignment that has both peer assessment and staff assessment steps, and make it available only to the members of one or more specific cohorts. For the members of the remaining cohorts, you create an ORA assignment that has only the peer assessment step.