Domain Review
The domain review checks that a code is scientifically valid. This review should be conducted once the code is ready for release. The reviewer should be someone with expertise in the scientific topic of the code. Although not required, it’s often useful for the reviewer to also have experience with the project’s programming language. The criteria for a domain review will vary with the type of code being released. See the following sections for details:
Reference Code
The domain review for reference code primarily focuses on the code’s output. Essentially, the reviewer should confirm that the code produces the output presented in the associated publication. In some cases, it may not be feasible to actually re-run the code (for example, code that runs a long simulation). In this case, it’s sufficient to check the code itself to see if it will reasonably produce the expected output.
Rubric
See the table below for a suggested domain review rubric for reference code:
Item |
Pass |
Fail |
|---|---|---|
Output aligns with published or expected results. |
Code does not produce expected or published results. |
Software Tools
Since software tools will be used many times, they have stricter review requirements. Here, the reviewer should ensure that the code documents the theory it is based on, correctly implements that theory, and describes the code’s scope and limitations. Items in this review include:
Scope
The project should describe the intended scope of the code. If the code has limitations or a restricted domain, then these limitations should be described.
Theory
The project should describe the scientific theory the code is based on, and this theory should be scientifically reasonable.
Functionality
The code should correctly implement the theory it intends to implement.
Rubric
See the table below for a suggested domain review rubric for reusable software tools:
Item |
Pass |
Fail |
|---|---|---|
The project describes the intended scope of the code, or the scope is implied by the code’s design. |
The code’s scope is unclear or not described. |
|
The project documents the theory the code is based on. |
The project does not document theory. |
|
Any theory is scientifically reasonable given the scope of the code. |
The theory the code is based on is not scientifically justified. |
|
The code successfully implements the scientific routine it is intended to do. |
The code does not reproduce expected scientific results. |
Gitlab How-To
You should implement a domain review using a Gitlab issue thread. Navigate to the main page of the upstream repository. In the navigation sidebar, click the Plan menu and select the Issues button.
Show button
Click the New issue button. This will take you to a page for the issue. Set the title to Domain Review and also include the release tag. In the description, you should include:
The new features of this release,
The name of the domain reviewer, and
Any pertinent information for the reviewer.
If your new code is complex, it may be useful to outline a plan for conducting the domain review as well.
Show Example
Next, add the Domain Review label to the issue. If you have not yet created these labels, you can do so by clicking the Create project label button in the dropdown box:
Show Example
Click the Create issue button, and direct your reviewer to the associated page. You and the reviewer should use the comment thread to document the review and reconciliation process. Once the review is reconciled, the reviewer should leave a comment explicitly stating that they are satisfied with the review reconciliation. You may then click the Close issue button to resolve the review. This will archive the review in the closed issues for the repository.
Show Example
