Peer review
To ensure that analytical pipelines available through the Biodiversity Evaluation Engine (BEE) are scientifically robust, reproducible, and accessible, BON in a Box has established a peer-review process for contributed pipelines.
The review process evaluates the scientific methodology, technical functionality, generalizability, documentation, and usability of each pipeline. It also provides an opportunity for researchers and practitioners to contribute their expertise to improving the tools available through BEE.
Pipelines are typically reviewed by 2–5 experts with complementary scientific, technical, or practical expertise. Where possible, we aim to include reviewers from both the academic and biodiversity monitoring or policy communities.
When is a pipeline ready for peer review?
Before a pipeline is submitted for external review, the BON in a Box team works with contributors to ensure that it:
Meets the requirements outlined in the Pipeline Standards.
Runs successfully using the provided example inputs and default parameters.
Has undergone initial testing with different input data, parameters, and geographical contexts, where applicable.
Produces scientifically reasonable results consistent with the underlying methodology.
Includes input and output descriptions, documentation, and a workflow description.
Has its code available in the BON in a Box GitHub repository, either in the main branch or in a development branch or pull request.
The BON in a Box team conducts initial testing and technical checks before inviting external reviewers. External reviewers are then asked to conduct additional, independent testing to evaluate the pipeline’s functionality, scientific validity, and applicability across different contexts.
How does the peer-review process work?
1. Reviewer selection
The pipeline contributor may suggest potential reviewers with relevant expertise. Additional reviewers will be identified by the BON in a Box team and GEO BON network.
We aim to select reviewers with complementary expertise across three areas:
| Preferred reviewer type | Primary areas of expertise |
|---|---|
| Scientific expert | Scientific methodology, assumptions, validation, and interpretation of results |
| Applied biodiversity monitoring expert | Generalizability, usability, and relevance to biodiversity monitoring, conservation planning, and reporting |
| Technical expert | Code quality, reproducibility, data handling, and pipeline functionality |
These categories are intended to guide reviewer selection rather than establish rigid roles. Reviewers may have expertise in multiple areas, and are not expected to evaluate aspects outside their expertise.
Reviewers are invited by the BON in a Box team and provided with the materials needed to evaluate the pipeline.
2. Pipeline evaluation
Reviewers are asked to evaluate the pipeline using a standardized review form, focusing on the following criteria:
Scientific rigor: Is the methodology scientifically supported and correctly implemented? Are the assumptions and limitations clearly communicated?
Generalizability: Can the pipeline be applied across different geographical areas, species, ecosystems, or scales, where appropriate?
Functionality: Does the pipeline run successfully and produce the expected outputs?
Data use: Are input datasets appropriate, accessible, and clearly documented? Can users provide their own data where relevant?
Usability: Are the inputs, parameters, and outputs understandable to users without extensive programming experience?
Documentation: Are the methodology, inputs, outputs, and interpretation of results adequately explained?
Code quality: Is the code readable, modular, reproducible, and appropriately documented?
Reviewers are encouraged to focus on the criteria most relevant to their expertise. They do not need to complete every section of the review form, but should provide sufficient feedback to support an assessment of the pipeline.
3. Review feedback and revisions
Reviewers submit their completed review forms to the BON in a Box team, including identified issues, recommendations, and an overall assessment of the pipeline.
The completed review forms serve as the formal record of the review. The BON in a Box team consolidates feedback and works with pipeline contributors to address reviewer comments.
Specific technical changes, bugs, and requested improvements are tracked through GitHub issues or pull-request comments, allowing contributors and the BON in a Box team to document and monitor revisions.
Depending on the extent of the revisions required, reviewers may be invited to reassess the pipeline before a final decision is made.
4. Review outcome
Following the review, a pipeline may be:
Accepted with minor revisions: The pipeline meets the main scientific and technical requirements, with only minor changes needed before publication.
Revisions required: More substantial changes are needed, and the pipeline will require additional review.
Not accepted: The pipeline does not currently meet the requirements for inclusion in BEE.
The BON in a Box team coordinates the final decision based on reviewer feedback and the Pipeline Standards.
Instructions for reviewers
Reviewers will receive a pipeline-specific review form containing the information and resources needed to complete their evaluation, including:
Instructions for accessing the pipeline through the online BEE platform.
Links to the relevant GitHub repository, branch, pull request, or source files.
Links to the pipeline description and supporting scientific documentation.
Any additional information or example datasets relevant to the review.
Pipelines undergoing review will be available through the online BEE platform, so reviewers do not need to install BEE locally. The corresponding source code will be publicly accessible through GitHub, including pipelines that have not yet been merged into the main branch.
Step 1. Access BEE
Reviewers can access BEE through the online platform.
If you do not already have access, create a free GEO BON account and follow the access instructions provided in your review form.
Step 2. Run the pipeline
Locate the pipeline identified in your review form and run it using the provided example inputs and default parameters.
Although the pipeline will have undergone initial testing by the BON in a Box team, reviewers are asked to independently verify that it runs successfully, produces the expected outputs, and provides clear input and output descriptions.
Step 3. Evaluate the methodology and documentation
Review the pipeline description and relevant scientific references.
Consider whether the methodology is appropriate, whether its assumptions and limitations are clearly explained, and whether the outputs are consistent with the intended application.
Step 4. Test different inputs and parameters
Where possible, run the pipeline with different parameters, geographical areas, or datasets.
We encourage reviewers to test the pipeline using data or study areas they are familiar with, as this can help identify unexpected results or methodological limitations that may not have been identified during initial testing.
Document any errors, unexpected behaviour, or concerns in the review form.
Step 5. Review the code (where applicable)
Reviewers with relevant programming experience are encouraged to examine the source code using the GitHub links provided in the review form.
Code review may include checking the implementation of the scientific methodology, reproducibility, modularity, and error handling.
Specific technical issues can be documented in the review form or raised directly through comments on the relevant GitHub pull request.
Step 6. Submit your review
Complete the relevant sections of the review form, including your assessment of the pipeline, any recommended changes, and your overall recommendation.
Reviewers are encouraged to provide detailed feedback within their areas of expertise but are not required to assess every aspect of the pipeline.
Return the completed form to the BON in a Box team using the instructions provided with your review invitation.
Pipeline publication
Peer reviewed and accepted pipelines will be published on Zenodo and given a DOI, which will be cited whenever the pipeline is used.
Review form
Reviewers will receive a pipeline-specific review form with their invitation. The form includes the pipeline details, relevant links, evaluation criteria, and instructions for submitting feedback.