The Code Review Analysis page works by analyzing pull request comments alongside their reactions and responses. It leverages LLMs and non-A.I. algorithms to analyze whether or not a comment lead to action and to categorize what type of comment it was. Only comments on other people’s merged pull requests are analyzed.
Analysis occurs once a day a 3 AM UTC
Category Definitions
Review Impact: A scoring algorithm that takes into account “Found Issue” and “Improved Code Quality” comments. See How Review Impact is calculated.
Found Issue: A comment or question suggesting that the implementation is not accurate in some capacity OR a comment or question suggesting that the implementation is missing something. For example, a missing edge case, log statement, or test.
Improved Code Quality: A comment or question suggesting that the code quality could be improved. For example, a comment about readability, maintainability, reusability, typos, dead code, simple syntax changes etc.
Other: Any comment that does not fit into the above categories. For example, a comment that is a compliment, a note to self, a 'see other comment' note, or a feature suggestion that is out of scope. Comments that did not lead to action, are also counted as “Other”.
Pending: Comments waiting for the pull request to be merged or for nightly analysis to occur.
How Review Impact works
Formula
Comments categorized as 'Found issue' or 'Improved code quality' are counted towards impact with the category of ‘Found Issue’ being weighted highest.
Qualification
A comment qualifies for Review Impact if ALL following criteria are met:
- The comment is the first comment on a code review thread OR is a top-level comment on the pull request
- The LLM categorizes the comment as Found Issue or Improved Code Quality
- The comment led to action. To be deemed as something that led to action, the comment must meet one of the criteria:
- The LLM deems the comment as having led to action in the context of the thread
- GitView’s non-llm algorithms determines the comment is as having led to action based on reactions and code changes
- The pull request is merged
- The comment is on someone else’s pull request.
Accuracy
For finding and analyzing trends, we and our customers have found the GitView Code Review Analysis tool to be very accurate.
While GitView’s Code Review Analysis occasionally makes mistakes due to its reliance on LLMs for analysis, these do not materially affect the overall trends or data.
However, if you believe you have found a bug that impacts the overall trend of a team or contributor over a 14-day period or longer, please message us via Intercom, and we will investigate.
Why can’t I view ‘Other’ comments?
We don’t surface the 'Other' category of comments to ensure people don’t 'miss the forest for the trees.' Focusing too much on why a specific comment was not counted towards impact could distract from the overall trends our algorithms detect. Since the feature is highly accurate at identifying trends, we want to prevent discouragement or mistrust in the big picture due to the misclassification of a few comments.
See Accuracy section above for more info.