RepoContextBench v2: publish Claude Code Hard runs#2
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request integrates support for the Claude Code answerer, introducing options for safe mode, expected versions, resolved models, and system prompt instructions. It also adds JSON/JSONL-specific sensitive content scanning during publication export, normalizes judge provider comparisons, and updates benchmark results. The review feedback identifies three key areas for improvement: handling potential JsonException when parsing malformed JSON, addressing case-sensitivity issues in file extension matching, and preventing a potential NullReferenceException when validating Claude Code metrics.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| private static bool ContainsSensitiveJson(string text) | ||
| { | ||
| JsonNode? node = JsonNode.Parse(text); | ||
| return node is not null && ContainsSensitiveJson(node); | ||
| } |
There was a problem hiding this comment.
If the input text is empty, contains only whitespace, or is otherwise malformed JSON, JsonNode.Parse(text) will throw a JsonException, causing the publication export process to crash.
To make this more robust, wrap the parsing logic in a try-catch block and fall back to plain text scanning (ContainsSensitiveText) if a JsonException is thrown. This ensures the export process continues safely while still catching any sensitive data.
private static bool ContainsSensitiveJson(string text)
{
try
{
JsonNode? node = JsonNode.Parse(text);
return node is not null && ContainsSensitiveJson(node);
}
catch (JsonException)
{
return ContainsSensitiveText(text);
}
}| || WindowsAbsolutePath().IsMatch(text) | ||
| || SecretValue().IsMatch(text) | ||
| || ObjectIdLike().IsMatch(text)) | ||
| bool containsSensitiveContent = Path.GetExtension(path) switch |
There was a problem hiding this comment.
Path.GetExtension(path) can return extensions with mixed casing (e.g., .JSON or .Json). Since the switch expression is case-sensitive, files with uppercase extensions will bypass the JSON-specific scanning and fall back to plain text scanning.
Convert the extension to lowercase using ToLowerInvariant() to ensure case-insensitive matching.
bool containsSensitiveContent = Path.GetExtension(path).ToLowerInvariant() switch| if (!string.IsNullOrWhiteSpace(command.ClaudeExpectedResolvedModel) | ||
| && !string.Equals( | ||
| result.Metrics.ResolvedModel, | ||
| command.ClaudeExpectedResolvedModel, | ||
| StringComparison.Ordinal)) | ||
| { | ||
| throw new InvalidOperationException( | ||
| $"Claude Code resolved model must be {command.ClaudeExpectedResolvedModel}; " + | ||
| $"observed {result.Metrics.ResolvedModel ?? "<missing>"}."); | ||
| } |
There was a problem hiding this comment.
If result.Metrics is null (for example, if the JSON response is missing the metrics object or deserialization fails), accessing result.Metrics.ResolvedModel will throw a NullReferenceException.
Use the null-conditional operator (?.) to safely access ResolvedModel and prevent potential crashes.
if (!string.IsNullOrWhiteSpace(command.ClaudeExpectedResolvedModel)
&& !string.Equals(
result.Metrics?.ResolvedModel,
command.ClaudeExpectedResolvedModel,
StringComparison.Ordinal))
{
throw new InvalidOperationException(
$"Claude Code resolved model must be {command.ClaudeExpectedResolvedModel}; " +
$"observed {result.Metrics?.ResolvedModel ?? \"<missing>\"}.");
}1 participant
Summary
Results
Both runs contain 20/20 scored tasks, use the frozen Sapling commit and Codex CLI gpt-5.6-sol medium judge, and pass publication admission with zero rejected runs.
Validation