Widen sizeInBytes beyond 32-bit Int so files over 2 GB are queryable — Closes #83

[conradbzura]

Summary

Introduce a BigInt custom scalar — a signed 64-bit integer serialized as a JSON number — and apply it to sizeInBytes on both FileMetadataType (output) and FileMetadataInput (input filter).

GraphQL's specification fixes Int at 32 bits, so any file over 2,147,483,647 bytes could not be represented: the field resolved to null and the response carried an Int cannot represent non 32-bit signed integer value entry in errors, degrading a whole page of results to a partial one. That is the common case rather than an edge case — every one of the 3,581 ENCODE .hic files runs 6–51 GB, and the larger 4DN mcools exceed the ceiling too.

The wire representation is the real decision here, so it is worth stating explicitly. GraphQL has no 64-bit Int, and the two realistic options are a BigInt that serializes as a JSON number and one that serializes as a String. This PR chooses the number.

The case against a number is real but does not bind here: a JSON number above Number.MAX_SAFE_INTEGER (2^53-1) is not exactly representable in JavaScript, and the consumer is a browser client (Gosling Designer), so "fits in 64 bits" is genuinely not the same as "the client can read it". But 2^53 bytes is roughly 9 PB. No file this API serves — or plausibly ever will — comes within five orders of magnitude of that ceiling, so the precision hazard is theoretical while the ergonomic cost of a String is immediate: every consumer of sizeInBytes would have to parse before comparing or summing, and a client that forgot would get silently wrong arithmetic ("6262125716" + 1 is "62621257161" in JavaScript) rather than a loud failure. A number keeps sizeInBytes directly usable and keeps the JSON body byte-for-byte what a client already expected, only correct.

The scalar's schema description records the 2^53 caveat so the reasoning survives in the published contract, and _coerce_big_int rejects non-integers (including true/false, since bool subclasses int in Python) and anything outside the signed 64-bit range, in both directions. The 64-bit bound is not arbitrary: it is exactly where BSON stops, so no value the scalar admits can fail on the way to MongoDB. Staying under 2^53 is deliberately not enforced — that would invent a second, softer limit inside a type whose name promises 64 bits — so it is documented as an admission criterion for routing any future field through BigInt.

What breaks for a client that assumed Int. Four things. A query declaring query Q($s: [Int!]) and passing it to sizeInBytes now fails variable-type validation and must declare [BigInt!]. Generated clients must re-run codegen against the new SDL and add a scalar mapping — graphql-codegen silently widens an unrecognised custom scalar to any, so without scalars: { BigInt: 'number' } the field loses its type with no build failure. Any client validating responses against a stored copy of the schema must refresh it. And a client sending an integral float (1234.0) is now rejected, where Int coerced it. A client that merely reads sizeInBytes out of the JSON response needs no change at all — it was already receiving a JSON number, and now receives a correct one instead of null.

The break is also not atomic on the server side, and the README now records the way around that: the leaf type only has to be named when a client declares a variable for it, so a consumer that passes the filter as an inline literal or hoists the variable to the whole [FileMetadataInput!] validates against the old schema and the new one alike — which makes a rolling deploy, the dev/prod schema skew, and a SHA rollback all non-events.

The widening lands on the input filter as well as the output field. Widening only the output would leave exactly the files it newly exposes unfilterable by size, which is half the bug.

Closes #83

Proposed changes

BigInt scalar and a scoped override mechanism

src/cfdb/api/gql/types.py defines BigInt alongside the existing ObjectIdScalar, following the same module-level @strawberry.scalar structure. One coercion function serves both serialize and parse_value, because the wire form is symmetric — the value that goes out is the value that comes back.

The output types are generated from the Pydantic models by runtime introspection, so a bare int maps to GraphQL Int. Rather than widening every int in the schema, _SCALAR_OVERRIDES maps (model, field name) to a replacement scalar and annotate() consults it first. Keying on the pair rather than the field name matters: ExtraFile.file_size is a same-named-shaped sibling on a different model that must stay Int, and so must totalCount, fileCount, and the page/pageSize arguments. _substitute_scalar preserves the Optional wrapper so Optional[int] becomes Optional[BigInt].

src/cfdb/api/gql/inputs.py is hand-written, so FileMetadataInput.size_in_bytes changes type directly.

Regenerate schema.graphql, and keep it that way

schema.graphql is a generated artifact that nothing in the repo regenerated or verified — the strawberry CLI is not an installed extra, so refreshing it meant reconstructing the print_schema incantation by hand, and nothing failed when the checked-in copy went stale. Add scripts/export_schema.py and a make schema target, plus a test asserting the file on disk matches what the live schema renders. That gap predates this change but is worth closing in the PR that first exercises it: the SDL is what clients codegen against, so a stale copy ships a wrong public contract silently.

Documentation

Add a Custom Scalars section to README.md recording the wire-representation decision and the three concrete client breakages, and note make schema in the Makefile targets table.

Test cases

#	Test Suite	Given	When	Then	Coverage Target
1	TestSizeInBytesScalar	The 6,262,125,716-byte ENCODE file from the issue	The files query selects sizeInBytes	Returns the exact size with no errors	Reported symptom
2	TestSizeInBytesScalar	A size at zero, an ordinary size, either side of the old Int ceiling, the JavaScript safe-integer maximum, or the 64-bit maximum	The files query selects sizeInBytes	Returns that exact value with no errors	Round trip across the declared range
3	TestSizeInBytesScalar	A file with no recorded size	The files query selects sizeInBytes	Returns null with no errors	Optional field passthrough
4	TestSizeInBytesScalar	Two files, the first holding a size beyond the 64-bit range	The files query selects sizeInBytes alongside other fields	Nulls only that file's sizeInBytes, reports the failure at that field's path, and leaves every other field and the sibling file intact	Page survival on an unrepresentable stored value
5	TestSizeInBytesScalar	One file above the old ceiling and one ordinary file	The files query filters on the large size as a query literal	Returns only the large file	Input filter, literal path
6	TestSizeInBytesScalar	The same two files	The files query filters through a [BigInt!] variable	Returns only the large file	Input filter, variable coercion path
7	TestSizeInBytesScalar	A query declaring its size-filter variable as [Int!], as a pre-BigInt client would	The query is executed	Fails validation naming the expected [BigInt!] type rather than truncating	Documented breaking change
8	TestSizeInBytesScalar	A filter literal that is a boolean, or one step beyond either end of the 64-bit range	The files query is executed	Rejects the query with a BigInt error rather than coercing	Scalar bounds and the bool trap
9	TestSizeInBytesScalar	A [BigInt!] variable carrying a numeric string, a fractional number, or an integral float	The files query is executed	Rejects the query with a BigInt error	Wire form stays an unambiguous JSON integer
10	TestSizeInBytesScalar	The published schema	FileMetadataType and FileMetadataInput are introspected	Both name sizeInBytes as BigInt	Widening reached both sides
11	TestSizeInBytesScalar	The published schema	FileList.totalCount, fileCount, and the page/pageSize arguments are introspected	Each is still Int	Override did not over-apply
11b	TestSizeInBytesScalar	ExtraFileType.fileSize, the other byte-size field	The type is introspected	Still Int, recorded as a deliberate deferral rather than a rule	Scope of #83
12	TestSizeInBytesScalar	An arbitrary size drawn from the signed 64-bit range	The files query selects sizeInBytes	Returns that exact value with no errors	Round trip as a property, not six examples
13	TestSizeInBytesScalar	An arbitrary integer beyond either end of the range	The files query selects sizeInBytes	Nulls the field with a BigInt error	Range bound as a property
14	test_schema.py	The checked-in schema.graphql	The SDL is rendered by the same function make schema writes with	The two are byte-identical	Generated-artifact drift
15	test_metadata_endpoint.py	The issue's file stored in a mongomock-backed database	The reproduction query is POSTed to /metadata	The raw response body carries the size as an unquoted JSON number with no errors	HTTP and JSON boundary
16	test_metadata_endpoint.py	The same file	A /metadata query filters on its size through a [BigInt!] variable	Matches that file	BSON encode and decode of the filter value

Review round 1

Seven independent principal-engineer reviewers. No reviewer found a correctness, data-integrity, concurrency, or security defect; both blocking findings were Python test-guide MUST violations. Remediated: the new HTTP test now uses the mocker fixture rather than unittest.mock; the declared range is pinned by a Hypothesis round-trip property plus a rejection property, and both inclusive bounds are named cases (the six original examples were all non-negative, so _INT64_MIN was never exercised in the accepting direction); _substitute_scalar raises on any annotation it cannot express faithfully, instead of silently flattening a list into a scalar; _coerce_big_int raises GraphQLError rather than ValueError, so a malformed filter value on an unauthenticated endpoint no longer writes a stack trace at ERROR; the drift test calls the generator it names; the ExtraFileType.fileSize assertion is split out and reworded as a deferral rather than a rule; and the _JS_SAFE_INTEGER_MAX comment no longer describes a guard the code does not implement.

Four findings were rejected, with reasons recorded in the review document: migrating both scalars off the deprecated strawberry.scalar class form (would have to move ObjectIdScalar and the Schema(...) call — scope this issue does not carry; five of the seven reviewers who raised it agreed no change was needed here); widening ExtraFile.file_size to BigInt (a different model, and #83 scopes this to sizeInBytes — the test that would otherwise have defended the bug is reworded instead); renaming or relocating tests/test_metadata_endpoint.py (tests/test_cors.py is the in-repo precedent for this exact shape, and with the datastore mocked it is not a true integration test either); and a registry-walking parity test over _SCALAR_OVERRIDES (would have to import a private dict and would restate the implementation; the new import-time TypeError covers the realistic failure).

Two adjacent defects are noted and left for follow-ups: size_in_bytes is stored as a BSON string for 4DN and HuBMAP because the C2M2 TSV path never narrows it, so exact-match size filtering is inert for those DCCs regardless of the scalar's width (the README now says so); and the field is indexed on the raw file collection but not on the materialized files collection the resolvers actually query.