fix: implement more robust retries for Watch

[miparnisari]

Description

Similar to authzed/authzed-java#137, I'm modifying the implementation of the zed relationships watch command so that

1. we request checkpoints to keep the stream alive in case spicedb is behind a proxy that closes idle connection
2. we add manual retries in case the error code is UNAVAILABLE

Testing

Locally against a locally running instance that is behind a Contour/Envoy reverse proxy.

1. Tested the scenario of Watch API not returning anything (being idle) for at least 30 seconds:

before

 $ zed watch --log-level trace
Command "watch" is deprecated, deprecated; please use `zed watch relationships` instead
7:07PM DBG configured logging async=false format=auto log_level=trace provider=zerolog
starting watch stream over types [] and revision 
7:07PM TRC context={"APIToken":"......","CACert":null,"Endpoint":"maria-ps-6fc3f6.cloud.authzed.dev:443","Insecure":false,"Name":"us-east-1","NoVerifyCA":false} context-override-via-cli=false

(nothing happens)

7:08PM ERR retrying gRPC call error="rpc error: code = Unavailable desc = stream timeout" attempt=1

with this PR

./zed watch --log-level trace           
Command "watch" is deprecated, please use `zed relationships watch` instead
4:06PM DBG configured logging async=false format=auto log_level=trace provider=zerolog
starting watch stream over types [] and revision 
4:06PM TRC context={"APIToken":".....","CACert":null,"Endpoint":"maria-ps-6fc3f6.cloud.authzed.dev:443","Insecure":false,"Name":"us-east-1","NoVerifyCA":false} context-override-via-cli=false
CHECKPOINT REACHED
CHECKPOINT REACHED
CHECKPOINT REACHED
CHECKPOINT REACHED
CHECKPOINT REACHED
CHECKPOINT REACHED
CHECKPOINT REACHED
CHECKPOINT REACHED
CHECKPOINT REACHED
CHECKPOINT REACHED
CHECKPOINT REACHED
CHECKPOINT REACHED
CHECKPOINT REACHED

3. Tested the scenario of Watch API continually returning data

References

[tstirrat15]

See comments, otherwise LGTM

[tstirrat15]

What is this?

[miparnisari]

This is establishing a connection to my custom-made Watch server.

I see that other tests do it differently:

// Set up client
	ctx := t.Context()
	srv := zedtesting.NewTestServer(ctx, t)
	go func() {
		require.NoError(srv.Run(ctx))
	}()
	conn, err := srv.GRPCDialContext(ctx)
	require.NoError(err)

	originalClient := client.NewClient
	defer func() {
		client.NewClient = originalClient
	}()

	client.NewClient = zedtesting.ClientFromConn(conn)

	c, err := zedtesting.ClientFromConn(conn)(cmd)
	require.NoError(err)

but in my case i don't want to rely on a real SpiceDB server; I want to have full control on the response given by Watch

[tstirrat15]

Ah yeah in the tests that I was working on I used a fake client and then supplied responses directly through it. Is there a reason to prefer one approach to the other in this case?

[miparnisari]

Updated

[tstirrat15]

Nice, LGTM