Streaming Data With FS2

[SarpongAbasimi]

Hey 👋🏿

I am trying to stream data from the twitter api https://api.twitter.com/2/tweets/sample/stream
Ref - https://developer.twitter.com/en/docs/twitter-api/tweets/sampled-stream/api-reference/get-tweets-sample-stream

For the client I have this

with sample tweet being

final case class Id(id: Int)                   extends AnyVal  
final case class Text(text: String)            extends AnyVal
final case class Data(id: Id, text: Text)
final case class SampleTweet(data: Data) extends Tweet

but when I make a request, I get this error expected eof got '{"data...' (line 2, column 1)

There isn't a comma , after each data. (I think that is the reason for the error)

This makes me think that I am probably not handling the stream well. Please is there a better way to handle streams in fs2?

[kubukoz]

Usually I'd recommend showing more code, including what imports you have and what codec is used for the JSON decoding.

First of all, you're consuming a streaming endpoint, which will keep producing input (I assume) forever, but even if it's short-lived you shouldn't use .expect. Use .stream to get a Stream[F, Response[F]], then take that response's body (stream of bytes) and decode it with a stream-aware decoder (I'm not sure http4s itself provides that, but https://github.com/circe/circe-fs2 does).

Roughly something like this:

response.body
  .covary[IO]
  .through(io.circe.fs2.byteStreamParser)
  .evalMap(_.as[SampleTweet].liftTo[IO])
  .compile
  .toList
  .unsafeRunSync()

btw. for syntactic convenience check out fromEither -> .liftTo[F], Authorization(Credentials.Token(AuthScheme.Bearer, ...)) :)

[SarpongAbasimi]

Hey @kubukoz thanks for the help, I will give it a go.
Also I am guessing that by using circe-f2s I will not need an Encoder/ Decoder for Stream[F, Response[F]]?
Lastly, it is possible to just handle the stream without using circe-fs2?

[kubukoz]

You only need a decoder for what's in the stream to do this. I don't know if it can be done without circe-fs2, I'd suggest asking somewhere on http4s.

[SarpongAbasimi]

sure, thank you 😊