EPUB / HTML: Read aloud iOS support

[michalrentka]

To properly support read aloud in HTML/EPUB documents I'll need these methods:

• Highlight text that's being read. I should also be able to clear the highlight.
• Highlight / underline text that is being annotated by the "Annotate Sentence" function. Or do I just actually annotate in reader and edit the temporary annotation until it's confirmed? For PDFs I just create a highlight/underline overlay and I create the annotation once it's confirmed.

I think that's everything for reader. I'll leave implementation details to you, I should be able to provide anything you need.

[AbeJellinek]

Highlight text that's being read.

This is implemented: window._view.setReadAloudSpotlight(selector), where selector is an EPUB/snapshot selector as in the position field of an annotation, or null. I just made it navigate as well.

Highlight / underline text that is being annotated by the "Annotate Sentence" function. Or do I just actually annotate in reader and edit the temporary annotation until it's confirmed?

That's essentially what the desktop reader does, yes. It makes sense there because we treat the annotation as implicitly "confirmed" (created) unless you explicitly modify/delete it. If that works for you, it would be simplest to do the same on mobile!

...But don't you also need a way to get the text segments? Should I expose something that resolves to an array of ReadAloudSegment-like objects?

[michalrentka]

Highlight / underline text that is being annotated by the "Annotate Sentence" function. Or do I just actually annotate in reader and edit the temporary annotation until it's confirmed?

That's essentially what the desktop reader does, yes. It makes sense there because we treat the annotation as implicitly "confirmed" (created) unless you explicitly modify/delete it. If that works for you, it would be simplest to do the same on mobile!

On iOS the preview is temporary and the annotation is created once the overlay with controls is hidden, but I should be able to make some adjustments to make it work. I'll let you know if I have some issues with it.

...But don't you also need a way to get the text segments? Should I expose something that resolves to an array of ReadAloudSegment-like objects?

@mvasilak is working on SDT support for document worker, I should be able to get text from there. I guess if there is some current temporary solution that I could implement immediately, we can do that.

[michalrentka]

Highlight text that's being read.

This is implemented: window._view.setReadAloudSpotlight(selector), where selector is an EPUB/snapshot selector as in the position field of an annotation, or null. I just made it navigate as well.

@AbeJellinek so I have SDT data available, but how do I select either a paragraph or sentence via window._view.setReadAloudSpotlight(selector)? I don't really have the selector available. SDT gives me anchor points, but I'm not sure how to select a sentence (so part of paragraph) or even the whole paragraph.

[michalrentka]

Highlight / underline text that is being annotated by the "Annotate Sentence" function. Or do I just actually annotate in reader and edit the temporary annotation until it's confirmed?

That's essentially what the desktop reader does, yes. It makes sense there because we treat the annotation as implicitly "confirmed" (created) unless you explicitly modify/delete it. If that works for you, it would be simplest to do the same on mobile!

So yeah after a bit of thinking I have the same issue here as above. How do I create highlight / underline annotation from iOS side? Normally I just select the annotation tool and html / epub reader handles it by itself. But with read-aloud I'd have to somehow be able to create highlight / underline annotation for specific sentence / paragraph. Do we have any tools in the reader to get the custom selector? Or could I build it myself on iOS side?

[AbeJellinek]

How are you doing it for PDFs now?

I can add an interface to the SDT position mapper in view.js, but I would've thought you would need some kind of position mapper interface for PDF already! PDF mapping (from SDT positions to PDF positions) is, if anything, a bit more complex than EPUB/snapshot.

[AbeJellinek]

See if aa4a47e does the trick.

[michalrentka]

How are you doing it for PDFs now?

I can add an interface to the SDT position mapper in view.js, but I would've thought you would need some kind of position mapper interface for PDF already! PDF mapping (from SDT positions to PDF positions) is, if anything, a bit more complex than EPUB/snapshot.

Well for iOS this was even bigger issue, because of PSPDFKit. I was using full text extraction from our document-worker. Our extraction didn't match PSPDFKit exactly, so I had to have some mapping system between those two.
It the text of sentence/paragraph I want to find (highlight or annotate), normalizes it and then looks for it in PSDPFKit layer. I improve it by using some anchor point from document-worker (in full text I just used starting index of given text, SDT provides some anchor points and rects) and look for the closest match.

So I'm pretty much doing a little specialized string search in the PSPDFKit layer to know what to highlight or annotate.

I'm not sure whether aa4a47e would work for sentences? SDT gives back some bigger chunks of text (paragraphs, headers, tables, ...), but what if I want to highlight a sentence (so sub-set of some SDT pack)? Or when I'm using the quick annotate function and I want to extend to next sentence / paragraph.

For highlighting I guess it would make sense to provide SDT pack and then just a string which needs to be highlighted?
For annotations probably similar, but I'd also need functions to move/extend/shorten backward/forward to cover full functionality of quick annotation?

[AbeJellinek]

I could just expose the full segmentation interface! Given an SDT pack, view.js could generate all the segments to read at a given granularity (paragraph, sentence), add an annotation to a provided segment, and extend/contract an annotation by segment boundary. Would that handle everything the app needs?

String search doesn't seem ideal because strings aren't necessarily unique, and SDT normalizes and often won't match text layer text exactly.

[AbeJellinek]

For SDT -> PSPDFKit, the app should be able to implement something like https://github.com/zotero/reader/blob/a9e5b8f56ca0006b9696aba1678fd2c50a22bc63/src/common/sdt/pdf-position-mapper.ts. SDT positions can be converted to source-format positions without a lot of effort. But the reader already implements that, so it's easy enough for it to expose that to you for EPUB/snapshot (and PDF if the app ever switches to using the reader for that as well).

[michalrentka]

I could just expose the full segmentation interface! Given an SDT pack, view.js could generate all the segments to read at a given granularity (paragraph, sentence), add an annotation to a provided segment, and extend/contract an annotation by segment boundary. Would that handle everything the app needs?

That sounds good and I think it should cover everything

String search doesn't seem ideal because strings aren't necessarily unique, and SDT normalizes and often won't match text layer text exactly.

I had this implementation from full text where I didn't really any other way. When we added SDT support I kept it because at this point it worked mostly fine. It doesn't just look for the string, which isn't unique, but it looks for a string which is closes to given position, so the combination of string + position was pretty unique and worked well for me so far.

For SDT -> PSPDFKit, the app should be able to implement something like https://github.com/zotero/reader/blob/a9e5b8f56ca0006b9696aba1678fd2c50a22bc63/src/common/sdt/pdf-position-mapper.ts. SDT positions can be converted to source-format positions without a lot of effort. But the reader already implements that, so it's easy enough for it to expose that to you for EPUB/snapshot (and PDF if the app ever switches to using the reader for that as well).

I don't know much about our SDT implementation, I just checked some document-worker output and kind of went with it. Do we have any documentation on it? Because from what I've seen the SDT contains paragraphs, but we're reading sentence-by-sentence, so I detect sentences manually in given paragraph. I didn't know we can get sentence granularity from SDTPack. So I may be doing other things wrong as well. I'll check the mapper and try to replicate on iOS.

[AbeJellinek]

See if 34aada5 does it.

[michalrentka]

@AbeJellinek what about the previously mentioned window._view.setReadAloudSpotlight(selector) to highlight text that's being read? Can I use SDT segments for it as well? Also what's the difference between createAnnotationFromSDT and setReadAloudAnnotation?

Actually, I was able to resolve it all with claude. We had some issues along the way, but we were able to resolve them mostly by using some internal/private calls. You might want to take a look and probably provide a more direct way for clients to get that. If you don't mind I'll drop some comments from claude as well. I created a PR with index.ios.js fixes: #233.

Possible bugs:
View.setReadAloudSpotlight(selector) didn't seem to follow the spotlight to new page. AI comment:
View.setReadAloudSpotlight(selector) calls this._view.navigate(selector, …) with a raw selector, but navigate only acts on { position } / { annotationID } locations — a raw selector is a silent no-op. So the spotlight highlight moves but the view never follows. The reader's own read-aloud follow (dom/common/lib/read-aloud.ts) correctly uses this._view.navigateToSelector(selector, …).
Fix: make setReadAloudSpotlight navigate via navigateToSelector(selector, {...}) (or wrap { position: selector }). We currently work around it by calling _view.navigateToSelector ourselves.

Missing APIs:
To start reading at currently opened position in epub/html, we needed to get current in-view SDT segment. Claude found getVisibleBlockIndex(structure) exists, but it's not directly available to us in View. If you could make that available in View I wouldn't have to call window._view._view.getVisibleBlockIndex?.(sdt.structure). To be able to get that we also have to call private window._view._loadSDT();.

While using the read aloud quick annotation tool, I don't actually create annotations immediately in PDF reader. I only preview them while the annotation popup is visible and create them (store to database and trigger sync) once the popup is gone and the "highlight session" is confirmed. So to match this in html/epub reader in index.ios.js we have to store the current "preview" annotation and ids. While doing that claude complained about this:
"creation ignores a caller-supplied id. In View.setReadAloudAnnotation, an id only resizes an existing annotation; on create, annotationManager.addAnnotation generates its own id. So the app can't preset a stable id and must learn it from the async return — which forces us to serialize calls to avoid a create race (rapid move/extend created duplicate previews)."
A way to "create" annotation without actually posting the onSaveAnnotations would be nice. I could change both to saving annotations immediately, but imo it would create unnecessary database writes and syncs if the user just moves/expands the annotation preview a bunch. Having a temporary preview until the annotation editing is finished seemed better.