concatenate() in SparqlAnythingConverter writes a \n between every pair of per-process outputs, unconditionally, “in case one does not end in one”. Every input has already passed the non-empty check, and Jena’s N-Triples writer terminates each triple with .\n, so the separator is never needed and always produces a blank line at each boundary.
The result is valid N-Triples, but it is not the same file cat would have produced: a real geonames-rdf run ships one blank line per chunk boundary, wc -l disagrees between the shell and LDE conversions, and geonames-rdf’s golden test has to strip blank lines before diffing the two outputs (grep -v '^$' in its test.sh), which is a consumer-side workaround for an upstream choice.
Suggested fix
Write the separator only when the previous file’s last byte is not \n – read the last byte with stat and a one-byte read at size - 1 – or drop it altogether given the writer guarantees a trailing newline. Then the concatenation is byte-identical to cat, and the grep in geonames-rdf can go.
Found while porting geonames-rdf (netwerk-digitaal-erfgoed/geonames-rdf#50), see #782.
concatenate()inSparqlAnythingConverterwrites a\nbetween every pair of per-process outputs, unconditionally, “in case one does not end in one”. Every input has already passed the non-empty check, and Jena’s N-Triples writer terminates each triple with.\n, so the separator is never needed and always produces a blank line at each boundary.The result is valid N-Triples, but it is not the same file
catwould have produced: a real geonames-rdf run ships one blank line per chunk boundary,wc -ldisagrees between the shell and LDE conversions, and geonames-rdf’s golden test has to strip blank lines before diffing the two outputs (grep -v '^$'in itstest.sh), which is a consumer-side workaround for an upstream choice.Suggested fix
Write the separator only when the previous file’s last byte is not
\n– read the last byte withstatand a one-bytereadatsize - 1– or drop it altogether given the writer guarantees a trailing newline. Then the concatenation is byte-identical tocat, and thegrepin geonames-rdf can go.Found while porting geonames-rdf (netwerk-digitaal-erfgoed/geonames-rdf#50), see #782.