[BUG] SINAN Dengue data differs between direct DATASUS FTP and PySUS for 2020

[AmmarAhmedAlvi]

Summary

I am seeing differences between SINAN Dengue data downloaded directly from the DATASUS FTP server and the corresponding datasets returned by PySUS.

I tested 2020 as a minimal example.

The direct DATASUS file is:

• DENGBR20.dbc

from:

/dissemin/publicos/SINAN/DADOS/FINAIS

The corresponding PySUS datasets were retrieved with:

from pysus import sinan

df = sinan(
    disease="deng",
    year=year,
    as_dataframe=True,
)

I expected the two methods to represent the same logical final SINAN/Dengue datasets for each year, modulo expected format/dtype transformations.

However, the resulting dataset differ in row counts and/or row contents.

Comparison

I compared both the complete datasets and confirmed Dengue cases, where:

confirmed_codes = {"10", "11", "12"}

df["CLASSI_FIN"] = (
    df["CLASSI_FIN"]
    .astype("string")
    .str.strip()
    .str.replace(r"\.0$", "", regex=True)
)

confirmed = df[
    df["CLASSI_FIN"].isin(confirmed_codes)
]

Results

================================================================================
FTP vs PySUS SUMMARY
================================================================================
 year  ftp_raw_rows  pysus_raw_rows  raw_row_count_delta_pysus_minus_ftp  ftp_confirmed_rows  pysus_confirmed_rows  confirmed_row_count_delta_pysus_minus_ftp  
 2020        975842         1495117                               519275              813459                793010                                     -20449

The row comparison normalizes simple representation differences such as 10 versus 10.0 before hashing common columns.

Direct DATASUS reproduction

from ftplib import FTP
import pyreaddbc

FTP_HOST = "ftp.datasus.gov.br"
FTP_DIR = "/dissemin/publicos/SINAN/DADOS/FINAIS"

ftp = FTP(FTP_HOST)
ftp.login(user="anonymous", passwd="")
ftp.cwd(FTP_DIR)

year = 2020
filename = f"DENGBR{year % 100:02d}.dbc"

with open(filename, "wb") as f:
    ftp.retrbinary(f"RETR {filename}", f.write)

df_ftp = pyreaddbc.read_dbc(
    filename,
    encoding="iso-8859-1",
)

ftp.quit()

print(df_ftp.shape)

For reproducibility I also calculated the SHA-256 checksum of each downloaded DBC:

DENGBR20.dbc: 52baabbad73986d078866d1f3b538eb7f58f025dc354ae187720974a137fca02

PySUS reproduction

from pysus import sinan

for year in [2020]:
    df_pysus = sinan(
        disease="deng",
        year=year,
        as_dataframe=True,
    )

    print(year, df_pysus.shape)

CLASSI_FIN comparison

The distributions of CLASSI_FIN also show the following differences:

CLASSI_FIN comparison — 2020
               ftp   pysus  delta_pysus_minus_ftp
CLASSI_FIN                                       
10          802804  782498                 -20306
8           160599  161882                   1283
11            9720    9578                   -142
<NA>          1561    1571                     10
12             935     934                     -1
0              223       0                   -223
5                0  538654                 538654

Expected behavior

I would expect the PySUS dataset and the current DATASUS FINAIS FTP file for the same SINAN disease/year to contain the same logical records, unless the PySUS/DuckLake dataset intentionally represents a different snapshot or applies additional transformations/filtering.

If differences are expected, it would be helpful for the documentation or dataset metadata to indicate which upstream snapshot/version was used.

Additional Issues noted across most years

1. While the datasets from ftp and pysus are consistent in most cases, sometimes a value is never available. For example the DT_DIGITA column is consistently missing from 2014 onwards in pysus with all values being NA, whereas this column is found with date values when pulling the data from ftp.
2. Sometimes the dtypes for columns can be different. For example NU_IDADE_N is int32 format in ftp but object in pysus.

Questions

Could you please clarify:

1. Is the PySUS DuckLake/S3 artifact for a SINAN year expected to correspond exactly to the current file in /SINAN/DADOS/FINAIS?
2. Are any records or columns filtered, deduplicated, transformed, or otherwise changed when the FTP data is converted to the PySUS Parquet/DuckLake representation?
3. Can the DuckLake artifact for a given SINAN file originate from a different DATASUS snapshot than the file currently available through FTP?
4. Is there metadata available that identifies the upstream FTP file checksum or modification date used to produce each DuckLake artifact?
5. Is pysus frequently updated with the data on source ftp? If yes then at what frequency?

Thank you for looking through this!

[luabida]

Hi @AmmarAhmedAlvi, thanks for the detailed bug report. Unfortunately, your description matches the expected behavior of the package in the current state. I hadn't realized the files from the two incoming sources were this different, so in the package as it is now, these files are threaten as mirrors, and when the parameter origin= is None (default), it chooses the most updated file based on its metadata (mostly last update date, but also file size, rows count, and other metadata specs). I've talked with @fccoelho and we are going to mitigate this behavior by enforcing the user to verbosely import the module of the origin they would like to download from, instead of "guessing" which file to download. Here's a breakdown of the problem:

Root cause. pysus.sinan(...) with no origin queries the DuckLake catalog's default mirror for each logical file, and for SINAN that default is the DadosGov open-data snapshot (built from the ckan.saude.gov.br CSV), not the conversion of the DATASUS FTP FINAIS .dbc. The two upstream sources are different datasets, so they don't have to agree. This is not a conversion/transformation artifact in PySUS — the FTP-origin artifact is a faithful conversion of the DBC.

To get exactly the file in /dissemin/publicos/SINAN/DADOS/FINAIS, request the FTP origin explicitly:

from pysus import sinan

df = sinan(
    disease="deng",
    year=2020,
    origin="FTP",          # "FTP" | "DadosGov" | "DuckLake" | "Saude"
    as_dataframe=True,
)

I checked this against the live artifacts for DENG 2020 — the row counts and CLASSI_FIN distributions match your two columns exactly:

	origin="FTP" (FINAIS DBC)	DadosGov (default)
rows	975,842	1,495,117
CLASSI_FIN 10	802,804	782,498
CLASSI_FIN 11	9,720	9,578
CLASSI_FIN 12	935	934
CLASSI_FIN 8	160,599	161,882
CLASSI_FIN 0 / 5	223 / —	— / 538,654
<NA>	1,561	1,571
DT_DIGITA (2020)	964,215 values	all NA (as you saw, empty from 2014+)

The 538,654-row "5" block and missing DT_DIGITA are intrinsic to the DadosGov CSV snapshot, which also explains the dtype differences (NU_IDADE_N int32/object etc.) — CSV string inference.

To your numbered questions:

1. The artifact corresponds exactly to the current FINAIS file only when origin="FTP". The default serves the DadosGov snapshot.
2. No — rows/columns are not filtered or post-transformed on conversion; the difference is upstream provenance.
3. Yes. The catalog treats ftp and dadosgov as independent origins, each with its own snapshot and metadata.
4. Yes — the catalog stores per-origin origin_path, origin_modified, origin_size, and (for FTP) the parquet sha256. For DENG 2020: FTP origin_path=/dissemin/publicos/SINAN/DADOS/FINAIS/DENGBR20.dbc, origin_modified=2026-04-10 12:09, origin_size=47,733,119. We can surface these through the API if useful.
5. SINAN artifacts are rerun as files land on the FTP server; each origin records its own modification date, which is the authoritative answer for "which snapshot am I reading".

Mitigation: this will be further addressed in an upcoming PR that forces an explicit/verbose origin selection on the user-facing call, so a call can no longer silently resolve to a different snapshot than the caller expects.