feat: improve date parsing

[ThibaudDauce]

• Replace fix: cast dd/mm/yyyy dates day-first (was silently swapping day and month) #268

• add date_format to profile to have a faster cast in hydra

• by finding a unique date format for the column it prevent having US dates (month first) vs FR dates (day first) mixed

• date_fr is removed, date reads the same values and gives them a real date python_type. Stored analyses naming it still validate, nothing to regenerate.

• we do not accept mixed separators from now on 2026/04-14 for example is no longer a date

• hydra stats (require perf: cast dates with the format inferred by csv-detective hydra#470):

Fichier	inspection			csv to db			total
	main	branche	Δ	main	branche	Δ	main	branche	Δ
ValeursFoncieres-2024.txt	44,2	44,3	+0 %	50,1	31,6	−37 %	94,3	75,9	−20 %
ValeursFoncieres-2025-S1.txt	16,4	16,5	+0 %	20,2	12,7	−37 %	36,7	29,2	−20 %
MN_07 (sans date)	30,5	28,9	−5 %	9,9	10,1	+2 %	40,3	39,0	−3 %
joconde.csv	106,8	104,2	−2 %	22,6	18,5	−18 %	129,4	122,8	−5 %
irve.csv	7,9	6,7	−15 %	12,6	5,7	−55 %	20,6	12,4	−40 %

Some analyses will be invalidated and replayed after the merge. Values dateutil used to read as dates and that we don't detect anymore, on the files I have locally:

Fichier	Colonne	Valeurs	Exemple
joconde.csv	Mesures	693	D. 2.9 ; E. 0.6
joconde.csv	Numero_inventaire	270	M 1089
joconde.csv	Millesime_de_creation	76	134,138
irve.csv	consolidated_latitude	1700	48.825613
irve.csv	telephone_operateur	12	0680428426
MN_07	AAAAMMJJHHMN	2000	202501010000

Everything but the last one is noise dateutil was finding in phone numbers, latitudes and museum inventory numbers. AAAAMMJJHHMN is a real one: it's now datetime_naive where it was date, so the column type changes in the db (it holds hours and minutes that the date type was dropping).

[estellebertrand]

A necessary change! The logic looks good, but I need to take more time to review all changes more carefully :)

[bolinocroustibat]

LGTM, with two remarks

[estellebertrand]

LGTM - still reading it all to ensure no regression

[estellebertrand]

Here, unsure why do we drop this format ? Because not an known separate format and this can be included in date.py just as a type of date format like any other ?
[the comment was for the deleted date_fr file, not appearing on the right one here + this has been managed since by commit 48219e8]

[estellebertrand]

@ThibaudDauce maybe to add to your PR description : I found the answer to my question about how strict we want to be with mixed separators in a date format :

your new function

def separator_of(val: str) -> str | None:
    """The single separator the value uses, "" if it uses none, None if it mixes several."""
    found = {
        char
        for index, char in enumerate(val)
        if char in SEPARATORS
        # the full stop of an abbreviated month is not a separator ("15 janv. 1985")
        and not (char == "." and index and val[index - 1].isalpha())
    }
    if len(found) > 1:
        return None
    return found.pop() if found else ""

is a breaking change if I understand well : before, if we consider only - and / for our example, a date could be DD-MM/YYYY, now it can only be DD-MM-YYYY or DD/MM/YYYY.

The thing is, I don't know why it was accepted before and that might be because some datasets were using this shape. I believe this is a bad practice and we don't want to encourage this kind of format, but we need to keep in mind some datasets could loose their date columns and associated metadata to string only.

[estellebertrand]

Also, one thing, not introduced here but we might want to include it (or in a second PR) :

in parsing/columns.py, the function test_parquet_cols doesn't include date as format to test for stringcolumns in parquet. That means that if a user declares a date column as a string column, we never try to detect this format.

remaining_tests_per_col = {
        col: {
            fmt_label
            for fmt_label, fmt in formats.items()
            # keeping formats that have the valid python type
            if fmt.python_type == pytype
            # except if the column label doesn't fit
            and fmt_label not in mandatory_label_skip.get(col, set())
            # we already know pure types are valid, only formats remain
            and fmt_label != pytype
        }
        for col, pytype in columns.items()
    }

We might want to add this exception so it performs as well on parquet and csv ?

On another hand, we could say that parquet files are supposed to be typed and if the user didn't declare cleanly a type that was available amongst parquet ones (like a date or timestamp, time by opposition to a custom code_commune format we added), we shouldn't enforce it (?)

[ThibaudDauce]

is a breaking change if I understand well : before, if we consider only - and / for our example, a date could be DD-MM/YYYY, now it can only be DD-MM-YYYY or DD/MM/YYYY.

This one is not new: date already has python_type = "date" on main, so a string
column in a parquet file was never tested for it.

And it's not only about dates, the fmt.python_type == pytype filter also drops
datetime_naive, datetime_aware, int, float… from string columns. Adding an
exception for date only would be a bit weird?

So I'm more with your second option: parquet is typed, if the user declared a string we
keep a string. If we want to change that it's the whole parquet policy and not just
dates, so another PR?

[ThibaudDauce]

@ThibaudDauce maybe to add to your PR description : I found the answer to my question about how strict we want to be with mixed separators in a date format :

your new function

def separator_of(val: str) -> str | None:
    """The single separator the value uses, "" if it uses none, None if it mixes several."""
    found = {
        char
        for index, char in enumerate(val)
        if char in SEPARATORS
        # the full stop of an abbreviated month is not a separator ("15 janv. 1985")
        and not (char == "." and index and val[index - 1].isalpha())
    }
    if len(found) > 1:
        return None
    return found.pop() if found else ""

is a breaking change if I understand well : before, if we consider only - and / for our example, a date could be DD-MM/YYYY, now it can only be DD-MM-YYYY or DD/MM/YYYY.

The thing is, I don't know why it was accepted before and that might be because some datasets were using this shape. I believe this is a bad practice and we don't want to encourage this kind of format, but we need to keep in mind some datasets could loose their date columns and associated metadata to string only.

Yes I think it's more a bug fix, I'm not sure we wanted to have mixed separators in dates, but the previous code allowed this… But it seems really weird to accept that…

[estellebertrand]

Ok so recap :

Parquet string cols not being tested for date format
"So I'm more with your second option: parquet is typed, if the user declared a string we
keep a string. If we want to change that it's the whole parquet policy and not just
dates, so another PR?" => Yes let's keep that for another PR if we think it should be updated.

Mixed separators not allowed anymore in date columns
"Yes I think it's more a bug fix, I'm not sure we wanted to have mixed separators in dates, but the previous code allowed this… But it seems really weird to accept that…" => Agreed though I wouldn't say "bug fix" as it was clearly implemented in the test (so it was known and accepted it would accept mixed separators). Let's just add that to a release note as a breaking change ? Seems something we don't want to forget and be able to point to when and why this change, in case users ask why some columns are not dates anymore

[bolinocroustibat]

@ThibaudDauce @estellebertrand I'm getting a bit lost with this PR to review. Would that make sense to break it into smaller, more readable PRs, or would that be overcomplicated?