SDL3 File Dialogs

[nspitko]

This PR adds support for File/Folder open/save dialogs via SDL3. These are platform agnostic async calls that should function consistently across all supported targets, and use modern APIs that should match the user's OS.

The implementation here is a bit icky in a few spots as we need to jump through some hoops to get strings in and out of HL. I chose to hide as much of this complexity in sdl.hx as possible, but arguments can be made for shifting it up or down the call tree. I don't think it matters much where this code lives as long as it's not user facing, so the important bit is the sdl.hx interface which exposes FileDialogCallback and DialogFileFilter.

Some notes:

• SDL will always trigger the callback, even if the user cancels or the function fails due to other reasons (eg, bad filter data, etc). With this contract it's safe to leave cleanup to FileDialogCallback in sdl.c and not worry about leaking memory or gc roots.
• Zeta assisted on the gc root code, but it's worth getting a second set of eyes on the implementation since I blindly ran with his suggestion; I don't fully understand the entire memory ownership model of the gc and it may not make sense the way I implemented it. It appears to work in my testing.
• I really wish we could pass strings through HL without the intermediate hl.bytes stage, but I couldn't find a way to do this.

Use cases

• Currently, my in-house tools for heaps heavily lean on an older version of this code for save/load/selection on linux.
• Hide likely also needs this if we ever want it to work reasonably well outside of windows.

Why not use UI.hdll?

Currently there are no other options for file dialogs in HL outside of windows. ui.hdll has some vague hints towards being portable but there's no work currently done on other platforms, and maintaining versions across all platforms is a non-trivial endeavor, especially on linux where there are a large number of windowing systems with subtly different APIs. SDL3 has already done this work for us, and is likely in use for window creation on non-windows platforms anyway.

[yuxiaomao]

Seems that we should not free filters until callback?

https://wiki.libsdl.org/SDL3/SDL_ShowOpenFileDialog
filters: a list of filters, may be NULL. Not all platforms support this option, and platforms that do support it may allow the user to ignore the filters. If non-NULL, it must remain valid at least until the callback is invoked.

[yuxiaomao]

If we need to keep the filters until callback, we might need to do add_root on a struct / context that contains both callback and filters?

[nspitko]

I added a struct. I THINK we only need to add_root on the closure_store here, the GC shouldn't know/care about mallocs in hlsdl, but happy to be corrected if there's a need to root the whole struct.

[yuxiaomao]

GC should not know about mallocs, but it seems that Filters' name/pattern still reference to hl.Bytes. We might need to avoid GC on them (either copy bytes with malloc; or make the struct gc managed pointer + add raw varray into it)

[nspitko]

Did a quick cleanup of this and the memory ownership a bit, strdup should work fine here but I can malloc instead if you prefer.

There's still an edge case where passing null as a filter var crashes hlsdl, need to handle that somewhere but I'll probably just throw in haxe before sending bad data unless we have a general policy that hl libs should be hardened at the C side.

[yuxiaomao]

strdup is good.
I'm not sure what do you reference by filter var.
hl libs can assume that data is validated by haxe, and haxe API can also assume that the call argument nullable or not is respected (if the function signature is explicit). So just let null filter name/pattern throw in haxe when performing .toUtf8 is acceptable from my point of view.

[yuxiaomao]

I'll certainly complains about the formatting in the files x) But I can probably fix that by my own if it's the only remaining things

[nspitko]

Old habits die hard... I thought I cleaned them all up but guess not. Should be fixed now (unless there are more)

[yuxiaomao]

When I'm trying to call with a filter on Windows, it return immediatly an empty array without open any window (works normally if no filters specifield). From C side I saw that filelist is NULL so error occurs, checking the error string it says

Invalid dialog file filters: Invalid character in pattern (Only [a-zA-Z0-9_.-] allowed, or a single *

I think it would be good if we can at least give some information to the haxe side about the error, instead of returning empty list here (return NULL or throw?).

sdl.Sdl.showOpenFileDialog(function(arr) {
		trace("Open " + arr);
	}, null, [{ name : "hxml", pattern : "*.hxml" }]
);

[nspitko]

This one's kinda tricky since we can't validate filters before the callback (the SDL internal validate_filters function is not exposed to us), so showOpenFileDialog can only ever be void.

What I've done is to pass the NULL state into haxe, and throw getError() in that case. It works here, but I'm not sure there's any way to catch these since they fire in a thread, so I'm not super happy with the solution. The other option is we just pass null through and leave it to the user to read the API, which is more "correct" but probably not intuitive.

[yuxiaomao]

I don't think it's only about filters, errors can come from elsewhere, so it's good that at least we return NULL from C side.
Yes I agree with you it's not trivial. Another related problem is if we throw inside the callback (either inside SDL.hx or user code), hl_remove_root and free are never called.