libobs: Ensure buffer size for utf-16 converted filepaths

[Dankirk]

Description

File paths from Windows W apis get stored internally in UTF-8, but there are many cases the internal buffer is too small to contain a valid path from W api. For example char file[MAX_PATH] (260) is too small. In other cases 512 bytes is assumed long enough, but many OSs/filesystems support paths longer than that.

This update:

• Prefers dynamic heap allocation for file paths over static stack size, because their max size varies widely by filesystem/OS

  • Stack allocation is still used when it might be a performance issue, such as scanning directories
  • Defines a constant for static allocation of utf-8 paths (Not intended as actual filesystem/OS limit)

• Changed many A api calls to W variants to support longer paths. (255 utf-16 code units vs 255 bytes)

  • This also ensures encoding is correct, since A api will return active code page encoded strings, which are not utf-8 by default.

• Minor bugfixes

  • Fix overflow in os_stat() when used with long filename (incorrectly used sizeof() instead _countof())
  • Fix off-by-one length limit check in nvafx-load and nvvfx-load sdk path
  • Fix BPtr re-assignment leaking the pointer (re-assignment was not used previously)

• Minor changes

  • Change GetAppConfigPathPtr and os_get_config_path_ptr to format the path identically to GetAppConfigPath and os_get_config_path. Previously there was difference with null/empty string.
  • os_dirent uses a char pointer instead of an array to hide the platform specific allocation size of filename.

Motivation and Context

This is mainly findings while working on other things with encodings, but is also somewhat related to recent change about using absolute paths for DLLs, which could trigger one of these length checks.

How Has This Been Tested?

Tried happy paths using a scene with all source types active and restarting the app.
EDIT: Tested long paths as detailed in comment below.

Types of changes

• Bug fix (non-breaking change which fixes an issue)

• Tweak (non-breaking change to improve existing functionality)

Checklist:

• I have read the contributing document.
• My code has been run through clang-format.
• My code follows the project's style guidelines
• My code is not on the master branch.
• My code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[Dankirk]

Tested this on Windows with both long directory names and long file names. For naming I used 180 "€"s that when converted from wchar_t (utf-16) to char (utf-8) expand to triple size of 540 chars, well over the MAX_PATH of 260 for ANSI apis and some 512 byte allocations in the app.

List of things that couldn't, but can now handle such filenames:

• slideshow files in directory
• vlc playlist files in directory
• recording filename formatting

Notably the GetConfigPath and os_get_config_path return relative paths, which means many parts of OBS can work while OBS resides in a path with long name. I still changed these to use dynamic allocation instead of static 512 bytes.

Some of the configs (and others) also end up being turned into absolute paths. Absolute paths had a limit of 260 bytes. At least these can now handle longer paths:

• browser (config/cookies storage)
• game capture injects/hooks
• script dirs
• OBS already-running-check (portable mode)

The nvidia video and audio SDKs work (Switched to W apis). Though they currently work only with the default install directory, which shouldn't(?) really have any non-ascii or long paths. Could probably check if the SDKs have a registery entry for the install directory, in case they weren't installed in default directory. Out of scope for this though.

Freetype2 font loading from Windows fonts directory works (Switched to W apis). Could probably test it with with font filenames that aren't ascii.

There is also parts I haven't really tested, but they follow the pattern as everywhere else:

• OBS updater
• crash handler logging modules with long names.
• importers auto-search / folder scanning
• recursive missing files check

For the future
Long filename handling could be probably be further improved in another PR by making long and relative paths use the Windows long path notation \\?\ with W apis. This could be done in os_wfopen(), which is ultimately what ends up being called for reading files, unless it's Qt native.