ci(android): 一个包在每个 Android 版本上跑,从 10 开始 - #53
Merged
Conversation
A compatibility run installs the same package on one emulator per Android version, and the question it asks is not the one this harness was built for. The frame is there to be LOOKED AT: a scene that is legitimately dark and a renderer that died produce the same dark PNG, and no threshold separates them. So --no-frame-judge drops the pixel question entirely, leaving the gate at what can be decided without a human — did it install, did the host reach ready, did the boot record name an error. What replaces it is measurement. --metrics-out files what the launch cost, read while the app is still up because a dead process has no memory, no threads and no layer: startup from three independent clocks (am start -W, the host's own "ready in", and the platform's Displayed line), PSS with Graphics broken out because Dawn's Vulkan allocations never appear in the native heap, CPU from /proc diffed over a window, and frame intervals from SurfaceFlinger. SurfaceFlinger, not dumpsys gfxinfo: gfxinfo reports HWUI, and a NativeActivity drawing to its own ANativeWindow through Vulkan never touches HWUI, so its framestats for this app are empty. A field a version cannot answer for is null, never 0 — the point is comparing one Android release against another, and a zero reads as "free" rather than "not measurable here". notes says which ones those were.
native-smoke asks whether the templates can produce a launching app at all, and asks it on ONE platform version — API 34. That leaves the failure it exists for uncovered: a host that runs on the version CI happens to boot and dies on the ones users have. A user reported Android 10 crashing on launch; nothing here could have caught it, and nothing here can reproduce it without a device. So a package is built ONCE and installed on one emulator per Android release. One build, many versions: a difference between two rows cannot then be a difference between two builds. The version list is DISCOVERED from the runner's own system images rather than written down, so a new Android release is picked up by the next run instead of by whoever remembers to edit the file, and a floor that stops existing fails loudly instead of quietly shrinking the matrix. Preview platforms are named by codename and are skipped — a matrix that reds on an unreleased platform gets ignored within a week. The report goes into the PR because that is where it gets read. It states the renderer on every run rather than trusting anyone to remember it: a hosted runner has no GPU, Dawn is on the emulator's SwiftShader, so the frame times are a CPU rasteriser's and GPU utilisation does not exist to be read at all. Comparing versions against each other is still valid; reading any of it as device performance is not. The frames are published to an orphan branch because GitHub renders an image in a comment only from a URL, and an artifact has none.
Android 兼容性同一个包(space-shooter),在 8 个 Android 版本上各跑一台模拟器,每个版本装的是同一个构建产物。 运行时模板从这个分支构建 —— 测的是这里改的代码,不是已发布的版本。
截图 — 需要人工看这里没有任何像素判据。上面的表只回答了「装上了、起来了、没报错」;
这些数字的边界渲染器: 是软件渲染。 runner 没有 GPU,Dawn 跑在模拟器自带的 SwiftShader 上,
没测到的项,以及原因
|
The first run spent nineteen minutes building and then failed to emit anything, for two unrelated reasons worth separating. It failed because of an x86_64-only build. Every emulator in the matrix is x86_64, so arm64 looked like pure waste — but ANDROID_ABIS names both as what a template must carry and the emitter refuses an incomplete one. The compile succeeded and the emit step rejected it: "Emitted template is incomplete: lib/arm64-v8a/...". Building both is ~35 minutes, which makes the second reason the interesting one. It was slow because it built at all. A released template was already built by the release pipeline, and it has the stronger claim besides: it is the binary users installed, so a crash reproduced against it is THE crash rather than a rebuild that resembles it. Nothing in a compatibility run needs a fresh host unless the host is what changed. So the source is decided from what the PR touches — native/ or the dependency pins means build, anything else means download — and which way it went is reported next to the numbers, because "the shipped binary" and "a build of this branch" are different claims and a red row means something different under each. Separately: the Dawn cache the release pipeline writes is scoped to a tag ref, which no branch can read and no later tag can either. Every release rebuilds Dawn from scratch and so did this. Warming it from the default branch is its own change; this one just stops needing it in the common case.
A GitHub cache is readable from the ref that created it, that ref's descendants, and the default branch. The release pipeline runs on a TAG, so the caches it saved are readable by nothing: not by a branch, not by a PR, and not by the next release either, since every new tag is its own scope. Both are still sitting there — 937 MB for android under refs/tags/v0.37.0, 401 MB for ios, plus v0.36.0's — having never been read once. Every release has paid for a cold sixteen-minute-per-ABI Dawn build and then written the result somewhere it could never be fetched from. Writing it from master instead is the fix, because a default-branch cache is in scope for every run in the repository. The job restores first and builds only on a miss, which is what makes a weekly trigger cheap rather than absurd: GitHub evicts a cache untouched for seven days and a restore counts as touching it, so an ordinary week downloads the cache, finds it complete and stops. It rebuilds when the pins move, which is what the push trigger watches. `--build-deps` exists because a warm-up wants Dawn and nothing else. Building the host alongside it would only add ways for the cache job to fail for reasons that have nothing to do with what it is caching. It covers every ABI on android and both sysroot slices on ios — warming one and leaving the other cold would look like a hit and behave like a miss.
android-emulator-runner runs `script` through /usr/bin/sh. `set -o pipefail` is a bashism, dash rejects it on line one, and the step died before installing anything — on all eight versions at once. Every job reported failure and not one of them had tested an Android version, which is the worst shape a red matrix can take: it looked exactly like the compatibility break it was built to find. There are no pipes in that script, so `set -eu` loses nothing.
workflow_dispatch resolves only against the default branch, so this job cannot be run before it is merged — dispatching it on the branch returns 404. And once merged, a pins-only trigger would leave the cache cold until the pins next moved or the Monday cron came round, which is up to a week of releases still paying for a build nobody can reuse. Watching this file too costs one extra run, at the moment it lands, and it is the run that makes every later one cheap.
Switching to POSIX sh was necessary and not sufficient. The action does not run
`script` as a script at all — it runs one `sh -c` per line:
[command]/usr/bin/sh -c set -eu
[command]/usr/bin/sh -c mkdir -p build/compat
[command]/usr/bin/sh -c rc=0
[command]/usr/bin/sh -c for apk in apks/*.apk; do
/usr/bin/sh: 1: Syntax error: end of file unexpected (expecting "done")
So a loop cannot exist there, and nothing assigned on one line is visible on the
next — `rc=0` was three separate shells ago by the time anything read it.
native-smoke has been folding its invocation into a single line with `>-` all
along, which is why it works and this did not.
A file instead of a fold: shell syntax that cannot be run outside CI cannot be
tested outside CI either, and this one now resolves its own root so it runs from
any directory.
Both failures presented identically to the compatibility break this exists to
find — eight red versions, none of them having installed anything. Worth naming:
a matrix that cannot tell "the platform is broken" from "the harness is broken"
is not yet an instrument.
Also a concurrency group, because `paths` on a pull_request matches the PR's whole
diff rather than the push that arrived: every push re-ran all eight emulators, and
three pushes had queued twenty-four of them.
…e instrument
Six versions produced data and it was wrong in three ways, each visible only
because there were finally numbers to read.
The device identity was read after an early return taken when the process is
gone, so API 30 — the one version that crashed, the row a reader most needs
labelled — came out as "Android ? (API ?)". It is read first now: which phone
this is does not depend on whether the app survived on it.
`Graphics` was "—" on every row. dumpsys labels the same figure both ways in one
report ("Native Heap 1234" in the table, "Native Heap: 1234" in the App
Summary) and the pattern demanded whitespace, so it matched neither. That was the
one memory number worth having: Dawn's Vulkan allocations are counted there and
nowhere else, which is why 67-73 MB looked plausible and was not.
"首帧上屏" and "am start" were the same measurement. They agreed to the
millisecond on all six versions, because TotalTime from `am start -W` IS the
Displayed event. Two columns claimed two independent clocks where there is one;
both are still collected as a cross-check, one is shown.
And API 29 filed nothing at all — its emulator never booted and the job sat for
its full 45 minutes — so the table said "7 versions" and left no trace of the
eighth. An absent row reads as "that version does not exist", which is the one
thing a compatibility matrix must never imply by accident. Expected versions are
passed in now and a missing one gets a row saying so, counted separately from the
versions that ran and broke: the first is a result about Android, the second is a
result about this pipeline. The cap is 20 minutes, near what a real run costs.
Each crashing version cost seven minutes: the wait for `ready` sat out its full two-minute timeout, twice, on a process that had died in the first second. That is what pushed the slower emulators past the job cap — and the version it cost was reported as "no data" when it had in fact crashed, which is the wrong answer twice over. API 29 and API 30 traded places between two runs for no reason other than which one's emulator took longer to boot. `am start -W` has already waited for the launch to finish, so a missing pid is final rather than early. Checking it turns seven minutes into about twenty seconds and gives the whole matrix the headroom it was missing. Also: `Graphics` parses now and parses as 0, which invites reading it as "the renderer allocates nothing". It means the column counts gralloc/GPU-attributed memory and software rendering makes no such allocations — said in the report, next to the SwiftShader caveat it belongs to.
…e data API 29 and API 30 traded places across three runs, each time costing the matrix a version. Neither was our failure: older levels download their system image and then sometimes never reach sys.boot_completed at all. A matrix built to say "every version works" cannot drop one at random and still mean anything. So each version is attempted twice — but the retry is conditional on NOTHING having been filed, because a metrics file means the script ran and reached a verdict. Re-running a crash would burn an emulator to learn what is already known. That condition is also the distinction the old arrangement could not draw: "the app crashed" and "the emulator never came up" both failed the step, and only one of them is an answer. Which moves the verdict. The jobs MEASURE and the report JUDGES: a matrix job now fails only when it measured nothing, and whether the run is red comes from the table's own contents. Previously it came from `needs.compat.result`, which is the same conflation one level up — a version whose emulator died reddened the run identically to a version whose app died, and the PR comment was left as the only place the two could be told apart. The report exits non-zero for broken OR untested, after writing the comment, so a red check always arrives with the table that explains it.
The released host cannot be loaded on Android 10 or 11. Not slow, not broken on
screen — the dynamic linker refuses the library and the process is gone before a
line of our code runs:
java.lang.UnsatisfiedLinkError: dlopen failed:
cannot locate symbol "APerformanceHint_getManager"
The NDK emits a weak reference for a symbol newer than the build target and a
strong one otherwise, and compiles `__builtin_available` out in the second case.
Built at android-33, the guard around ADPF — correctly written for its API 31 —
was dead code, and the symbol became a load-time requirement on every device.
The guard was never wrong. The build target was.
So the target follows the manifest, and both are 29 rather than 26. 26 was
already fiction: loadFont calls AFontMatcher_create unguarded and that is API 29,
so Android 8 and 9 could only ever have crashed differently. Declaring what the
code actually supports costs nothing and stops promising three releases nobody
has tested. android-29 also makes the ADPF guard live again, which is what it
was written to be.
The compatibility matrix decides where its template comes from by what a PR
touches, and watched only native/. This change is entirely in build-tools —
the API level is a default there, not a value in native/ — so the rule would
have tested the old released binary and reported the fix as not working.
Lowering the build target to the manifest's floor made these guards live, and the
compiler immediately rejected them:
error: 'APerformanceHint_getManager' is unavailable: introduced in Android 33
note: 'APerformanceHint_getManager' has been explicitly marked unavailable here
.../sysroot/usr/include/android/performance_hint.h:128
Two things are true at once, and reading only the second is what put 31 here.
The header declares __INTRODUCED_IN(33); the symbol nevertheless resolves on 31
and 32 devices, which is why those two versions ran a host built against 33 whose
guards were dead code and whose reference was consequently strong. Runtime says
31, the header says 33, and the header is the authority at compile time — it
refuses a guard below its own introduced version regardless of what a device
happens to export.
So a hint session is given up on Android 12 and 12L in exchange for the app
loading at all on 10 and 11. That is the right trade twice over: ADPF is a
scheduling hint, and the alternative is a build target that silently disarms
every availability guard in the file.
Raising the guard to 33 was still wrong, and the compiler said so twice:
error: 'APerformanceHint_getManager' is unavailable: introduced in Android 33
note: explicitly marked unavailable here
`__builtin_available` does not unlock an API the NDK has marked unavailable. That
annotation is not "call me under a check" — it is "this build cannot see me",
and no guard version changes it. The only ways through are to raise the build
target, which is the bug being fixed, or to opt the build into weak API
references with ANDROID_WEAK_API_DEFS.
Neither belongs in a toolchain flag. A flag that silently stops applying restores
exactly this failure — a host that loads on the CI emulator and not on a user's
phone — and nothing in the source would show it. So the four symbols are looked
up by hand, against RTLD_DEFAULT, with opaque pointers so the header's annotated
typedefs are not needed either. A device that exports them gets a hint session; a
device that does not gets no session and a line in the log saying so.
The dependency now lives in the code that depends on it. That it compiles at all
is the proof it is honest about its own floor.
…fails The Android 10 fix works — API 29 reaches ready in 461 ms. API 30 then failed a different way, and the way it failed is the point: the game ran, drew, held 89.5% of a CPU and presented frames every 114 ms, and the verdict was "never reported ready". The record was there. The reader could not open it. It lives in the app's external files directory, which scoped storage put out of the shell user's reach. So root is taken once in prepare(), where an emulator image will grant it, rather than inside the two-second poll that reads the file. The rest of this is the instrument admitting what it could not distinguish. "No such file" — the app never got that far — and "Permission denied" — it did, and this cannot see it — are opposite conclusions, and both arrived as an empty string. That cost a run reported as a broken Android version. The error is kept and printed first in the diagnostics, with a listing of the directory beside it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
有玩家报 Android 10 闪退。我们没有 Android 10 设备,而 CI 只在 API 34 上测过 —— 也就是说这个失败既没人能复现,也没有任何门禁能挡住它。
这个 PR 建的是复现手段本身:一个包,构建一次,装到每个 Android 版本上各一台机器。
怎么工作
versions— 版本列表不写死。跑sdkmanager --list发现 runner 上真实存在的google_apis;x86_64镜像,取 ≥ API 29(Android 10)。新 Android 发布后下一次跑自动纳入;预览版(代号命名)自动排除;声明的下限镜像消失则硬失败,而不是静默缩小矩阵。apk— 从 HEAD 构建 template 并出包,只出一次(x86_64)。两行之间的差异因此不可能来自两次构建。共用 release 流水线的 Dawn 缓存键。compat— 矩阵按版本,fail-fast: false。装、启动、量。report— 汇总进 PR 评论,重跑原地编辑。判据:只判机器能判的
没有像素检测。 一个合法的暗场景和一个死掉的渲染器产出同一张暗 PNG,没有阈值能区分它们。所以门禁只剩三件不需要人的事:装上了、host 到了
ready、boot record 没报错。画面对不对靠截图,人工看。量了什么
ready/ 首帧上屏 /am start -W/ PSS(Graphics 单列)/ CPU% / 帧间隔中位与 p95。Graphics 单独列出来是因为 Dawn 的 Vulkan 分配不进 native heap —— 渲染器涨内存在堆数字里是看不见的。
帧时间来自
dumpsys SurfaceFlinger --latency,不是dumpsys gfxinfo:gfxinfo 报的是 HWUI,而 NativeActivity 经 Vulkan 直画ANativeWindow根本不碰 HWUI,它对这个 app 的 framestats 是空的。这些数字的边界
runner 没有 GPU,Dawn 跑在模拟器的 SwiftShader 上,所以帧时间是 CPU 光栅化的,GPU 占用和功耗在这里不存在。跨版本互相比较有意义,把任何一个数当成真机性能没有。 报告每次运行自己声明这一点,不指望谁记得。
预期:这次跑应该是红的
native/host/platform/android.cpp用了 ADPF 的APerformanceHint_*(API 33 引入),守卫写的是__builtin_available(android 31, *),而构建目标是-DANDROID_PLATFORM=android-33。NDK 的 availability 机制只在「目标 API < 符号引入版本」时才发 weak 引用,33 vs 33 → 强未定义引用,守卫被常量折叠成死代码。所以预期 API 29/30/31/32 全部
never reported ready(dlopen阶段就失败,跑不到我们任何代码),33 以上正常。也就是说 manifest 声明的minSdkVersion="26"是假的,实际下限是 33 —— 不只 Android 10,11 和 12 也一样挂。修复有意留在后面:先让这个矩阵把红色打出来,那才是没有设备的验证。
还悬着两件事
ci-android-shots,代价是 CI 会往仓库写图片。🤖 Generated with Claude Code