Skip to content

fix: make feature flag local tests deterministic#193

Merged
marandaneto merged 4 commits into
mainfrom
fix/deterministic-feature-flag-local-tests
Jul 1, 2026
Merged

fix: make feature flag local tests deterministic#193
marandaneto merged 4 commits into
mainfrom
fix/deterministic-feature-flag-local-tests

Conversation

@marandaneto

@marandaneto marandaneto commented Jul 1, 2026

Copy link
Copy Markdown
Member

💡 Motivation and Context

Feature flag local evaluation assertions were brittle in CI because they compared raw JSON payload strings. Generated UUIDs and payload serialization details can differ by environment, making failures appear in PHP 8.2/8.3/8.4 even when behavior is correct.

Semgrep also flagged the existing secrets: inherit usage in the feature flags project board workflow. This mirrors PostHog/posthog-js#4034: the reusable workflow currently requires inherited secrets, so the finding is suppressed inline with context.

💚 How did you test it?

  • ./vendor/bin/phpcs --standard=phpcs.xml test/FeatureFlagLocalEvaluationTest.php
  • ./vendor/bin/phpunit --bootstrap vendor/autoload.php --configuration phpunit.xml --filter FeatureFlagLocalEvaluationTest --no-coverage
  • ./vendor/bin/phpunit --bootstrap vendor/autoload.php --configuration phpunit.xml --no-coverage
  • docker run ... semgrep --config /rules/.semgrep/rules/ --config p/owasp-top-ten --config p/security-audit --config p/trailofbits --config p/github-actions ... --error --metrics=off .github/

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran pnpm changeset to generate a changeset file

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

The pi coding agent made the feature flag local evaluation test assertions deterministic by normalizing JSON payloads before comparing mocked HTTP calls, validating generated batch UUIDs without comparing their random values, and suppressing expected legacy feature-flag deprecations inside this legacy behavior test class. Deprecation-specific behavior remains covered by the dedicated deprecation tests.

After CI reported the same Semgrep secrets: inherit finding handled in PostHog/posthog-js#4034, the agent applied the same inline nosemgrep suppression to the feature flags project board workflow because the called reusable workflow requires inherited secrets today.

@marandaneto marandaneto self-assigned this Jul 1, 2026
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

posthog-php Compliance Report

Date: 2026-07-01 09:36:07 UTC
Duration: 95137ms

✅ All Tests Passed!

45/45 tests passed


Capture Tests

29/29 tests passed

View Details
Test Status Duration
Format Validation.Event Has Required Fields 14ms
Format Validation.Event Has Uuid 6ms
Format Validation.Event Has Lib Properties 7ms
Format Validation.Distinct Id Is String 6ms
Format Validation.Token Is Present 7ms
Format Validation.Custom Properties Preserved 7ms
Format Validation.Event Has Timestamp 6ms
Retry Behavior.Retries On 503 5316ms
Retry Behavior.Does Not Retry On 400 2011ms
Retry Behavior.Does Not Retry On 401 2009ms
Retry Behavior.Respects Retry After Header 8016ms
Retry Behavior.Implements Backoff 15730ms
Retry Behavior.Retries On 500 5116ms
Retry Behavior.Retries On 502 5111ms
Retry Behavior.Retries On 504 5115ms
Retry Behavior.Max Retries Respected 16525ms
Deduplication.Generates Unique Uuids 12ms
Deduplication.Preserves Uuid On Retry 5113ms
Deduplication.Preserves Uuid And Timestamp On Retry 10323ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry 5118ms
Deduplication.No Duplicate Events In Batch 12ms
Deduplication.Different Events Have Different Uuids 7ms
Compression.Sends Gzip When Enabled 6ms
Batch Format.Uses Proper Batch Structure 6ms
Batch Format.Flush With No Events Sends Nothing 4ms
Batch Format.Multiple Events Batched Together 11ms
Error Handling.Does Not Retry On 403 2007ms
Error Handling.Does Not Retry On 413 2010ms
Error Handling.Retries On 408 5115ms

Feature_Flags Tests

16/16 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id 7ms
Request Payload.Flags Request Uses V2 Query Param 5ms
Request Payload.Flags Request Hits Flags Path Not Decide 5ms
Request Payload.Flags Request Omits Authorization Header 5ms
Request Payload.Token In Flags Body Matches Init 4ms
Request Payload.Groups Round Trip 5ms
Request Payload.Groups Default To Empty Object 5ms
Request Payload.Person Properties Distinct Id Auto Populated When Caller Omits It 4ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False 5ms
Request Payload.Disable Geoip Omitted Defaults To False 5ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key 4ms
Request Lifecycle.No Flags Request On Init Alone 3ms
Request Lifecycle.No Flags Request On Normal Capture 5ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests 8ms
Request Lifecycle.Mock Response Value Is Returned To Caller 5ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event 7ms

@marandaneto marandaneto marked this pull request as ready for review July 1, 2026 09:28
@marandaneto marandaneto requested a review from a team as a code owner July 1, 2026 09:28
@greptile-apps

greptile-apps Bot commented Jul 1, 2026

Copy link
Copy Markdown

Reviews (1): Last reviewed commit: "ci: suppress inherited secrets semgrep w..." | Re-trigger Greptile

Comment thread test/FeatureFlagLocalEvaluationTest.php Outdated
@marandaneto marandaneto merged commit 9b2d9a8 into main Jul 1, 2026
21 checks passed
@marandaneto marandaneto deleted the fix/deterministic-feature-flag-local-tests branch July 1, 2026 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants