test: exercise SDK APIs in macOS compliance coverage - #805
dustinbyrne wants to merge 4 commits into
Conversation
|
Profile: macOS shared core (not iOS-device validation), gzip /batch, preload/autocapture disabled, public identity/group + reload + cached getter. SDK-triggered reloads are retained; unresolved queue acknowledgments time out. See sdk_compliance_adapter/README.md for contract limitations. posthog-ios Compliance ReportDate: 2026-09-14T19:43:18.138518+00:00
|
| Test | Status | Duration |
|---|---|---|
| Format Validation.Event Has Required Fields | ✅ | 501ms |
| Format Validation.Event Has Uuid | ✅ | 363ms |
| Format Validation.Event Has Lib Properties | ✅ | 376ms |
| Format Validation.Distinct Id Is String | ✅ | 242ms |
| Format Validation.Token Is Present | ✅ | 406ms |
| Format Validation.Custom Properties Preserved | ✅ | 203ms |
| Format Validation.Event Has Timestamp | ✅ | 294ms |
| Format Validation.Non Utc Event Timestamp Is Converted To Utc | ✅ | 214ms |
| Retry Behavior.Retries On 503 | ✅ | 9261ms |
| Retry Behavior.Does Not Retry On 400 | ✅ | 2255ms |
| Retry Behavior.Does Not Retry On 401 | ✅ | 2187ms |
| Retry Behavior.Respects Retry After Header | ✅ | 8680ms |
| Retry Behavior.Implements Backoff | ✅ | 22254ms |
| Retry Behavior.Retries On 500 | ✅ | 6748ms |
| Retry Behavior.Retries On 502 | ✅ | 6908ms |
| Retry Behavior.Retries On 504 | ✅ | 6640ms |
| Retry Behavior.Max Retries Respected | ❌ | 32215ms |
| Deduplication.Generates Unique Uuids | ✅ | 403ms |
| Deduplication.Preserves Uuid On Retry | ✅ | 6635ms |
| Deduplication.Preserves Uuid And Timestamp On Retry | ✅ | 13831ms |
| Deduplication.Preserves Uuid And Timestamp On Batch Retry | ✅ | 10184ms |
| Deduplication.No Duplicate Events In Batch | ✅ | 229ms |
| Deduplication.Different Events Have Different Uuids | ✅ | 209ms |
| Compression.Sends Gzip When Enabled | ✅ | 231ms |
| Batch Format.Uses Proper Batch Structure | ✅ | 224ms |
| Batch Format.Flush With No Events Sends Nothing | ✅ | 118ms |
| Batch Format.Multiple Events Batched Together | ✅ | 310ms |
| Error Handling.Does Not Retry On 403 | ✅ | 2321ms |
| Error Handling.Does Not Retry On 413 | ✅ | 2255ms |
| Error Handling.Retries On 408 | ✅ | 6834ms |
Failures
retry_behavior.max_retries_respected
Expected 4 requests, got 6
Feature_Flags Tests
View Details
| Test | Status | Duration |
|---|---|---|
| Request Payload.Request With Person Properties Device Id | ❌ | 558ms |
| Request Payload.Flags Request Uses V2 Query Param | ❌ | 502ms |
| Request Payload.Flags Request Hits Flags Path Not Decide | ❌ | 431ms |
| Request Payload.Flags Request Omits Authorization Header | ❌ | 402ms |
| Request Payload.Token In Flags Body Matches Init | ✅ | 309ms |
| Request Payload.Groups Round Trip | ❌ | 430ms |
| Request Payload.Groups Default To Empty Object | ❌ | 402ms |
| Request Payload.Disable Geoip False Propagates As Geoip Disable False | ❌ | 418ms |
| Request Payload.Disable Geoip Omitted Defaults To False | ❌ | 544ms |
| Request Payload.Flag Keys To Evaluate Contains Only Requested Key | ❌ | 475ms |
| Request Lifecycle.No Flags Request On Init Alone | ✅ | 110ms |
| Request Lifecycle.No Flags Request On Normal Capture | ✅ | 315ms |
| Request Lifecycle.Two Flag Calls Produce Two Remote Requests | ❌ | 515ms |
| Request Lifecycle.Mock Response Value Is Returned To Caller | ❌ | 449ms |
| Retry Behavior.Retries Flags On 502 | ❌ | 629ms |
| Retry Behavior.Retries Flags On 504 | ❌ | 795ms |
| Side Effect Events.Get Feature Flag Captures Feature Flag Called Event | ❌ | 492ms |
Failures
request_payload.request_with_person_properties_device_id
Expected 1 /flags requests, got 2
request_payload.flags_request_uses_v2_query_param
Expected 1 /flags requests, got 2
request_payload.flags_request_hits_flags_path_not_decide
Expected 1 /flags requests, got 2
request_payload.flags_request_omits_authorization_header
Expected 1 /flags requests, got 2
request_payload.groups_round_trip
Field 'company' not found in /flags request body at path 'groups.company'. Available keys: []
request_payload.groups_default_to_empty_object
Field 'group_properties' not found in /flags request body at path 'group_properties'. Available keys: ['person_properties', 'groups', '$anon_distinct_id', 'api_key', 'distinct_id', 'timezone', '$device_id']
request_payload.disable_geoip_false_propagates_as_geoip_disable_false
Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['groups', '$anon_distinct_id', '$device_id', 'api_key', 'distinct_id', 'timezone', 'person_properties']
request_payload.disable_geoip_omitted_defaults_to_false
Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['groups', 'distinct_id', 'person_properties', 'api_key', '$anon_distinct_id', '$device_id', 'timezone']
request_payload.flag_keys_to_evaluate_contains_only_requested_key
Field 'flag_keys_to_evaluate' not found in /flags request body at path 'flag_keys_to_evaluate'. Available keys: ['timezone', '$device_id', 'distinct_id', 'person_properties', 'api_key', 'groups', '$anon_distinct_id']
request_lifecycle.two_flag_calls_produce_two_remote_requests
Expected 2 /flags requests, got 3
request_lifecycle.mock_response_value_is_returned_to_caller
Expected last action result 'variant-a', got None
retry_behavior.retries_flags_on_502
Expected 2 /flags requests, got 3
retry_behavior.retries_flags_on_504
Expected 2 /flags requests, got 3
side_effect_events.get_feature_flag_captures_feature_flag_called_event
Event '$feature_flag_called' property '$feature_flag_response': expected True, got None
Compliance failures and follow-up planAt
The plan is to make these changes in the shared harness, adopt the compatible client preparation/profile bindings, and rerun the affected cases with positive and negative controls. The current reports remain red; later local results do not replace this PR's CI evidence. This PR is being put up for review as macOS shared-core adapter/CI coverage, not iOS-device or all-green conformance certification. Profile and public-API boundaries. Workflow success is advisory for these assertions. |
Prompt To Fix All With AI### Issue 1
sdk_compliance_adapter/Sources/RequestTracker.swift:24-36
**Capture UUIDs can collide**
Concurrent `/capture` requests can read the same `captured.count` index before either finishes. The global `beforeSend` callback then appends both UUIDs, but both requests can return the first UUID, corrupting capture responses and compliance observations. Serialize the SDK operation through UUID retrieval or correlate each callback with its originating capture instead of using a shared array index.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Reviews (1): Last reviewed commit: "test: exercise SDK APIs in macOS complia..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| func beginCapture() -> Int { | ||
| lock.lock() | ||
| defer { lock.unlock() } | ||
| unobservedCaptures += 1 | ||
| return captured.count | ||
| } | ||
|
|
||
| func finishCapture(after index: Int) -> String? { | ||
| lock.lock() | ||
| defer { lock.unlock() } | ||
| guard captured.count > index else { return nil } | ||
| unobservedCaptures -= 1 | ||
| return captured[index] |
There was a problem hiding this comment.
Concurrent /capture requests can read the same captured.count index before either finishes. The global beforeSend callback then appends both UUIDs, but both requests can return the first UUID, corrupting capture responses and compliance observations. Serialize the SDK operation through UUID retrieval or correlate each callback with its originating capture instead of using a shared array index.
Knowledge Base Used: Analytics capture and identity
Prompt To Fix With AI
This is a comment left during a code review.
Path: sdk_compliance_adapter/Sources/RequestTracker.swift
Line: 24-36
Comment:
**Capture UUIDs can collide**
Concurrent `/capture` requests can read the same `captured.count` index before either finishes. The global `beforeSend` callback then appends both UUIDs, but both requests can return the first UUID, corrupting capture responses and compliance observations. Serialize the SDK operation through UUID retrieval or correlate each callback with its originating capture instead of using a shared array index.
**Knowledge Base Used:** [Analytics capture and identity](https://app.greptile.com/posthog-org-19734/-/custom-context/knowledge-base/posthog/posthog-ios/-/docs/analytics-capture-and-identity.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
turnipdabeets
left a comment
turnipdabeets
left a comment
There was a problem hiding this comment.
LGTM, one greptile that might be worth addressing
|
This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, post a comment or remove the |
This branch has not been deployed
2 participants
💡 Motivation and Context
Make compliance results exercise the SDK rather than adapter-written flags HTTP, retries, parsing, and called-events.
Dateargument and observe SDK-generated UUIDs.Adapter/CI changes only: no shipped SDK source, public API, defaults, or release changes.
💚 How did you test it?
make test: 6 Swift and 5 Python tests passed.make smoke: timestamp/UUID/503 flush, invalid timestamps, and native flags 502/504 retry/result/group/called-event scenarios passed.The capture failure is the adapter's explicit timeout when retry exhaustion cannot be observed through a public drain callback. Flags failures expose extra identity/group-triggered requests, native field/lifecycle differences, and one-shot fixtures consumed before the explicit reload. The initial expectation was 34 passes; returned-value and called-event-value fixtures exposed two additional failures. Endpoint-specific smoke fixtures verify those SDK paths separately without replacing harness results.
Local Docker was unavailable; the hybrid Docker/Colima CI run remains to be checked. No iOS simulator/device validation is claimed. See
sdk_compliance_adapter/README.mdfor profile limitations.📝 Checklist
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Implemented with Pi worker using file and shell tools, followed by a fresh read-only agent review. The profile preserves SDK ownership of transport and side effects; known contract mismatches remain visible rather than filtered. Human review is required. The local agent session is not publicly accessible.