Functional and performance testing suite - #27
Closed
byrnedj wants to merge 0 commit into
Closed
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a CMake/CTest-based test suite for DTO, including functional correctness tests and an A/B performance benchmarking harness (with optional plotting and baseline-building helpers) to detect regressions.
Changes:
- Added CTest-driven functional tests covering memset/memcpy/memmove/memcmp across sizes, edge cases, and multithreaded use.
- Added performance benchmarks: a single-version sampler (
test_perf.c) and a statically linked baseline-vs-current A/B comparator (test_perf_combined.c) with CSV export + plotting. - Added CMake build/install packaging, plus scripts/docs for building and managing performance baselines.
Reviewed changes
Copilot reviewed 10 out of 11 changed files in this pull request and generated 17 comments.
Show a summary per file
| File | Description |
|---|---|
| CMakeLists.txt | Adds CMake build, install/export targets, and CTest integration for functional/perf tests. |
| DTOConfig.cmake.in | Adds CMake package config for consumers. |
| tests/test_functional.c | New functional correctness test suite. |
| tests/test_perf.c | New single-build perf sampler that writes .dat and .samples. |
| tests/test_perf_combined.c | New statically-linked baseline/current A/B perf comparator with CSV export. |
| tests/perf_summary.c | New regression summarizer/analysis tool over .dat/.samples. |
| tests/dto_test_utils.h | Adds lightweight assertion/test runner + baseline I/O helpers. |
| tests/build_baseline.sh | Builds baseline object from upstream perf_baseline branch for A/B comparison. |
| tests/plot-distributions.R | Plots perf_combined distribution CSV output. |
| TESTING.md | Documents how to build/run tests and interpret perf outputs. |
| .gitignore | Ignores build outputs and baseline artifacts. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+34
to
+41
| #include <stdint.h> | ||
| #include <sched.h> | ||
| #include <dlfcn.h> | ||
| #include <sys/mman.h> | ||
| #include <sys/stat.h> | ||
| #include <sys/wait.h> | ||
| #include <math.h> | ||
| #include <x86intrin.h> |
Comment on lines
+656
to
+660
| qsort(bl_s, iters, sizeof(uint64_t), cmp_u64); | ||
| qsort(cur_s, iters, sizeof(uint64_t), cmp_u64); | ||
| slot->nsamples = iters; | ||
| slot->ok = 1; | ||
| exit(0); |
Comment on lines
+676
to
+677
| pid = fork(); | ||
| if (pid < 0) { perror("fork"); return -1; } |
Comment on lines
+8
to
+11
| * Measures throughput (GB/s) and per-operation latency (ns) for memset, | ||
| * memcpy, memcmp using cold-cache methodology (clflushopt before each | ||
| * operation). Results are compared against checked-in baselines to detect | ||
| * performance regressions from incoming DTO library changes. |
Comment on lines
+35
to
+41
| * Environment variables: | ||
| * BASELINE_DIR - Directory containing baseline .dat files | ||
| * RESULTS_DIR - Directory to write result files for summary | ||
| * UPDATE_BASELINES - If set, overwrite baseline file with measured results | ||
| * PERF_TOLERANCE - Allowed %% drop below baseline before failing (default: 0) | ||
| * PERF_CPU - CPU core to pin benchmark thread to (default: 1) | ||
| * PERF_PASSES - Full passes per benchmark; median-of-medians when >1 (default: 1) |
Comment on lines
+50
to
+54
| # Compile to object file with same flags as CMake Release | ||
| gcc -c -O3 -DNDEBUG -march=native -mwaitpkg -fPIC \ | ||
| -D_GNU_SOURCE -DDTO_STATS_SUPPORT \ | ||
| "${WORKTREE_DIR}/dto.c" -o "${OUTPUT_DIR}/dto_baseline_raw.o" \ | ||
| >>"${OUTPUT_DIR}/build.log" 2>&1 |
|
|
||
| ```bash | ||
| cd build | ||
| Rscript ../tests/plot-distributions.R perf_results/distributions.csv [4k|2m] [memcpy|memset|memcmp] |
Comment on lines
+4
to
+6
| # Usage: Rscript tests/plot-distributions.R [results_dir] [pagesize] | ||
| # results_dir - directory with distributions_*.csv (default: build/perf_results) | ||
| # pagesize - "4k" or "2m" (default: "4k") |
| ${CMAKE_BINARY_DIR}/dto_current.o | ||
| ${PERF_BASELINE_DIR}/dto_baseline.o | ||
| ) | ||
| target_link_libraries(dto-test-perf-combined PRIVATE m dl accel-config numa) |
| @@ -0,0 +1,208 @@ | |||
| cmake_minimum_required(VERSION 3.5...3.31) | |||
byrnedj
commented
Jun 24, 2026
| #define WARMUP_ITERS 10000 | ||
|
|
||
| /* Default tolerance: fail on any regression */ | ||
| #define DEFAULT_TOLERANCE 2.0 |
Contributor
Author
There was a problem hiding this comment.
we can make this more rigorous statistically - or just provide some guidance here
byrnedj
force-pushed
the
perf_baseline
branch
2 times, most recently
from
July 8, 2026 14:43
15ef0e8 to
d2d277a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit adds support for ctest suite and cmake build system
See TESTING.md for more details on performance testing. We use the perf_baseline branch as a stable reference point for performance comparison to avoid drift.