Support add_subdirectory and ExternalProject_Add in third-party projects, improve CI by adamant-pwn · Pull Request #565 · ratschlab/metagraph

adamant-pwn · 2025-11-13T20:29:25Z

Makes MetaGraph easy to integrate into external projects using CMake's add_subdirectory() or ExternalProject_Add.

Added metagraph/api/metagraph.hpp convenience header
Removed obsolete sdsl-lite manual installation steps (now handled by CMake)
Added CI workflow to test https://github.com/ratschlab/metagraph-examples on Ubuntu and MacOS
Split CI workflow into multiple parts for better GitHub runner utilization
Use ccache for faster builds

Use CMAKE_CURRENT_SOURCE_DIR instead of CMAKE_SOURCE_DIR to correctly locate package.json when metagraph is used as a submodule via add_subdirectory()

…atibility

…per include propagation

…include propagation - Moved metagraph-core target definition early (before includes) - Converted all include_directories() to target_include_directories(metagraph-core PUBLIC ...) - Used GLOB_RECURSE for cleaner file globbing - Moved findFolly() section to after target definition for better organization - macOS system includes now added to metagraph-core target - All includes now properly propagate to dependent targets

…or folly-config.h generation This fixes the issue where folly-config.h was generated in the wrong location when metagraph is used as a submodule.

…nfig.hpp

…lobal add_definitions with METAGRAPH_ALPHABET_DEF variable\n- Apply target_compile_definitions(metagraph-core PUBLIC )\n- Ensures external dependents (examples) receive the macro

…d_definitions with target_compile_definitions; propagate alphabet define via METAGRAPH_PUBLIC_DEFS; remove global link_directories; fix folly config include path for submodule builds

When metagraph is used as a subdirectory, CMAKE_CXX_FLAGS changes don't propagate. Add OpenMP_CXX_FLAGS directly to metagraph-core target with PUBLIC visibility so consumers inherit -fopenmp and get automatic OpenMP library linking from GCC.

GCC requires -fopenmp during linking to automatically add -lgomp. Previously only added it to compile options, but link stage also needs it.

SDSL is now built automatically via CMake ExternalProject, so manual installation with install.sh is no longer needed. Changes: - Removed 'install sdsl-lite' step from Linux CI workflow - Removed 'install sdsl-lite' step from MacOS CI workflow - Removed sdsl-lite installation instructions from documentation

Tests that the examples repository builds and runs correctly: - Checks out ratschlab/metagraph-examples with submodules - Builds the examples using CMake - Runs basic_query example with test data This ensures the examples stay compatible with MetaGraph changes.

- Fetches and checks out current metagraph commit in the examples submodule - Tests examples against the PR/push being evaluated - Removed custom path to use default checkout location

Test metagraph-examples on both Ubuntu and MacOS: - Ubuntu 22.04 with g++-13 - MacOS latest (Apple Silicon) - Platform-specific dependency installation

…roject

…udata, as it is needed for boost.

karasikov · 2025-11-17T23:51:01Z

 #include <optional>
 #include <vector>

+#include "common/config.hpp"


why do we need this here?

We use some compile definitions, in this particular file MODE_TI, which are used by sdsl. If Metagraph is added via ExternalProject_Add, the parent project must also provide them as defines (which is done by generating them into common/config.hpp), otherwise there will be linking errors.

So the parent project doesn't have to define those flags? But how can we be sure not to miss these flags in some files? Or, are we going to add this common/config.hpp into every file in the project?

Yes, parent projects need to provide them when using ExternalProject_Add or make install.

When using add_subdirectory, parent project inherits them automatically, but with ExternalProject_Add or system-wide install via make install, it's tricky, and could be argued that parent project shouldn't be responsible for matching all of them when using library. For example, it wouldn't be sufficient to define CMAKE_DBG_ALPHABET like it's done in add_subdirectory, as one would also need to provide internal definitions like _DNA_GRAPH.

When choosing places to add common/config.hpp, I added it anywhere the file or its external header dependency relies on these definitions. As it's now in CI, we'll be notified if missing a definition straight up leads to compilation or linking error, or if it prevents even basic usage of Metagraph.

In any case, it will not affect directly building Metagraph from sources or using it via add_subdirectory because we still keep compile definitions in this case. If we're not satisfied it covers all use-cases for ExternalProject, we can also recommend parent projects to use -include common/config.hpp compilation flag when in doubt, to force-include it? But I'm not sure there's much more we can do on our end to ensure proper propagation, short of adding it to all files, which I don't think we want to do.

I think it's fine to leave this to the user, if we specify which flags need to be set. Similar how we add MODE_TI when we use sdsl.
IMO The more config.hpp files we add, the more potential freedom for bugs we create

I suppose another option for us would be to remove include "commom/config.hpp" from our files and to tell external projects that they need to include it before any other of our headers? Not sure what would be the best practice here.

Yes, I think that stuff can be inside metagraph.hpp
https://github.com/ratschlab/metagraph-examples/blob/master/examples/basic_query.cpp#L29C1-L29C25

karasikov · 2025-11-19T12:41:41Z

+// Core graph and annotation structures
+#include "graph/annotated_dbg.hpp"
+
+// Configuration management


This comment could be more informative. As now, it's not helpful at all

karasikov · 2025-11-19T12:50:59Z

+// Auto-generated compile definitions for MetaGraph
+// This file is auto-generated by CMake and should not be edited manually
+// Note: These definitions complement (not replace) CMake compile definitions


How should this be edited if not manually?

The config.hpp.in should be edited manually. The comment refers to the config.hpp file that CMake generates based on it, as .in file is a template for it.

Not ideal. First, because unclear what the comment refers to. Second, because there is another config.hpp in the project (cli/config/config.hpp). Last, the things we discussed in DM

Will clarify. Can also rename file to compile_definitions.hpp or similar.

karasikov · 2025-11-19T13:23:49Z

 #include <sdsl/uint128_t.hpp>
 #include <sdsl/uint256_t.hpp>

+#include "common/config.hpp"


Shouldn't it go before <sdsl/uint128_t.hpp>? That uses MODE_TI

adamant-pwn added 30 commits June 20, 2024 18:17

Install all included headers

1dd0eed

Collect include dirs from subdirs

5feea0c

Ignore PROJECT_SOURCE_DIR

e9e1b85

Add metagraph.hpp convenience header for API usage

b65b473

Add root CMakeLists.txt wrapper for submodule usage

d9405a3

Fix package.json path for submodule usage

70ccd91

Use CMAKE_CURRENT_SOURCE_DIR instead of CMAKE_SOURCE_DIR to correctly locate package.json when metagraph is used as a submodule via add_subdirectory()

Include sdsl-lite via ExternalProject_Add

27f2825

Use PROJECT_BINARY_DIR instead of CMAKE_BINARY_DIR for submodule comp…

7bcfd47

…atibility

Move metagraph.hpp to src/ and use target_include_directories for pro…

2de0513

…per include propagation

Fix CMAKE_BINARY_DIR to PROJECT_BINARY_DIR in CMakeListsHelpers.txt f…

e4ea6f3

…or folly-config.h generation This fixes the issue where folly-config.h was generated in the wrong location when metagraph is used as a submodule.

Fix include path in metagraph.hpp: config/config.hpp -> cli/config/co…

9113eaf

…nfig.hpp

Propagate alphabet macro via target usage requirements\n\n- Replace g…

b6c3be2

…lobal add_definitions with METAGRAPH_ALPHABET_DEF variable\n- Apply target_compile_definitions(metagraph-core PUBLIC )\n- Ensures external dependents (examples) receive the macro

CMake: export third-party link dirs via target interfaces; replace ad…

900e785

…d_definitions with target_compile_definitions; propagate alphabet define via METAGRAPH_PUBLIC_DEFS; remove global link_directories; fix folly config include path for submodule builds

Fix OpenMP: add target_link_options for -fopenmp flag

f460b90

GCC requires -fopenmp during linking to automatically add -lgomp. Previously only added it to compile options, but link stage also needs it.

Update examples workflow to test against current commit

e463b9e

- Fetches and checks out current metagraph commit in the examples submodule - Tests examples against the PR/push being evaluated - Removed custom path to use default checkout location

Add MacOS testing to examples workflow

3997863

Test metagraph-examples on both Ubuntu and MacOS: - Ubuntu 22.04 with g++-13 - MacOS latest (Apple Silicon) - Platform-specific dependency installation

Fix path to metagraph submodule in test-examples workflow

f1deb09

fix options propagation

d7025b6

Bettter CI naming

b905060

limit -j

0dc31f6

Don't build Metagraph binary and tests unless requested / top-level p…

2f1ec36

…roject

libboost no longer provides static lzma in Ubuntu 24.04

b25eb20

Fix cmake module path when used as a subdirectory. Always look for ic…

6c891e4

…udata, as it is needed for boost.

source_dir -> current_source_dir

d001a52

try capturing traces in macos runner...

01b7d13

Setup tmate session for failed workflows

08df45c

adamant-pwn added 19 commits November 16, 2025 19:43

tmate session to investigate failure

410f580

Propagate arch to shell

f92689d

Composite actions don't support defaults :(

182348a

Only use arch-aware shell on MacOS

0a361ec

Use ccache more

2d4ff67

Don't install ccache on Linux examples

f573b88

fix ccache stat call in Linux examples

9585881

Update

37ecce9

split install ccache on linux/macos

e4861dd

meaningful names for cache key on avx / static

8c3c386

runner.image instead of passing platform

7646886

Pass runner platform

fe14195

always setup ccache

58ba677

Update

a1e90ea

symlink repo to external project example

065dcd2

cache submodules in externalproject examples

ec1e8ff

hack with timestamps to prevent re-download in ExternalProject

7fb197d

skip fetch in ExternalProject

4aad9b4

Update

d68b396

adamant-pwn changed the title ~~Make MetaGraph usable with add_subdirectory in external projects, add CI workflow for it~~ Support add_subdirectory and ExternalProject_Add in third-party projects, improve CI Nov 17, 2025

adamant-pwn added 4 commits November 17, 2025 16:59

cache docker build, use target arch in integration tests

2b11c8c

Use ccache in external projects

7e6bb29

put ccache in COMPILER_LAUNCHER

a4a83f4

Minor

32135bc

adamant-pwn merged commit 00945af into master Nov 17, 2025

adamant-pwn deleted the api-header-example branch November 17, 2025 19:41

karasikov reviewed Nov 17, 2025

View reviewed changes

karasikov reviewed Nov 19, 2025

View reviewed changes

Conversation

adamant-pwn commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamant-pwn Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamant-pwn Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamant-pwn Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamant-pwn Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

adamant-pwn commented Nov 13, 2025 •

edited

Loading

adamant-pwn Nov 19, 2025 •

edited

Loading

adamant-pwn Nov 19, 2025 •

edited

Loading

adamant-pwn Nov 19, 2025 •

edited

Loading

adamant-pwn Nov 19, 2025 •

edited

Loading