From 21da369cbf337fc4bfd8d19c1197fabb2ff8cddb Mon Sep 17 00:00:00 2001 From: Greta Date: Mon, 19 Oct 2026 12:54:31 +0800 Subject: [PATCH] docs: unify bilingual guides and result navigation --- CONTRIBUTING.md | 19 ++ README.md | 2 +- README_zh.md | 4 +- docs/README.md | 16 +- docs/README_zh.md | 23 +- docs/agent_memory/README.md | 359 +++++++++++--------- docs/agent_memory/README_zh.md | 405 +++++++++++----------- docs/agent_memory/architecture.md | 7 +- docs/agent_memory/architecture_zh.md | 84 ++--- docs/agent_memory/results.md | 42 ++- docs/agent_memory/results_zh.md | 81 ++--- docs/user_memory/README.md | 481 +++++++++++---------------- docs/user_memory/README_zh.md | 427 ++++++++++-------------- docs/user_memory/results.md | 10 +- docs/user_memory/results_zh.md | 46 +++ 15 files changed, 983 insertions(+), 1023 deletions(-) create mode 100644 docs/user_memory/results_zh.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d7c4e92..455a94c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -30,6 +30,25 @@ Match the surrounding code style, avoid unrelated formatting changes, and add a regression test for a behavior fix. Documentation-only changes should pass the link checks; include a minimal before/after example when explaining a bug. +## Maintaining documentation + +- Keep track guides in `docs/user_memory/` and `docs/agent_memory/`, with + `README.md` for English and `README_zh.md` for Chinese. Use the existing + `name.md` / `name_zh.md` convention for other paired pages; avoid renaming URLs + just for style. +- Update both languages when changing guide sections, commands, options, or + required setup. Translate explanations, not product names, paths, or flags. +- Maintain numeric result tables only in each track's English `results.md`. + Chinese result pages provide translated context and links to the canonical + tables rather than copying scores. Keep methodology explanations aligned. +- Keep parameter defaults, dataset/license details, and test commands in their + existing reference files; link to them instead of duplicating full references. + Distinguish bounded smoke checks, full evaluations, and destructive operations. +- Check file links and preserve existing heading anchors with aliases when + reorganizing sections. + The automated link checker validates relative file targets, not anchors or + external URLs. + ## Adding a User Memory adapter The public entry points use `add(messages, user_id, ...)` and diff --git a/README.md b/README.md index 7fafaa0..4184789 100644 --- a/README.md +++ b/README.md @@ -80,7 +80,7 @@ hf download EverMind-AI/EvoAgentBench \ ## Results - User Memory public result snapshot: [docs/user_memory/results.md](./docs/user_memory/results.md) -- AgentBench evaluation results: [docs/agent_memory/results.md](./docs/agent_memory/results.md) +- Agent Memory evaluation results: [docs/agent_memory/results.md](./docs/agent_memory/results.md) Documentation index: [docs/README.md](./docs/README.md) diff --git a/README_zh.md b/README_zh.md index 9433f27..9a33e05 100644 --- a/README_zh.md +++ b/README_zh.md @@ -77,8 +77,8 @@ hf download EverMind-AI/EvoAgentBench \ ## 结果 -- User Memory 公开结果快照（英文）：[docs/user_memory/results.md](./docs/user_memory/results.md) -- AgentBench 评测结果：[docs/agent_memory/results_zh.md](./docs/agent_memory/results_zh.md) +- User Memory 评测结果：[中文说明与结果表入口](./docs/user_memory/results_zh.md) +- Agent Memory 评测结果：[中文说明与结果表入口](./docs/agent_memory/results_zh.md) 文档索引：[docs/README_zh.md](./docs/README_zh.md) diff --git a/docs/README.md b/docs/README.md index caef676..fed917e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,11 +8,25 @@ matches the system you want to evaluate: | Track | Evaluation target | Guide | Results | |---|---|---|---| | User Memory Evaluation | Memory backend APIs exposed through `add()` and `search()` | [Guide](./user_memory/README.md) | [Results](./user_memory/results.md) | -| Agent Memory Evaluation | Agent runtimes equipped with memory plugins | [Guide](./agent_memory/README.md) | [Results](./agent_memory/results.md) | +| Agent Memory Evaluation | Agent runtimes with or without memory plugins | [Guide](./agent_memory/README.md) | [Results](./agent_memory/results.md) | Agent Memory contributors can also read the current [architecture and extension contracts](./agent_memory/architecture.md). +## Reading path + +Each track guide follows the same sequence: overview, environment/configuration, +data preparation, quick start, advanced runs, outputs/results, and contribution. +User Memory additionally introduces its five benchmarks. Start with the bounded +quick-start check before launching complete runs; both call real services. + +English and Chinese guides maintain matching sections and runnable commands. +Numeric result tables are maintained only in the English result pages; Chinese +result pages explain the metrics and link to those tables. Product names and +code identifiers are kept unchanged across languages. + +## Related references + Related documentation remains next to the files it describes: - Dataset preparation and licenses: `data//README.md` diff --git a/docs/README_zh.md b/docs/README_zh.md index 4ea4a0d..3dd5954 100644 --- a/docs/README_zh.md +++ b/docs/README_zh.md @@ -6,18 +6,29 @@ OmniMemEval 包含两条互补的评测线。请根据待评测对象选择对 | 评测线 | 评测对象 | 指南 | 结果 | |---|---|---|---| -| User Memory Evaluation | 通过 `add()` 和 `search()` 暴露能力的 Memory Backend API | [指南](./user_memory/README_zh.md) | [结果（英文）](./user_memory/results.md) | -| Agent Memory Evaluation | 安装记忆插件后的 Agent Runtime | [指南](./agent_memory/README_zh.md) | [结果](./agent_memory/results_zh.md) | +| User Memory Evaluation（用户记忆评测） | 通过 `add()` 和 `search()` 暴露能力的记忆后端 API | [指南](./user_memory/README_zh.md) | [结果](./user_memory/results_zh.md) | +| Agent Memory Evaluation（智能体记忆评测） | 有无记忆插件时的智能体运行时 | [指南](./agent_memory/README_zh.md) | [结果](./agent_memory/results_zh.md) | 参与 Agent Memory 开发时，还可以阅读当前的 [架构与扩展契约](./agent_memory/architecture_zh.md)。 +## 阅读顺序 + +两份运行指南采用相同顺序：概览、环境与配置、数据准备、快速开始、进阶运行、 +输出与结果、扩展与贡献。User Memory 另外介绍五个 benchmark。 +建议先运行有明确范围限制的快速检查，再运行完整评测；两者都会调用真实服务。 + +中英文指南保持章节与可执行命令对应。结果数值只在英文结果页维护， +中文结果页提供指标说明和表格入口。产品名与代码标识在两种语言中保持原样。 + +## 相关参考 + 与具体文件直接相关的文档继续就近维护： - 数据准备与许可证：`data//README.md` - 环境模板与参数： - [env_examples/README.md](../env_examples/README.md) 和 - [env_examples/PARAMETERS.md](../env_examples/PARAMETERS.md) -- 测试命令：[scripts/tests/README.md](../scripts/tests/README.md) + [env_examples/README.md（英文）](../env_examples/README.md) 和 + [env_examples/PARAMETERS.md（英文）](../env_examples/PARAMETERS.md) +- 测试命令：[scripts/tests/README.md（英文）](../scripts/tests/README.md) - 开发与结果提交规范：[CONTRIBUTING.md（英文）](../CONTRIBUTING.md) -- 第三方归属：[THIRD_PARTY_NOTICES.md](../THIRD_PARTY_NOTICES.md) +- 第三方归属：[THIRD_PARTY_NOTICES.md（英文）](../THIRD_PARTY_NOTICES.md) diff --git a/docs/agent_memory/README.md b/docs/agent_memory/README.md index 3b4a142..b9f116e 100644 --- a/docs/agent_memory/README.md +++ b/docs/agent_memory/README.md @@ -1,121 +1,110 @@ # Agent Memory Evaluation -[中文版](./README_zh.md) +[中文版](./README_zh.md) · [Documentation index](../README.md) · [Results](./results.md) -[Documentation index](../README.md) +## Overview -Agent Memory Evaluation is the AgentBench-based evaluation track in OmniMemEval. It measures the task performance of an agent runtime after a memory plugin is installed. The current implementation evaluates OpenClaw across five task domains and supports both plain agent runs and memory-plugin lifecycle runs. +This track evaluates an agent runtime with or without a memory plugin using +the AgentBench module in OmniMemEval. The current public runtime adapter covers +OpenClaw across five task domains. -Results are recorded in [results.md](./results.md). The current architecture -and extension contracts are documented in [architecture.md](./architecture.md). +There are two evaluation setups: -## Evaluation Protocols +- Baseline: `test_only` or `train_then_test` without the target memory plugin. +- Memory plugin: `memory_train_backup_test`, including cleanup, training, + settling, backup, restore, and testing. -- plain AgentBench: runs `test_only` or `train_then_test` without enabling the target memory plugin. -- memory plugin AgentBench: runs the `memory_train_backup_test` protocol, including memory cleanup, train execution, memory settling, backup, restore, and test execution. +Read this guide in order for a first run. For an existing setup, jump to +[advanced runs](#advanced-runs), [outputs and results](#outputs-and-results), +or [extensions and contribution](#extensions-and-contribution). +Run all commands from the repository root. -## Data Source + +## Environment and configuration -AgentBench uses the `EvoAgentBench` dataset published by EverMind (Hugging Face/GitHub namespace: `EverMind-AI`). Dataset ownership, task splits, and upstream benchmark definitions follow the EverMind-AI/EvoAgentBench dataset and project documentation. OmniMemEval only provides runner integration, configuration, lifecycle management, and result organization. +### Install dependencies -References: - -- Hugging Face dataset: `EverMind-AI/EvoAgentBench` -- GitHub project: `EverMind-AI/EvoAgentBench` -- Underlying benchmarks: BrowseCompPlus, OmniMath, SWE-Bench, LiveCodeBench, GDPVal - -Dataset size: - -| Domain | Benchmark | Train | Test | -|---|---|---:|---:| -| `information_retrieval` | BrowseCompPlus | 154 | 65 | -| `reasoning` | OmniMath | 478 | 100 | -| `software_engineering` | SWE-Bench | 101 | 26 | -| `code_implementation` | LiveCodeBench | 97 | 39 | -| `knowledge_work` | GDPVal | 87 | 58 | - -## Environment - -Agent Memory Evaluation uses a separate conda environment from the User Memory track: +Use a dedicated Agent Memory environment, separate from User Memory: ```bash conda create -n agentmem python=3.12 -y conda activate agentmem python -m pip install -U pip -pip install -r requirements_agentbench.txt +python -m pip install -r requirements_agentbench.txt +npm install -g openclaw ``` -Do not reuse the User Memory environment for AgentBench. User Memory and AgentBench dependencies are maintained in separate requirements files. - -Install OpenClaw CLI and make sure it is available in `PATH`: +Configure a valid OpenClaw model/provider, then verify that the CLI is available: ```bash -npm install -g openclaw +openclaw --version +openclaw agent --help ``` -For memory-plugin lifecycle evaluations, install the target memory plugin before running AgentBench. For example, install the MemOS local plugin with: +### Configure credentials + +Copy the Agent Memory template, then fill in the settings for the selected domain: ```bash -curl -fsSL https://raw.githubusercontent.com/MemTensor/MemOS/main/apps/memos-local-plugin/install.sh | bash +cp env_examples/.env.agent .env.agent ``` -After installation, configure OpenClaw and the installed memory plugin before running validation. The exact plugin configuration depends on the plugin under evaluation. For the MemOS local plugin, complete the plugin configuration generated by the installer under the OpenClaw configuration directory, then confirm that the plugin entry is enabled in OpenClaw. - -At minimum, verify the following before running AgentBench: +AgentBench loads the project `.env.agent` by default; `--env FILE` loads an +additional env file. Keep these credentials separate from User Memory backend +files such as `.env.memos`. -- OpenClaw has a valid model/provider configuration. -- The memory plugin is installed and enabled in the OpenClaw config. -- Required plugin credentials, local paths, or service endpoints are configured. -- The AgentBench env file `.env.agent` is prepared as described below. +| Purpose | Configuration | +|---|---| +| Agent model/provider | `configs/agentbench/agents/openclaw.yaml`, using `LLM_BASE_URL` and `LLM_API_KEY` | +| Judge | `JUDGE_MODEL`, `JUDGE_API_BASE`, `JUDGE_API_KEY` | +| Information retrieval embeddings | `IR_EMBEDDING_ENDPOINT`, `IR_EMBEDDING_MODEL`, `IR_EMBEDDING_API_KEY` | +| External evaluation service | `EVALUATION_API_BASE`, `EVALUATION_API_KEY`, `EVALUATION_MODEL_OWNER`, `EVALUATION_MODEL_NAME`, `EVALUATION_TIMEOUT`, `EVALUATION_MAX_RETRIES` | -After installation and configuration are complete, verify that the OpenClaw CLI can be invoked normally: +Use the [env template](../../env_examples/.env.agent) and selected +`configs/agentbench/domains/*.yaml` as the field reference. Provider fields +not configured through the agent YAML/env settings can fall back to values +resolvable from `~/.openclaw/openclaw.json`. Do not commit filled credential files. -```bash -openclaw --version -openclaw agent --help -``` +### Prepare a memory plugin when needed -Create the Agent Memory env file from the template: +A baseline run does not require the target memory plugin. For +`memory_train_backup_test`, install and configure that plugin first. +For example, the MemOS local plugin provides this installer: ```bash -cp env_examples/.env.agent .env.agent +curl -fsSL https://raw.githubusercontent.com/MemTensor/MemOS/main/apps/memos-local-plugin/install.sh | bash ``` -AgentBench loads `.env.agent` by default. `--env FILE` can be used to supply an additional env file. This keeps Agent Memory credentials separate from User Memory backend env files such as `.env.memos` and `.env.mem0`. +Review upstream installers before executing them. Complete the generated plugin +configuration, required credentials, local paths, and service endpoints, then +confirm the plugin is enabled in OpenClaw. -OpenClaw model settings are configured in: +Memory-plugin runs enforce `runtime.home_mode: isolated_copy`, but +`home_links` can still link real plugin data into the temporary OpenClaw home. +Lifecycle clear/restore commands can delete or overwrite that data and restart +services. Inspect the selected plugin YAML and use dedicated evaluation data, +not personal memory. -```text -configs/agentbench/agents/openclaw.yaml -``` + +## Data preparation -Common `.env.agent` fields: +AgentBench uses the `EverMind-AI/EvoAgentBench` dataset. Dataset ownership, +splits, and underlying benchmark definitions follow the upstream project; +OmniMemEval provides runner integration, configuration, lifecycle management, +and result organization. -```bash -LLM_BASE_URL=... -LLM_API_KEY=... - -JUDGE_MODEL=... -JUDGE_API_BASE=... -JUDGE_API_KEY=... - -IR_EMBEDDING_ENDPOINT=... -IR_EMBEDDING_MODEL=... -IR_EMBEDDING_API_KEY=... - -EVALUATION_API_BASE=... -EVALUATION_API_KEY=... -EVALUATION_MODEL_OWNER=... -EVALUATION_MODEL_NAME=... -EVALUATION_TIMEOUT=240 -EVALUATION_MAX_RETRIES=3 -``` - -OpenClaw model/provider credentials are injected from `.env.agent` into `configs/agentbench/agents/openclaw.yaml`. If provider fields are not set there, the adapter falls back to values resolvable from `~/.openclaw/openclaw.json`. +- [Hugging Face dataset](https://huggingface.co/datasets/EverMind-AI/EvoAgentBench) +- [Upstream project](https://github.com/EverMind-AI/EvoAgentBench) -## Data Preparation +| Domain | Benchmark | Train | Test | +|---|---|---:|---:| +| `information_retrieval` | BrowseCompPlus | 154 | 65 | +| `reasoning` | OmniMath | 478 | 100 | +| `software_engineering` | SWE-Bench | 101 | 26 | +| `code_implementation` | LiveCodeBench | 97 | 39 | +| `knowledge_work` | GDPVal | 87 | 58 | -Run the following commands from the repository root: +### Download the dataset ```bash mkdir -p data/agentbench @@ -137,16 +126,21 @@ data/agentbench/ The `hf` CLI is provided by `huggingface_hub`; see the [upstream CLI guide](https://huggingface.co/docs/huggingface_hub/en/guides/cli). -Domain data paths are configured under `configs/agentbench/domains/*.yaml`. If a custom data location is used, update the relevant domain yaml fields. +If using custom data locations, update the corresponding paths in +`configs/agentbench/domains/*.yaml`. + +### Prepare domain-specific dependencies -LiveCodeBench requires its official verifier package: +Install only the additional tools needed by the selected domains. +LiveCodeBench uses its official verifier package: ```bash git clone https://github.com/LiveCodeBench/LiveCodeBench.git ./LiveCodeBench -pip install --no-deps -e ./LiveCodeBench +python -m pip install --no-deps -e ./LiveCodeBench ``` -BrowseComp-Plus dense retrieval requires preprocessing, an index, and embedding service configuration: +BrowseComp-Plus dense retrieval requires preprocessing, an index, and embedding +service configuration: ```bash python scripts/agentbench/utils/browsecomp-plus-tools/setup_data.py \ @@ -156,7 +150,9 @@ python scripts/agentbench/utils/browsecomp-plus-tools/build_dense_index.py \ --config configs/agentbench/domains/information_retrieval.yaml ``` -SWE-Bench requires Docker. GDPVal requires PDF/Office processing utilities. On Debian/Ubuntu: +SWE-Bench requires Docker and network access to PyPI/GitHub from the evaluation +containers. GDPVal requires PDF/Office processing utilities. On Debian/Ubuntu, +install the required system packages with administrator privileges: ```bash apt-get update @@ -164,12 +160,13 @@ apt-get install -y docker.io poppler-utils libreoffice openjdk-21-jdk systemctl start docker ``` -## Quick Start +Java is relevant to Pyserini/BM25 retrieval; a setup using only a prebuilt dense +index may not need it. These are domain-specific requirements, not prerequisites +for the reasoning quick start below. + +## Quick start -After completing environment, credentials, and data preparation above, run one -reasoning task. `--test-split 1` selects the first test question for this domain; -omit it to evaluate the complete test split. `--parallel` controls concurrency, -not the number of selected tasks. +After preparing the environment, credentials, and dataset, run one reasoning task: ```bash ./scripts/run_agent_eval.sh \ @@ -182,110 +179,113 @@ not the number of selected tasks. --parallel 1 ``` -## Protocols +This invokes real model/judge services and may incur charges. For reasoning, +`--test-split 1` selects the first test question; omitting it selects the full +test split. `--parallel` controls concurrency, not task count. A smoke run is +not a full benchmark result. + +For complete protocols, see [advanced runs](#advanced-runs). +The runner's `--help` lists CLI options. + +## Advanced runs -### `test_only` + +### Evaluation protocols -Runs the test split. +The following commands use complete configured splits. + +#### `test_only` + +Run the test split: ```bash ./scripts/run_agent_eval.sh \ - --agent openclaw \ - --domain reasoning \ - --protocol test_only \ - --version baseline_reasoning_test \ - --trials 1 \ - --parallel 1 + --agent openclaw --domain reasoning --protocol test_only \ + --version baseline_reasoning_test --trials 1 --parallel 1 ``` -### `train_then_test` +#### `train_then_test` -Runs train split followed by test split. This protocol does not execute memory plugin lifecycle commands. +Run training followed by testing, without memory-plugin lifecycle commands: ```bash ./scripts/run_agent_eval.sh \ - --agent openclaw \ - --domain reasoning \ - --protocol train_then_test \ - --version train_then_test_reasoning \ - --trials 1 \ - --parallel 1 + --agent openclaw --domain reasoning --protocol train_then_test \ + --version train_then_test_reasoning --trials 1 --parallel 1 ``` -### `memory_train_backup_test` +#### `memory_train_backup_test` -This protocol evaluates memory plugins separately from baseline results: +Evaluate a memory plugin separately from the baseline: -1. Set plugin to train mode. -2. Clear memory. -3. Run train split. -4. Submit verifier feedback as the next turn in the same OpenClaw session. -5. If structured feedback is enabled in plugin config, submit plugin-side explicit feedback while preserving the same session/episode semantics. -6. Wait for memory settling/evolution. -7. Back up domain memory. -8. Restore the domain backup before every test run. -9. Run test split and write results to `results/agentbench`. +1. Set the plugin to train mode and clear memory. +2. Run training. +3. Send verifier feedback as the next turn in the same OpenClaw session. +4. If enabled, submit plugin-side structured feedback with the same session/episode semantics. +5. Wait for memory settling/evolution. +6. Back up domain memory. +7. Restore the domain backup before every test run, even when test writes are disabled. +8. Run testing and write results under `results/agentbench/`. Single domain: ```bash ./scripts/run_agent_eval.sh \ - --agent openclaw \ - --domain reasoning \ - --protocol memory_train_backup_test \ - --memory-plugin memos \ - --version memos_reasoning_eval \ - --test-runs 1 \ - --trials 1 \ - --parallel 1 + --agent openclaw --domain reasoning --protocol memory_train_backup_test \ + --memory-plugin memos --version memos_reasoning_eval \ + --test-runs 1 --trials 1 --parallel 1 ``` -Five domains: +All five domains: ```bash ./scripts/run_agentbench_memory_train_backup_test.sh \ - --memory-plugin memos \ - --version memos_5domain_eval \ - --test-runs 1 \ - --trials 1 \ - --parallel 1 + --memory-plugin memos --version memos_5domain_eval \ + --test-runs 1 --trials 1 --parallel 1 ``` Domain subset: ```bash ./scripts/run_agentbench_memory_train_backup_test.sh \ - --memory-plugin memos \ - --domains reasoning,code_implementation \ + --memory-plugin memos --domains reasoning,code_implementation \ --version memos_reasoning_code_eval ``` -## Memory Plugin Configuration +### Memory plugin configuration -Plugin lifecycle configs live under: - -```text -configs/agentbench/memory_plugins/ - memos.yaml - everos.yaml - openviking.yaml - supermemory.yaml - hindsight.yaml -``` +Lifecycle configs live under `configs/agentbench/memory_plugins/`: +`memos.yaml`, `everos.yaml`, `openviking.yaml`, `supermemory.yaml`, +and `hindsight.yaml`. Each config declares: -- `plugin`: plugin label used in result directory names. -- `backup_dir` and `backup_file_template`: memory backup location. -- `settle_seconds` or `commands.wait_settle`: post-train memory settling logic. +- `plugin`: label used in result directory names. +- `backup_dir` and `backup_file_template`: backup location. +- `settle_seconds` or `commands.wait_settle`: post-training settling behavior. - `home_links`: plugin paths linked into the isolated OpenClaw home. -- `modes.train/test` or `commands.set_mode_*`: plugin read/write mode. -- `commands.clear/backup/restore`: memory cleanup, backup, and restore commands. -- `feedback`: train feedback settings and optional plugin-side structured feedback. +- `modes.train/test` or `commands.set_mode_*`: read/write modes. +- `commands.clear/backup/restore`: lifecycle operations. +- `feedback`: training feedback and optional plugin-side structured feedback. + +For example, `memos.yaml` enables structured feedback: + +```yaml +feedback: + enabled: true + timeout: 300 + memos_structured_submit: true + memos_submit_timeout: 900 +``` -## Results +Other plugin configs use ordinary training feedback by default without this +structured submit. Check the selected YAML rather than assuming identical +plugin behavior. -Each domain writes outputs under: + +## Outputs and results + +For the memory-plugin protocol, the output layout is: ```text results/agentbench/--/ @@ -293,15 +293,60 @@ results/agentbench/--/ memory_lifecycle.log memory_lifecycle.json train/ + summary.json + report.md + __trial_1/ + result.json + response.txt + session.jsonl + verifier/ test_run_1/ + summary.json + report.md + __trial_1/ + result.json + response.txt + session.jsonl + verifier/ ``` -Key files: +Key fields: - `summary.json`: `pass@1`, average reward, and average runtime. -- `result.json.agent_result`: OpenClaw completion status and session recovery metadata. -- `result.json.feedback_result`: train feedback turn status. +- `result.json.agent_result`: completion status, runtime, and session recovery metadata. +- `result.json.feedback_result`: same-session training feedback status. - `result.json.memos_feedback_result`: plugin-side structured feedback status when enabled. -- `memory_lifecycle.json`: clear/backup/restore lifecycle events. +- `memory_lifecycle.json`: clear/backup/restore events. + +See [evaluation results](./results.md) for published setups and tables. The +snapshot also includes Hermes results; the current public runtime adapter covers +OpenClaw, not a Hermes reproduction path. Model, judge, and embedding service +availability can affect per-sample rewards. + +## Extensions and contribution + +The [architecture and extension contracts](./architecture.md) describe the +runner, session identity, result fields, and agent/domain extension points. +Agent Memory adapters are separate from the User Memory +`scripts/client_factory/` adapter layer. + +See [CONTRIBUTING.md](../../CONTRIBUTING.md) for PR workflow and evidence +requirements. The root [repository layout](../../README.md#repository-layout) +describes where code, configuration, and data preparation live. + +## Verification and licenses + +Use a separate development environment with Python 3.11 or 3.12 for offline checks: + +```bash +python -m pip install -r requirements-dev.txt +python -m pytest -q +python scripts/check_doc_links.py +``` + +The [test guide](../../scripts/tests/README.md) distinguishes offline tests, +repository checks, and real-backend integration tests. Offline CI does not +validate benchmark scores or call live services. -Recent evaluation results are recorded in [results.md](./results.md). Model service, external judge service, and embedding service status may affect per-sample rewards. +See [LICENSE](../../LICENSE) for the code license. External datasets keep their +upstream licenses; see [THIRD_PARTY_NOTICES.md](../../THIRD_PARTY_NOTICES.md). diff --git a/docs/agent_memory/README_zh.md b/docs/agent_memory/README_zh.md index 09d7b70..b6e1446 100644 --- a/docs/agent_memory/README_zh.md +++ b/docs/agent_memory/README_zh.md @@ -1,30 +1,98 @@ -# Agent Memory Evaluation + +# Agent Memory Evaluation（智能体记忆评测） -[English](./README.md) +[English](./README.md) · [文档索引](../README_zh.md) · [评测结果](./results_zh.md) -[文档索引](../README_zh.md) +## 概览 -AgentBench 是 OmniMemEval 中面向 Agent Runtime 的评测模块，用于评估 OpenClaw 在五个任务域上的任务完成能力，并支持在相同任务集合上独立评测记忆插件。当前提供以下评测协议： +本评测线使用 OmniMemEval 的 AgentBench 模块，评估智能体运行时（agent runtime） +在有无记忆插件（memory plugin）时的任务表现。 +当前公开的运行时适配器覆盖 OpenClaw 和五个任务域。 -- plain AgentBench：不启用待测记忆插件，执行 `test_only` 或 `train_then_test`。 -- memory plugin AgentBench：按 `memory_train_backup_test` 协议清理、训练、等待沉淀、备份、恢复并测试记忆。 +评测设置分为两类： -评测结果见 [results_zh.md](./results_zh.md)。当前架构和扩展契约见 -[architecture_zh.md](./architecture_zh.md)。 +- 基线：不启用待测记忆插件，使用 `test_only` 或 `train_then_test`。 +- 记忆插件：使用 `memory_train_backup_test`，包含清理、训练、等待沉淀、 + 备份、恢复和测试。 -## 数据来源声明 +首次运行建议按本文顺序阅读。已有环境可以直接查看 +[进阶运行](#进阶运行)、[输出与结果](#输出与结果)或[扩展与贡献](#扩展与贡献)。 +所有命令均在仓库根目录执行。 -AgentBench 使用 EverMind（Hugging Face/GitHub namespace: `EverMind-AI`）发布的 `EvoAgentBench` 数据集。数据集、任务划分和基础 benchmark 归属以 EverMind-AI/EvoAgentBench 及其数据集说明为准；本目录仅提供 OmniMemEval 侧的运行适配、配置和结果组织方式，不重新发布或声明拥有上述数据。 +## 环境与配置 -参考来源： + +### 安装依赖 -- HuggingFace 数据集：`EverMind-AI/EvoAgentBench` -- GitHub 项目：`EverMind-AI/EvoAgentBench` -- EvoAgentBench 覆盖的五个基础 benchmark：BrowseCompPlus、OmniMath、SWE-Bench、LiveCodeBench、GDPVal +为 Agent Memory 创建独立环境，不与 User Memory 混用： -数据规模依据 EvoAgentBench 的公开说明整理如下： +```bash +conda create -n agentmem python=3.12 -y +conda activate agentmem +python -m pip install -U pip +python -m pip install -r requirements_agentbench.txt +npm install -g openclaw +``` + +为 OpenClaw 配置可用的模型/provider，再确认 CLI 可调用： + +```bash +openclaw --version +openclaw agent --help +``` + + +### 配置凭证 + +复制 Agent Memory 模板，再填写所选任务域需要的配置： + +```bash +cp env_examples/.env.agent .env.agent +``` + +AgentBench 默认加载项目根目录的 `.env.agent`；`--env FILE` 可额外加载环境文件。 +这些凭证与 User Memory 的 `.env.memos` 等后端文件分开维护。 + +| 用途 | 配置 | +|---|---| +| 智能体模型/provider | `configs/agentbench/agents/openclaw.yaml`，使用 `LLM_BASE_URL` 和 `LLM_API_KEY` | +| 评分模型 | `JUDGE_MODEL`、`JUDGE_API_BASE`、`JUDGE_API_KEY` | +| 信息检索 embedding | `IR_EMBEDDING_ENDPOINT`、`IR_EMBEDDING_MODEL`、`IR_EMBEDDING_API_KEY` | +| 外部评测服务 | `EVALUATION_API_BASE`、`EVALUATION_API_KEY`、`EVALUATION_MODEL_OWNER`、`EVALUATION_MODEL_NAME`、`EVALUATION_TIMEOUT`、`EVALUATION_MAX_RETRIES` | + +字段说明以[环境模板](../../env_examples/.env.agent)和所选 +`configs/agentbench/domains/*.yaml` 为准。 +未通过智能体 YAML/环境变量配置的 provider 字段，可以回退到 +`~/.openclaw/openclaw.json` 中可解析的配置。不要提交填写了凭证的文件。 + +### 按需准备记忆插件 + +基线运行不需要待测记忆插件。使用 `memory_train_backup_test` 时， +先安装并配置该插件。例如，MemOS local plugin 提供以下安装脚本： + +```bash +curl -fsSL https://raw.githubusercontent.com/MemTensor/MemOS/main/apps/memos-local-plugin/install.sh | bash +``` + +执行前应审阅上游安装脚本。补齐生成的插件配置、凭证、本地路径和服务地址， +并确认插件在 OpenClaw 中已启用。 -| 域 | 基础 benchmark | Train | Test | +记忆插件运行强制使用 `runtime.home_mode: isolated_copy`，但 `home_links` +仍可能把真实插件数据链接到临时 OpenClaw home。 +生命周期中的清理/恢复命令可能删除或覆盖这些数据，并重启服务。 +请检查所选插件 YAML，使用独立评测数据，不要使用个人记忆。 + + +## 数据准备 + +AgentBench 使用 `EverMind-AI/EvoAgentBench` 数据集。 +数据归属、划分和基础 benchmark 定义遵循上游项目； +OmniMemEval 提供运行适配、配置、生命周期管理与结果组织。 + +- [Hugging Face 数据集](https://huggingface.co/datasets/EverMind-AI/EvoAgentBench) +- [上游项目](https://github.com/EverMind-AI/EvoAgentBench) + +| 任务域 | Benchmark | Train | Test | |---|---|---:|---:| | `information_retrieval` | BrowseCompPlus | 154 | 65 | | `reasoning` | OmniMath | 478 | 100 | @@ -32,9 +100,7 @@ AgentBench 使用 EverMind（Hugging Face/GitHub namespace: `EverMind-AI`）发 | `code_implementation` | LiveCodeBench | 97 | 39 | | `knowledge_work` | GDPVal | 87 | 58 | -## 数据准备 - -以下命令均在仓库根目录执行。首先下载公开数据集： +### 下载数据集 ```bash mkdir -p data/agentbench @@ -43,7 +109,7 @@ hf download EverMind-AI/EvoAgentBench \ --local-dir ./data/agentbench ``` -下载完成后，数据目录应包含以下任务数据： +预期目录： ```text data/agentbench/ @@ -56,16 +122,18 @@ data/agentbench/ `hf` CLI 随 `huggingface_hub` 提供，详见 [上游 CLI 指南](https://huggingface.co/docs/huggingface_hub/en/guides/cli)。 -各域数据路径由 `configs/agentbench/domains/*.yaml` 管理。若采用自定义数据目录，请同步更新相应 domain yaml 中的 `data_path`、`train_file`、`test_file`、`repo_root` 或同类字段。 +若使用自定义数据位置，请更新 `configs/agentbench/domains/*.yaml` 中的相应路径。 -LiveCodeBench 域依赖官方 verifier 包： +### 准备任务域特有依赖 + +只需额外准备所选任务域需要的工具。LiveCodeBench 使用官方验证器包： ```bash git clone https://github.com/LiveCodeBench/LiveCodeBench.git ./LiveCodeBench -pip install --no-deps -e ./LiveCodeBench +python -m pip install --no-deps -e ./LiveCodeBench ``` -BrowseComp-Plus dense 检索依赖数据预处理、索引文件以及 embedding 服务配置： +BrowseComp-Plus 的 dense 检索需要预处理、索引和 embedding 服务配置： ```bash python scripts/agentbench/utils/browsecomp-plus-tools/setup_data.py \ @@ -75,7 +143,8 @@ python scripts/agentbench/utils/browsecomp-plus-tools/build_dense_index.py \ --config configs/agentbench/domains/information_retrieval.yaml ``` -SWE-Bench 域依赖 Docker daemon，且容器环境需能够访问 PyPI/GitHub。GDPVal 的 PDF/表格任务依赖文档处理工具。Debian/Ubuntu 环境可参考以下命令安装系统依赖： +SWE-Bench 需要 Docker，评测容器需要访问 PyPI/GitHub； +GDPVal 需要 PDF/Office 处理工具。Debian/Ubuntu 可使用管理员权限安装所需系统包： ```bash apt-get update @@ -83,231 +152,112 @@ apt-get install -y docker.io poppler-utils libreoffice openjdk-21-jdk systemctl start docker ``` -其中 `openjdk-21-jdk` 主要用于兼容依赖 Pyserini/BM25 的检索流程；若仅使用已构建的 dense index，可根据实际配置决定是否安装。 - -## 运行环境安装 - -推荐使用独立 conda 环境 `agentmem`： - -```bash -conda create -n agentmem python=3.12 -y -conda activate agentmem -python -m pip install -U pip -pip install -r requirements_agentbench.txt -``` - -不要复用 User Memory 的环境运行 AgentBench。User Memory 与 AgentBench 依赖分别维护在独立的 requirements 文件中。 - -安装 OpenClaw CLI，并确保其位于当前 shell 的 `PATH` 中： - -```bash -npm install -g openclaw -``` - -如需运行记忆插件生命周期评测，需要先安装待测记忆插件。以下以 MemOS local plugin 为例： - -```bash -curl -fsSL https://raw.githubusercontent.com/MemTensor/MemOS/main/apps/memos-local-plugin/install.sh | bash -``` - -插件安装完成后，还需要完成 OpenClaw 和记忆插件配置，再进行验证。具体配置项取决于待测插件。以 MemOS local plugin 为例，安装脚本完成后，需要按插件要求补齐 OpenClaw 配置目录下生成的插件配置，并确认该插件在 OpenClaw 配置中处于启用状态。 +Java 与 Pyserini/BM25 检索有关；仅使用预构建 dense index 时可能不需要。 +这些是任务域特有要求，不是下文 reasoning 快速开始的额外前置条件。 -运行 AgentBench 前至少确认： +## 快速开始 -- OpenClaw 已配置可用的模型/provider。 -- 记忆插件已安装，并在 OpenClaw 配置中启用。 -- 插件所需的凭证、本地路径或服务 endpoint 已配置。 -- 下文所述 `.env.agent` 已准备完成。 - -安装和配置都完成后，验证 OpenClaw CLI 是否可以正常调用： - -```bash -openclaw --version -openclaw agent --help -``` - -运行 LiveCodeBench 域前，应完成上文的 `LiveCodeBench` 安装。SWE-Bench 域依赖 Docker；信息检索域依赖 embedding endpoint；GDPVal 域依赖 PDF/Office 文件处理工具。 - -域相关依赖： - -- BrowseComp-Plus dense 检索依赖 embedding 配置和索引文件；相关工具位于 `scripts/agentbench/utils/browsecomp-plus-tools/`。 -- GDPVal 依赖 PDF/表格处理工具，PDF 场景使用 `poppler-utils`，PPTX 场景可使用 `libreoffice`。 -- SWE-Bench 依赖 Docker，容器内评测脚本需访问 PyPI/GitHub。 -- LiveCodeBench 使用 `LiveCodeBench/` verifier。 - -## 环境与模型配置 - -AgentBench 默认读取项目根目录 `.env.agent`，也支持通过 `--env FILE` 指定额外环境变量文件。使用独立的 `.env.agent` 可以避免与 User Memory Evaluation 使用的 `.env.memos`、`.env.mem0` 等 backend-specific 配置混淆。 - -从模板创建 Agent Memory Evaluation 环境文件： - -```bash -cp env_examples/.env.agent .env.agent -``` - -OpenClaw 模型配置文件为： - -```text -configs/agentbench/agents/openclaw.yaml -``` - -常用 `.env.agent` 字段如下： +完成环境、凭证和数据准备后，运行一道 reasoning 任务： ```bash -LLM_BASE_URL=... -LLM_API_KEY=... - -JUDGE_MODEL=... -JUDGE_API_BASE=... -JUDGE_API_KEY=... - -IR_EMBEDDING_ENDPOINT=... -IR_EMBEDDING_MODEL=... -IR_EMBEDDING_API_KEY=... - -EVALUATION_API_BASE=... -EVALUATION_API_KEY=... -EVALUATION_MODEL_OWNER=... -EVALUATION_MODEL_NAME=... -EVALUATION_TIMEOUT=240 -EVALUATION_MAX_RETRIES=3 +./scripts/run_agent_eval.sh \ + --agent openclaw \ + --domain reasoning \ + --protocol test_only \ + --version smoke_agentbench \ + --test-split 1 \ + --trials 1 \ + --parallel 1 ``` -配置说明： +该命令调用真实模型/评分服务，可能产生费用。对于 reasoning， +`--test-split 1` 选择第一道测试题；省略它则选择完整测试集。 +`--parallel` 只控制并发，不限制题数。小规模检查不等于完整 benchmark 结果。 -- OpenClaw agent 的模型来自 `configs/agentbench/agents/openclaw.yaml`，其中 provider credential 由 `.env.agent` 注入。 -- 若某些 provider 字段未在 `.env.agent` 中配置，adapter 会回退到 `~/.openclaw/openclaw.json` 中可解析的配置。 -- memory plugin 协议下始终使用 `runtime.home_mode: isolated_copy`，并通过 memory plugin 配置中的 `home_links` 把必要插件目录链接到临时 `OPENCLAW_HOME`。 +完整协议见[进阶运行](#进阶运行)。Runner 的 `--help` 提供 CLI 选项说明。 -## 目录结构 +## 进阶运行 -```text -scripts/agentbench/ - run_agent_eval.py # 单域评测入口 - runner.py # phase/trial/retry 调度 - memory_lifecycle.py # 记忆插件清理、备份、恢复命令执行 - feedback.py # train 后 verifier feedback prompt - memos_feedback.py # 插件侧 structured feedback 适配 - agents/openclaw.py # OpenClaw adapter - domains/ # 五个 domain adapter - -configs/agentbench/ - agents/openclaw.yaml - domains/*.yaml - memory_plugins/*.yaml - -results/agentbench/ - --/ - train/ - test_run_1/ - memory_lifecycle.log - memory_lifecycle.json -``` + +### 评测协议 -## 测评协议 +以下命令使用配置中的完整数据划分。 -### `test_only` +#### `test_only` -以下命令执行完整 test split。需要单任务 reasoning smoke 检查时，追加 -`--test-split 1` 以选择第一道测试题；`--parallel` 只控制并发，不限制题数。 -运行前完成上文的环境、凭证和数据准备。 +运行测试集： ```bash ./scripts/run_agent_eval.sh \ - --agent openclaw \ - --domain reasoning \ - --protocol test_only \ - --version baseline_reasoning_test \ - --trials 1 \ - --parallel 1 + --agent openclaw --domain reasoning --protocol test_only \ + --version baseline_reasoning_test --trials 1 --parallel 1 ``` -### `train_then_test` +#### `train_then_test` -依次执行 train split 和 test split；该协议不执行记忆插件生命周期。 +先训练再测试，不执行记忆插件生命周期命令： ```bash ./scripts/run_agent_eval.sh \ - --agent openclaw \ - --domain reasoning \ - --protocol train_then_test \ - --version train_then_test_reasoning \ - --trials 1 \ - --parallel 1 + --agent openclaw --domain reasoning --protocol train_then_test \ + --version train_then_test_reasoning --trials 1 --parallel 1 ``` -### `memory_train_backup_test` +#### `memory_train_backup_test` -该协议用于独立评测记忆插件，结果目录与 baseline 评测结果分离。流程如下： +记忆插件与基线分开评测： -1. 设置插件为 train 模式。 -2. 清理记忆。 -3. 执行 train split。 -4. train verifier 完成后，将 feedback 作为同一 OpenClaw session 的下一轮消息提交。 -5. 若插件配置启用 structured feedback，则提交插件侧显式反馈，并保证 task turn 和 feedback turn 保持同一 session/episode 语义。 -6. 等待插件沉淀/进化，默认由 `settle_seconds` 或 `wait_settle` 配置控制。 -7. 备份当前域的记忆。 -8. 每次 test 前都恢复该域备份，即使插件支持关闭写入也仍恢复。 -9. 执行 test split，并写入独立结果目录。 +1. 设置插件为训练模式并清理记忆。 +2. 执行训练。 +3. 在同一 OpenClaw 会话的下一轮提交验证器反馈。 +4. 如已启用，按相同会话/episode 语义提交插件侧结构化反馈。 +5. 等待记忆沉淀/进化。 +6. 备份当前任务域记忆。 +7. 每次测试前恢复该域备份，即使已禁止测试写入也仍恢复。 +8. 执行测试，将结果写入 `results/agentbench/`。 -单域运行： +单任务域： ```bash ./scripts/run_agent_eval.sh \ - --agent openclaw \ - --domain reasoning \ - --protocol memory_train_backup_test \ - --memory-plugin memos \ - --version memos_reasoning_eval \ - --test-runs 1 \ - --trials 1 \ - --parallel 1 + --agent openclaw --domain reasoning --protocol memory_train_backup_test \ + --memory-plugin memos --version memos_reasoning_eval \ + --test-runs 1 --trials 1 --parallel 1 ``` -五域顺序运行： +全部五个任务域： ```bash ./scripts/run_agentbench_memory_train_backup_test.sh \ - --memory-plugin memos \ - --version memos_5domain_eval \ - --test-runs 1 \ - --trials 1 \ - --parallel 1 + --memory-plugin memos --version memos_5domain_eval \ + --test-runs 1 --trials 1 --parallel 1 ``` -指定域子集： +任务域子集： ```bash ./scripts/run_agentbench_memory_train_backup_test.sh \ - --memory-plugin memos \ - --domains reasoning,code_implementation \ + --memory-plugin memos --domains reasoning,code_implementation \ --version memos_reasoning_code_eval ``` -## 记忆插件配置 +### 记忆插件配置 -插件生命周期配置位于： - -```text -configs/agentbench/memory_plugins/ - memos.yaml - everos.yaml - openviking.yaml - supermemory.yaml - hindsight.yaml -``` +生命周期配置位于 `configs/agentbench/memory_plugins/`： +`memos.yaml`、`everos.yaml`、`openviking.yaml`、`supermemory.yaml` +和 `hindsight.yaml`。 每个配置负责声明： -- `plugin`：插件标签，会进入结果目录名。 +- `plugin`：结果目录中的插件标签。 - `backup_dir` 和 `backup_file_template`：备份位置。 -- `settle_seconds` 或 `commands.wait_settle`：训练后等待沉淀/进化的逻辑。 -- `home_links`：在 isolated OpenClaw home 中需链接的插件目录。 -- `modes.train/test` 或 `commands.set_mode_*`：训练/测试时插件读写模式。 -- `commands.clear/backup/restore`：清理、备份、恢复记忆。 -- `feedback`：是否启用 train feedback、timeout，以及插件侧 structured feedback。 +- `settle_seconds` 或 `commands.wait_settle`：训练后的等待沉淀逻辑。 +- `home_links`：链接到隔离 OpenClaw home 的插件路径。 +- `modes.train/test` 或 `commands.set_mode_*`：读写模式。 +- `commands.clear/backup/restore`：生命周期操作。 +- `feedback`：训练反馈和可选的插件侧结构化反馈。 -以 `memos.yaml` 为例，structured feedback 配置如下： +例如，`memos.yaml` 启用结构化反馈： ```yaml feedback: @@ -317,14 +267,18 @@ feedback: memos_submit_timeout: 900 ``` -其他插件默认仅执行普通 train feedback，不执行 structured submit；需要显式提交反馈的插件可在对应 yaml 中开启相关开关。 +其他插件配置默认使用普通训练反馈，不执行上述结构化提交。 +请检查所选 YAML，不要假定所有插件行为相同。 -## 输出与结果读取 + + + +## 输出与结果 -每个 domain 的输出目录格式如下： +记忆插件协议的输出结构如下： ```text -results/agentbench/openclaw-memos-memos_5domain_eval-reasoning/ +results/agentbench/--/ experiment_config.json memory_lifecycle.log memory_lifecycle.json @@ -348,12 +302,37 @@ results/agentbench/openclaw-memos-memos_5domain_eval-reasoning/ 关键字段： -- `summary.json`：`pass@1`、平均 reward、平均耗时。 -- `result.json.agent_result`：OpenClaw 完成状态、耗时、是否从 session/trajectory 恢复。 -- `result.json.feedback_result`：train 后同 session feedback turn 状态。 -- `result.json.memos_feedback_result`：插件侧 structured feedback 状态。 -- `memory_lifecycle.json`：插件 clear/backup/restore 等命令事件。 +- `summary.json`：`pass@1`、平均 reward 和平均耗时。 +- `result.json.agent_result`：完成状态、耗时和会话恢复信息。 +- `result.json.feedback_result`：同会话训练反馈状态。 +- `result.json.memos_feedback_result`：启用时的插件侧结构化反馈状态。 +- `memory_lifecycle.json`：清理、备份与恢复事件。 + +公开评测设置与表格见[评测结果](./results_zh.md)。 +快照也包含 Hermes 结果，但当前公开运行时适配器覆盖 OpenClaw， +并未提供 Hermes 的复现路径。模型、评分与 embedding 服务可用性均可能影响单条奖励分数。 + +## 扩展与贡献 + +[架构与扩展契约](./architecture_zh.md)说明调度、会话标识、结果字段， +以及智能体/任务域扩展点。Agent Memory 适配器与 User Memory 的 +`scripts/client_factory/` 层分离。 + +PR 流程和证据要求见 [CONTRIBUTING.md（英文）](../../CONTRIBUTING.md)。 +代码、配置与数据准备脚本的位置见根目录的[仓库结构](../../README_zh.md#目录结构)。 + +## 验证与许可证 + +在独立的 Python 3.11 或 3.12 开发环境运行离线检查： + +```bash +python -m pip install -r requirements-dev.txt +python -m pytest -q +python scripts/check_doc_links.py +``` -## 验证结果 +[测试指南（英文）](../../scripts/tests/README.md)区分离线测试、仓库检查和真实后端集成测试。 +离线 CI 不验证 benchmark 分数，也不调用真实服务。 -5 域评测结果见 [results_zh.md](./results_zh.md)。模型服务、外部 judge 服务和 embedding 服务状态均可能影响单条样本 reward。 +代码许可证见 [LICENSE](../../LICENSE)。外部数据集保留上游许可证，详见 +[THIRD_PARTY_NOTICES.md（英文）](../../THIRD_PARTY_NOTICES.md)。 diff --git a/docs/agent_memory/architecture.md b/docs/agent_memory/architecture.md index 5109d7b..5d1eb62 100644 --- a/docs/agent_memory/architecture.md +++ b/docs/agent_memory/architecture.md @@ -1,8 +1,7 @@ # Agent Memory Evaluation Architecture -[中文版](./architecture_zh.md) - -[Documentation index](../README.md) +[中文版](./architecture_zh.md) · [Documentation index](../README.md) · +[Guide](./README.md) · [Results](./results.md) This document describes the current Agent Memory Evaluation architecture and its public extension contracts. Operational setup and evaluation commands remain @@ -13,7 +12,7 @@ in the [Agent Memory Evaluation guide](./README.md). Agent Memory Evaluation evaluates the following combination: ```text -Agent Runtime + Memory Plugin + Task Domain + Verifier +Agent Runtime + Optional Memory Plugin + Task Domain + Verifier ``` The AgentBench entrypoint owns protocol and memory-lifecycle ordering, while the diff --git a/docs/agent_memory/architecture_zh.md b/docs/agent_memory/architecture_zh.md index e9bbefa..46a5b2f 100644 --- a/docs/agent_memory/architecture_zh.md +++ b/docs/agent_memory/architecture_zh.md @@ -1,8 +1,8 @@ -# Agent Memory Evaluation 架构 + +# Agent Memory Evaluation（智能体记忆评测）架构 -[English](./architecture.md) - -[文档索引](../README_zh.md) +[English](./architecture.md) · [文档索引](../README_zh.md) · +[运行指南](./README_zh.md) · [评测结果](./results_zh.md) 本文档说明当前 Agent Memory Evaluation 的架构和公开扩展契约。环境准备和评测命令 仍以 [Agent Memory Evaluation 指南](./README_zh.md)为准。 @@ -12,11 +12,11 @@ Agent Memory Evaluation 评估以下组合： ```text -Agent Runtime + Memory Plugin + Task Domain + Verifier +智能体运行时 + 可选记忆插件 + 任务域 + 验证器 ``` -AgentBench 入口负责协议和记忆生命周期的执行顺序，Runner 负责任务、Trial、Retry -和 Feedback 的执行。Runtime 相关行为由 `AgentAdapter` 实现；任务加载、Prompt +AgentBench 入口负责协议和记忆生命周期的执行顺序，调度器负责任务、试次（trial）、 +重试和反馈的执行。运行时相关行为由 `AgentAdapter` 实现；任务加载、提示词 构造、环境准备和验证由 `DomainAdapter` 实现。记忆插件生命周期操作通常配置在 `configs/agentbench/memory_plugins/` 下。 @@ -24,37 +24,38 @@ AgentBench 入口负责协议和记忆生命周期的执行顺序，Runner 负 ### `AgentAdapter` -[`AgentAdapter`](../../scripts/agentbench/agents/base.py) 定义 Runner 如何驱动一个 -Agent Runtime。它负责构造 Session 元数据、调用 Runtime、收集 Session 产物和 -Token 用量，以及判断可重试的 Runtime 结果。 +[`AgentAdapter`](../../scripts/agentbench/agents/base.py) 定义调度器如何驱动一个 +智能体运行时。它负责构造会话元数据、调用运行时、收集会话产物和 +token 用量，以及判断运行时结果是否可重试。 -当前公开 Agent Registry 注册了 +当前公开智能体注册表包含 [`OpenClawAgentAdapter`](../../scripts/agentbench/agents/openclaw.py)。 ### `DomainAdapter` [`DomainAdapter`](../../scripts/agentbench/domains/base.py) 定义任务域如何加载任务、 -准备单任务资源、构造 Prompt、验证 Agent 输出并清理任务状态。 +准备单任务资源、构造提示词、验证智能体输出并清理任务状态。 -当前 Domain Registry 覆盖 reasoning、information retrieval、knowledge work、 -code implementation 和 software engineering。 +当前任务域注册表覆盖 `reasoning`、`information_retrieval`、`knowledge_work`、 +`code_implementation` 和 `software_engineering`。 -### Runner 与记忆生命周期 + +### 调度器与记忆生命周期 [`run_agent_eval.py`](../../scripts/agentbench/run_agent_eval.py) 选择评测协议，并负责 -Phase 和记忆生命周期的执行顺序。 +评测阶段和记忆生命周期的执行顺序。 -[`runner.py`](../../scripts/agentbench/runner.py) 执行任务和 Trial、写入单次 Trial -产物、应用重试策略并生成 Phase 汇总。 +[`runner.py`](../../scripts/agentbench/runner.py) 执行任务和试次、写入单次试次 +产物、应用重试策略并生成阶段汇总。 [`memory_lifecycle.py`](../../scripts/agentbench/memory_lifecycle.py) 执行配置中的插件 -生命周期命令。`memory_train_backup_test` 协议负责 validation、模式切换、清理、 -训练、等待沉淀、备份、恢复和测试的执行顺序。可选的 Structured Feedback 集成 -仍在 Runner 和插件配置中显式声明。 +生命周期命令。`memory_train_backup_test` 协议负责验证、模式切换、清理、 +训练、等待沉淀、备份、恢复和测试的执行顺序。可选的结构化反馈集成 +仍在调度器和插件配置中显式声明。 ## 评测执行流程 -每个任务 Trial 按以下顺序执行： +每个任务试次按以下顺序执行： ```text domain.setup @@ -70,40 +71,41 @@ agent.cleanup_task write result.json ``` -Domain 级 initialize 和 finalize 包围完整 Phase。全部选定任务和 Trial 完成后生成 -Phase 汇总。 +任务域级初始化（initialize）和收尾（finalize）包围完整评测阶段。 +全部选定任务和试次完成后生成阶段汇总。 -## Session 标识契约 + +## 会话标识契约 [`SessionSpec`](../../scripts/agentbench/session.py) 将不同用途的标识分开： -- `cli_session_id`：经过清理、可被 Agent CLI 接受的标识。 -- `semantic_session_id`：保留 Benchmark Phase、Domain、Split、Task 和 Trial +- `cli_session_id`：经过清理、可被智能体 CLI 接受的标识。 +- `semantic_session_id`：保留 benchmark 阶段、任务域、数据划分、任务和试次 语义的标识。 -- `source_ref`：稳定的 Benchmark 来源引用。 -- `metadata`：Benchmark、Phase、Domain、Split、Task、Trial 和 Agent 的结构化 +- `source_ref`：稳定的 benchmark 来源引用。 +- `metadata`：benchmark、阶段、任务域、数据划分、任务、试次和智能体的结构化 元数据。 -- `agent_session_ref`：可选的 Runtime 专用 Session 引用。 -- `openclaw_session_key`：使用 OpenClaw 时对应的 Session Key。 -- `openclaw_gateway_session_id`：对应的 OpenClaw Gateway 标识。 +- `agent_session_ref`：可选的运行时专用会话引用。 +- `openclaw_session_key`：使用 OpenClaw 时对应的会话 key。 +- `openclaw_gateway_session_id`：对应的 OpenClaw 网关标识。 -OpenClaw Adapter 将 `cli_session_id` 传给 `openclaw agent --session-id`，并默认通过 +OpenClaw 适配器将 `cli_session_id` 传给 `openclaw agent --session-id`，并默认通过 `OMNIMEMEVAL_AGENT_CONTEXT` 暴露序列化后的完整 `SessionSpec`。这样可以让 CLI 安全标识与插件和结果元数据保持对应。 ## 结果与指标契约 -每个 Trial 写入一个 `result.json`，其中包含 Task、Phase、Split、Trial、Session、 -Agent 结果、Verifier 结果、Feedback 结果、Token 用量、时间戳和异常信息。辅助产物 -可以包括 `response.txt`、`session.jsonl` 和 Verifier 输出。 +每个试次写入一个 `result.json`，包含任务、阶段、数据划分、试次、会话、 +智能体结果、验证器结果、反馈结果、token 用量、时间戳和异常信息。辅助产物 +可以包括 `response.txt`、`session.jsonl` 和验证器输出。 -每个 Phase 写入 `summary.json` 和 `report.md`。汇总包括任务数和 Trial 数、 -`pass@1`、配置的 `pass@n`、平均通过率、Reward、耗时、Turn、Token、Failure -Class、排除基础设施错误后的统计，以及 Domain 专用指标。 +每个阶段写入 `summary.json` 和 `report.md`。汇总包括任务数和试次数、 +`pass@1`、配置的 `pass@n`、平均通过率、奖励分数、耗时、轮次、token 用量、 +失败类别、排除基础设施错误后的统计，以及任务域专用指标。 ## 扩展点 -- 新增 Agent Runtime：实现 `AgentAdapter`，并注册到 +- 新增智能体运行时：实现 `AgentAdapter`，并注册到 `scripts/agentbench/agents/__init__.py`。 - 新增任务域：实现 `DomainAdapter`，并注册到 `scripts/agentbench/domains/__init__.py`。 @@ -111,5 +113,5 @@ Class、排除基础设施错误后的统计，以及 Domain 专用指标。 `configs/agentbench/memory_plugins/` 配置；当通用命令生命周期不足时，使用显式 集成代码。 -Agent Memory Evaluation 与 User Memory 的 `scripts/client_factory/` Adapter 层保持 +Agent Memory Evaluation 与 User Memory 的 `scripts/client_factory/` 适配器层保持 分离，因为两条评测线评估的是不同系统边界。 diff --git a/docs/agent_memory/results.md b/docs/agent_memory/results.md index b3cae73..f18b7ea 100644 --- a/docs/agent_memory/results.md +++ b/docs/agent_memory/results.md @@ -1,22 +1,27 @@ -# AgentBench Evaluation Results + +# Agent Memory Evaluation Results -[Chinese](./results_zh.md) - -[Agent Memory Evaluation guide](./README.md) · -[Architecture](./architecture.md) +[中文版](./results_zh.md) · [Documentation index](../README.md) · +[Guide](./README.md) · [Architecture](./architecture.md) These tables record OpenClaw and Hermes evaluation setups. The current public Agent Memory runner guide and registered runtime adapter cover OpenClaw. +This page is the canonical source for numeric result tables; the Chinese page +provides translated explanations and links to the same tables. ## Metrics -> Acc: the average single-run pass rate across 3 independent runs of the same task, used to measure stable task completion in one run. - -> Avg turns: the average number of model response turns triggered by the agent per task, used to measure interaction and reasoning depth. +| Metric | Meaning | +|---|---| +| Acc (%) | Average single-run pass rate across 3 independent runs of the same task | +| Avg turns | Average model response turns per task | +| Avg chars | Average response-text character count per task | -> Avg chars: the average character length of the agent's response text, used to measure output length. +Acc measures single-run task completion, not pass-at-3. Numeric cells are shown +without ranking or statistical-significance markers; no significance test is +documented in this snapshot. Service availability can affect per-sample rewards. -## Data And Evaluation Setup +## Data and Evaluation Setup Dataset source and split logic: @@ -42,7 +47,7 @@ Evaluation judge model: qwen3.6-flash in thinking mode. ### OpenClaw -| **Method** | BrowseComp-Plus Acc | BrowseComp-Plus Avg turns | OmniMath
Acc | OmniMath Cost
Avg chars | SWE-Bench
Acc | SWE-Bench
Avg turns | LiveCodeBench
Acc | LiveCodeBench Avg turns | GDPVal
Acc | GDPVal
Avg turns | +| Method | BrowseComp-Plus Acc (%) | BrowseComp-Plus Avg turns | OmniMath Acc (%) | OmniMath Avg chars | SWE-Bench Acc (%) | SWE-Bench Avg turns | LiveCodeBench Acc (%) | LiveCodeBench Avg turns | GDPVal Acc (%) | GDPVal Avg turns | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | **Baseline** | 18.46 | 35.1 | 52 | 5658.6 | 26.92 | 58.8 | 51.28 | 23.9 | 34.48 | 17.2 | | **Mem0** | 13.33 | 36.4 | 53 | 6090.53 | 26.92 | 55.78 | 60.68 | 6.27 | 41.38 | 17.28 | @@ -51,11 +56,11 @@ Evaluation judge model: qwen3.6-flash in thinking mode. | **Hindsight** | 16.92 | 35.6 | 55 | 6177 | 38.46 | 56.7 | 71.79 | 14.2 | 50.00 | 17.5 | | **mem9**
**(Cloud service)** | 12.31 | 31.6 | 54.33 | 5880.9 | 30.77 | 59.1 | 64.1 | 8.23 | 47.7 | 15.8 | | **Memori**
**(Cloud service)** | 16.92 | 36.3 | 52 | 6189.2 | 34.62 | 43.1 | 51.28 | 9.3 | 37.93 | 15.8 | -| **MemOS** | **23.85*** | 43.3 | **61*** | 5164.2 | **38.46*** | 49.55 | 64.96 | 9.43 | **62.07*** | 23.93 | +| **MemOS** | 23.85 | 43.3 | 61 | 5164.2 | 38.46 | 49.55 | 64.96 | 9.43 | 62.07 | 23.93 | ### Hermes -| **Method** | BrowseComp-Plus Acc | BrowseComp-Plus Avg turns | OmniMath
Acc | OmniMath Cost
Avg chars | SWE-Bench
Acc | SWE-Bench
Avg turns | LiveCodeBench
Acc | LiveCodeBench Avg turns | GDPVal
Acc | GDPVal
Avg turns | +| Method | BrowseComp-Plus Acc (%) | BrowseComp-Plus Avg turns | OmniMath Acc (%) | OmniMath Avg chars | SWE-Bench Acc (%) | SWE-Bench Avg turns | LiveCodeBench Acc (%) | LiveCodeBench Avg turns | GDPVal Acc (%) | GDPVal Avg turns | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | **Baseline** | 18.97 | 43.23 | 62 | 10872.8 | 37.18 | 84.93 | 52.99 | 22.27 | 54.6 | 62.57 | | **Mem0** | 17.95 | 42.73 | 64.33 | 11740 | 34.62 | 94.37 | 57.26 | 25.63 | 56.9 | 57.73 | @@ -64,4 +69,13 @@ Evaluation judge model: qwen3.6-flash in thinking mode. | **Viking** | 21.54 | 39.33 | 58.67 | 12514.20 | 35.90 | 88.70 | 60.68 | 27.77 | 58.05 | 54.27 | | **mem9**
**(Cloud service)** | 18.46 | 40.13 | 66 | 12419.4 | 39.74 | 98.4 | 58.97 | 18.77 | 55.75 | 53.17 | | **Memori**
**(Cloud service)** | 14.87 | 40.13 | 64.33 | 14811 | 28.21 | 91.5 | 54.7 | 22.37 | 55.75 | 53.17 | -| **MemOS** | 20.75 | 36.20 | **72.67*** | 5915.7 | **52.56*** | 76.37 | 64.1 | 25.63 | 55.17 | 58.6 | +| **MemOS** | 20.75 | 36.20 | 72.67 | 5915.7 | 52.56 | 76.37 | 64.1 | 25.63 | 55.17 | 58.6 | + +## Contributing Results + +Include the evaluated runtime/plugin versions, protocol, task scope, run/trial +counts, configuration, failure handling, aggregation rules, and stable public +artifacts sufficient to recompute the reported metrics. See the +[result evidence requirements](../../CONTRIBUTING.md#submitting-benchmark-results). +An entry in this historical snapshot does not imply that its complete setup is +supported by the current public runner. diff --git a/docs/agent_memory/results_zh.md b/docs/agent_memory/results_zh.md index b8d350a..1f1fff9 100644 --- a/docs/agent_memory/results_zh.md +++ b/docs/agent_memory/results_zh.md @@ -1,67 +1,56 @@ -# AgentBench 评测结果 + +# Agent Memory Evaluation（智能体记忆评测）结果 -[English](./results.md) +[English / 完整结果表](./results.md) · [文档索引](../README_zh.md) · +[运行指南](./README_zh.md) · [架构](./architecture_zh.md) -[Agent Memory Evaluation 指南](./README_zh.md) · -[架构](./architecture_zh.md) +本文提供中文指标说明与结果表入口。数值统一维护在 +[英文结果页](./results.md)，避免中英文表格更新不同步。 -以下表格记录 OpenClaw 和 Hermes 两套评测设置。当前公开的 Agent Memory Runner -指南和已注册 Runtime Adapter 覆盖 OpenClaw。 +该快照记录 OpenClaw 和 Hermes 两套评测设置。 +当前公开运行指南和已注册运行时适配器覆盖 OpenClaw，并未提供 Hermes 的复现路径。 ## 评测指标 -> Acc：统计同一任务 3 次独立运行中的平均一次通过率，用于衡量单次运行的稳定完成情况。 +| 指标 | 含义 | +|---|---| +| Acc（%） | 同一任务 3 次独立运行的平均单次通过率 | +| Avg turns | 每个任务平均触发的模型回复轮次 | +| Avg chars | 每个任务回复文本的平均字符数 | -> Avg turns：表示每个任务中 agent 平均触发的模型回复轮次，用于衡量完成任务过程中的交互/推理迭代深度。 - -> Avg chars：Agent 回答文本的平均字符长度，用于衡量输出答案的文本长度。 +Acc 衡量单次运行的完成表现，不是 pass-at-3。 +表格数值不使用排名或统计显著性标记；当前快照没有提供显著性检验说明。 +服务可用性可能影响单条奖励分数。 ## 数据与评测设置 -数据集来源及切分逻辑： - -数据参考：[https://huggingface.co/datasets/EverMind-AI/EvoAgentBench](https://huggingface.co/datasets/EverMind-AI/EvoAgentBench) - -测评 Agent：OpenClaw 和 Hermes + 对应产品的插件或集成。 - -baseline 是指对应被测 Agent 不加任何插件运行的结果。 - -OpenClaw 评测使用的 OpenClaw 版本为 2026.5.7。 +数据来自 [EverMind-AI/EvoAgentBench](https://huggingface.co/datasets/EverMind-AI/EvoAgentBench)。 +基线是对应智能体不启用记忆插件的运行结果；未标明云服务的产品使用本地服务和插件或集成。 -Hermes 评测使用的 Hermes 版本为 Hermes Agent v0.18.0（2026.7.1）。 +快照记录的配置如下： -未标明云服务的产品，均使用本地部署的服务和对应 Agent 的插件或集成。 +- OpenClaw：2026.5.7。 +- Hermes：Hermes Agent v0.18.0（2026.7.1）。 +- MemOS：Memos-Local-Plugin 2.0.8。 +- 回答模型：qwen3.6-flash，no_thinking 模式。 +- 评分模型：qwen3.6-flash，thinking 模式。 -MemOS 使用 Memos-Local-Plugin 2.0.8 版本。 - -OpenClaw 和 Hermes 回答模型配置：qwen3.6-flash no_thinking 模式。 - -评测判别模型：qwen3.6-flash thinking 模式。 +这些是快照配置，不代表当前安装命令会自动得到相同版本。 +完整设置见[英文说明](./results.md#data-and-evaluation-setup)。 ## 结果 ### OpenClaw -| **Method** | BrowseComp-Plus Acc | BrowseComp-Plus Avg turns | OmniMath
Acc | OmniMath Cost
Avg chars | SWE-Bench
Acc | SWE-Bench
Avg turns | LiveCodeBench
Acc | LiveCodeBench Avg turns | GDPVal
Acc | GDPVal
Avg turns | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| **Baseline** | 18.46 | 35.1 | 52 | 5658.6 | 26.92 | 58.8 | 51.28 | 23.9 | 34.48 | 17.2 | -| **Mem0** | 13.33 | 36.4 | 53 | 6090.53 | 26.92 | 55.78 | 60.68 | 6.27 | 41.38 | 17.28 | -| **EverOS** | 18.98 | 30.8 | 54.67 | 6085.87 | 32.05 | 54.77 | 59.83 | 7.2 | 38.51 | 16.97 | -| **Supermemory** | 14.36 | 32.03 | 58 | 5632.1 | 30.7 | 51.37 | 59.83 | 6.83 | 43.10 | 16.70 | -| **Hindsight** | 16.92 | 35.6 | 55 | 6177 | 38.46 | 56.7 | 71.79 | 14.2 | 50.00 | 17.5 | -| **mem9**
**（云服务）** | 12.31 | 31.6 | 54.33 | 5880.9 | 30.77 | 59.1 | 64.1 | 8.23 | 47.7 | 15.8 | -| **Memori**
**（云服务）** | 16.92 | 36.3 | 52 | 6189.2 | 34.62 | 43.1 | 51.28 | 9.3 | 37.93 | 15.8 | -| **MemOS** | **23.85*** | 43.3 | **61*** | 5164.2 | **38.46*** | 49.55 | 64.96 | 9.43 | **62.07*** | 23.93 | +查看 [OpenClaw 完整结果表](./results.md#openclaw)，包含五个任务域的准确率和交互成本指标。 ### Hermes -| **Method** | BrowseComp-Plus Acc | BrowseComp-Plus Avg turns | OmniMath
Acc | OmniMath Cost
Avg chars | SWE-Bench
Acc | SWE-Bench
Avg turns | LiveCodeBench
Acc | LiveCodeBench Avg turns | GDPVal
Acc | GDPVal
Avg turns | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| **Baseline** | 18.97 | 43.23 | 62 | 10872.8 | 37.18 | 84.93 | 52.99 | 22.27 | 54.6 | 62.57 | -| **Mem0** | 17.95 | 42.73 | 64.33 | 11740 | 34.62 | 94.37 | 57.26 | 25.63 | 56.9 | 57.73 | -| **Supermemory** | 18.46 | 41.03 | 64.00 | 10675.90 | 35.90 | 92.23 | 66.67 | 19.73 | 57.47 | 58.60 | -| **Hindsight** | 21.54 | 48.33 | 64.00 | 12089.63 | 28.20 | 84.47 | 61.54 | 30.07 | 57.47 | 59.67 | -| **Viking** | 21.54 | 39.33 | 58.67 | 12514.20 | 35.90 | 88.70 | 60.68 | 27.77 | 58.05 | 54.27 | -| **mem9**
**（云服务）** | 18.46 | 40.13 | 66 | 12419.4 | 39.74 | 98.4 | 58.97 | 18.77 | 55.75 | 53.17 | -| **Memori**
**（云服务）** | 14.87 | 40.13 | 64.33 | 14811 | 28.21 | 91.5 | 54.7 | 22.37 | 55.75 | 53.17 | -| **MemOS** | 20.75 | 36.20 | **72.67*** | 5915.7 | **52.56*** | 76.37 | 64.1 | 25.63 | 55.17 | 58.6 | +查看 [Hermes 完整结果表](./results.md#hermes)。 +此表保留历史评测记录，不表示当前公开 runner 支持该运行时的完整复现。 + +## 提交结果 + +请提供运行时/插件版本、协议、任务范围、运行与 trial 次数、配置、失败处理、 +汇总方式，以及足以重算指标的稳定公开产物。 +详见[结果证据要求（英文）](../../CONTRIBUTING.md#submitting-benchmark-results)。 diff --git a/docs/user_memory/README.md b/docs/user_memory/README.md index c502fec..cedbc89 100644 --- a/docs/user_memory/README.md +++ b/docs/user_memory/README.md @@ -1,121 +1,69 @@ # User Memory Evaluation -[中文版](./README_zh.md) +[中文版](./README_zh.md) · [Documentation index](../README.md) · [Results](./results.md) -[Documentation index](../README.md) +## Overview -This document covers the User Memory Evaluation track in OmniMemEval. This track evaluates memory backend APIs through a shared benchmark pipeline and a common adapter layer. Users can switch memory backends with `--lib` and compare mainstream memory products, self-hosted memory frameworks, and custom adapters under the same benchmark flow. +This track evaluates memory backend APIs through a shared benchmark pipeline +and adapter layer. Select a backend with `--lib` to evaluate memory products, +self-hosted frameworks, or custom adapters under the same benchmark flow. -The adapter layer covers 14 mainstream memory solutions through 15 adapter entries, including MemOS, Mem0, Zep/Graphiti, Supermemory, EverOS, Letta, Hindsight, Cognee, Viking Memory, Memori, MemMachine, MemoryLake, Backboard.io, and mem9. This track supports five benchmark tasks: LoCoMo, LongMemEval, BEAM, PersonaMem v2, and HaluMem. These tasks cover complementary -long-term memory capabilities, including conversation recall, cross-session -updates, large-scale retrieval, personalization, and robustness under -hallucination, conflict, and dynamic-update scenarios. +There are 15 adapter entries covering 14 memory solutions (Zep and Graphiti +have separate entries). Supported benchmarks are LoCoMo, LongMemEval, BEAM, +PersonaMem v2, and HaluMem. -Benchmark coverage: +Read this guide in order for a first run. For an existing setup, jump to +[advanced runs](#advanced-runs), [outputs and results](#outputs-and-results), +or [extensions and contribution](#extensions-and-contribution). +Run all commands from the repository root. -- [LoCoMo](#locomo): long-conversation QA for multi-hop recall, temporal - reasoning, and open-domain memory use. -- [LongMemEval](#longmemeval): cross-session long-term memory with knowledge - updates, temporal reasoning, and preference questions. -- [BEAM](#beam): large-scale memory retrieval from 128K to 10M token contexts. -- [PersonaMem v2](#personamem-v2): personalized memory evaluation focused on - preferences, sensitive information, and user-specific behavior. -- [HaluMem](#halumem): robustness evaluation for memory hallucination, boundary - detection, conflicts, multi-hop inference, and dynamic updates. +## Environment and configuration -## Pipeline + +### Install dependencies -OmniMemEval benchmark pipelines use the same staged flow: - -```text -┌──────────────────┐ -│ Benchmark Data │ -│ dataset-specific │ -└────────┬─────────┘ - │ - ▼ -┌──────────────────┐ add() ┌──────────────────┐ -│ 1. Ingest ├─────────────────▶│ Memory Backend │ -│ conversations │ │ selected by --lib│ -└────────┬─────────┘ └────────┬─────────┘ - │ │ - ▼ │ search() -┌──────────────────┐ │ -│ 2. Search │◀──────────────────────────┘ -│ retrieve context │ -└────────┬─────────┘ - │ - ▼ -┌──────────────────┐ ANSWER LLM -│ 3. Answer ├─────────────────▶ generated answers -│ generation │ -└────────┬─────────┘ - │ - ▼ -┌──────────────────┐ EVAL LLM / NLP -│ 4. Evaluation ├─────────────────▶ judged records -│ LLM-as-Judge │ -└────────┬─────────┘ - │ - ▼ -┌──────────────────┐ -│ 5. Metrics │ -│ accuracy/latency │ -└────────┬─────────┘ - │ - ▼ -┌──────────────────┐ -│ 6. Report │ -│ markdown/results │ -└──────────────────┘ -``` - -- Ingest calls the selected memory client `add()`. -- Search calls the selected memory client `search()`. -- Answer generation uses an OpenAI-compatible ANSWER model. -- Evaluation uses an OpenAI-compatible EVAL model for LLM-as-Judge plus NLP metrics. -- Metrics and reports are written under `results//-/`. - -The shell runners and Python stages support checkpoint/resume so interrupted -runs can continue from the last completed step. - -## Quick Start - -### 1. Create Environment +Use a dedicated User Memory environment: ```bash conda create -n omnimemeval python=3.12 -y conda activate omnimemeval -pip install -r requirements_user_memory.txt +python -m pip install -r requirements_user_memory.txt ``` -AgentBench dependencies live in `requirements_agentbench.txt` and should be installed in a separate Agent Memory environment. +Install Agent Memory dependencies in a separate environment, as described in the +[Agent Memory guide](../agent_memory/README.md). -### 2. Configure Credentials + +### Configure credentials -Start from a product-specific template: +Copy the template for the backend you want to evaluate, then fill in its values. +For example: ```bash cp env_examples/.env.memos .env.memos ``` -Fill in the required memory product credentials and the OpenAI-compatible -ANSWER/EVAL LLM settings: +Configure the answer model, judge model where applicable, and backend credentials: - `ANSWER_MODEL`, `ANSWER_API_KEY`, `ANSWER_BASE_URL` -- `EVAL_MODEL`, `EVAL_API_KEY`, `EVAL_BASE_URL` -- Product-specific memory credentials such as `MEMOS_API_KEY` or `MEM0_API_KEY` +- `EVAL_MODEL`, `EVAL_API_KEY`, `EVAL_BASE_URL` for LLM judging +- Backend credentials such as `MEMOS_API_KEY` or `MEM0_API_KEY` -See [env_examples/README.md](../../env_examples/README.md) and -[env_examples/PARAMETERS.md](../../env_examples/PARAMETERS.md). +See the [template index](../../env_examples/README.md) and +[parameter reference](../../env_examples/PARAMETERS.md) for backend-specific +settings, defaults, and benchmark applicability. PersonaMem v2 uses option +matching, not an LLM judge. Do not commit filled credential files. -### 3. Prepare Data + +## Data preparation + +Download only the benchmarks you plan to run: ```bash # LoCoMo python data/locomo/prepare_locomo.py -# LongMemEval S +# LongMemEval-S python data/longmemeval/prepare_longmemeval.py # BEAM 100K @@ -124,274 +72,229 @@ python data/beam/prepare_beam.py # PersonaMem v2 python data/personamem_v2/prepare_personamem.py -# HaluMem Medium +# HaluMem-Medium python data/halumem/prepare_halumem.py ``` -Benchmark data is downloaded on demand and is not committed to this repository. -See [THIRD_PARTY_NOTICES.md](../../THIRD_PARTY_NOTICES.md) and the dataset README -files for upstream dataset licenses and redistribution notes. - -### 4. Run Evaluations - -```bash -./scripts/run_locomo_eval.sh --lib memos --env .env.memos -./scripts/run_lme_eval.sh --lib memos --env .env.memos -./scripts/run_beam_eval.sh --lib memos --env .env.memos -./scripts/run_pmv2_eval.sh --lib memos --env .env.memos -./scripts/run_halumem_eval.sh --lib memos --env .env.memos -``` - -Use `--help` on a runner for its accepted options. Shared defaults and -benchmark-specific applicability are maintained in the -[parameter reference](../../env_examples/PARAMETERS.md). +Data is downloaded on demand and is not committed to this repository. See the +[benchmark descriptions](#benchmarks), dataset READMEs, and +[third-party notices](../../THIRD_PARTY_NOTICES.md) for data preparation and licenses. -Streaming mode is available for LongMemEval, BEAM, PersonaMem v2, and -HaluMem. In streaming mode, OmniMemEval runs add, search, save, and delete for -each benchmark unit before moving to the next unit. Use `--streaming 1` on the -corresponding runner: +## Quick start -```bash -./scripts/run_lme_eval.sh --lib memos --env .env.memos --streaming 1 -./scripts/run_beam_eval.sh --lib memos --env .env.memos --streaming 1 -./scripts/run_pmv2_eval.sh --lib memos --env .env.memos --streaming 1 -./scripts/run_halumem_eval.sh --lib memos --env .env.memos --streaming 1 -``` - -Streaming runs support `--start-idx`, `--end-idx`, `--restart-unit`, -`--no-resume`, and `--skip-failed-streaming`. - -Ingestion/search checks (these call the backend; only the LongMemEval example -below limits the dataset to one unit): +After configuring the backend and preparing LongMemEval data, check ingestion +and search for one conversation: ```bash -# LoCoMo: run ingestion and search only -./scripts/run_locomo_eval.sh --lib memos --env .env.memos --version smoke_locomo --to-step 2 - -# LongMemEval: run one streaming conversation through search only ./scripts/run_lme_eval.sh --lib memos --env .env.memos --version smoke_lme \ --streaming 1 --start-idx 0 --end-idx 0 --to-step 2 - -# BEAM: run ingestion and search only on the default 100K scale -./scripts/run_beam_eval.sh --lib memos --env .env.memos --version smoke_beam --to-step 2 - -# PersonaMem v2: run ingestion and search only -./scripts/run_pmv2_eval.sh --lib memos --env .env.memos --version smoke_pmv2 --to-step 2 - -# HaluMem: run ingestion and search only -./scripts/run_halumem_eval.sh --lib memos --env .env.memos --version smoke_hm --to-step 2 ``` -Replay later stages from an existing result directory: +This calls a real backend and may incur charges. The index range selects the +first evaluation conversation. Streaming mode adds, searches, saves results, +and deletes that unit's backend memory. `--to-step 2` stops before answer +generation and judging; this check does not produce a full benchmark score. +Use a dedicated evaluation account. -```bash -./scripts/run_locomo_eval.sh --lib memos --env .env.memos --replay results/locomo/{LIB}-{VERSION}/ -./scripts/run_lme_eval.sh --lib memos --env .env.memos --replay results/lme/{LIB}-{VERSION}/ -./scripts/run_beam_eval.sh --lib memos --env .env.memos --replay results/beam/{LIB}-{VERSION}/ -./scripts/run_pmv2_eval.sh --lib memos --env .env.memos --replay results/pmv2/{LIB}-{VERSION}/ -./scripts/run_halumem_eval.sh --lib memos --env .env.memos --replay results/halumem/{LIB}-{VERSION}/ -``` - -## Benchmark Results - -See [benchmark results](./results.md) for the public -result snapshot reproduced under OmniMemEval's shared evaluation setup. The -document includes reproduced scores, context-token metrics, deployment notes, -published reference scores, and reproduction commands for the currently public -benchmark pipelines. +For complete scored runs, see [advanced runs](#advanced-runs). CLI options are +also available through each runner's `--help`. -## Supported Memory Backends +## Benchmarks -The public adapter layer exposes a common `add()` / `search()` / `delete()` -interface for mainstream memory products and self-hosted memory frameworks: +### LoCoMo -Use `--lib` to run the same benchmark against different memory solutions -without changing the benchmark stages, prompt flow, or metric calculation. +Long-conversation QA covering multi-hop recall, temporal reasoning, and +open-domain memory use. The reproduced results exclude adversarial questions. +See [data and license notes](../../data/locomo/README.md). -| `--lib` | Adapter | -|---------|---------| -| `memos` | MemOS | -| `mem0` | Mem0 | -| `zep` | Zep | -| `supermemory` | Supermemory | -| `everos` | EverOS | -| `letta` | Letta | -| `hindsight` | Hindsight | -| `graphiti` | Zep Graphiti local/self-hosted | -| `cognee` | Cognee | -| `viking` | Viking Memory | -| `memori` | Memori | -| `memmachine` | MemMachine | -| `memorylake` | MemoryLake | -| `backboard` | Backboard.io | -| `mem9` | mem9 | +### LongMemEval -To contribute a backend, follow the [adapter checklist](../../CONTRIBUTING.md#adding-a-user-memory-adapter). +Cross-session long-term memory covering knowledge updates, temporal reasoning, +and preferences. The shared loader uses `longmemeval_s_cleaned.json`, removing +known bad special tokens consistently for ingestion and search. +See [data and license notes](../../data/longmemeval/README.md). -## Benchmarks +### BEAM - -### LoCoMo +Large-scale long-memory QA. The runner supports 100K (approximately 128K +tokens), 500K, 1M, and 10M scales, with per-nugget LLM judging. +See [data and license notes](../../data/beam/README.md). -LoCoMo evaluates long-conversation memory with multi-hop, temporal, and -open-domain QA. Data and license notes live in -[data/locomo/README.md](../../data/locomo/README.md). + +### PersonaMem v2 -```bash -./scripts/run_locomo_eval.sh --lib memos --env .env.memos -``` +Personalized memory and preference-aware multiple-choice QA. Scoring matches the +selected option against the gold answer without an LLM judge. +See [data and license notes](../../data/personamem_v2/README.md). -Results: `results/locomo/{LIB}-{VERSION}/` +### HaluMem -Replay later stages: +Memory robustness covering hallucinations, memory boundaries, conflicts, +multi-hop inference, and dynamic updates. +See [data and license notes](../../data/halumem/README.md). -```bash -./scripts/run_locomo_eval.sh --lib memos --env .env.memos --replay results/locomo/{LIB}-{VERSION}/ -``` +## Advanced runs - -### LongMemEval + +### Full benchmark runs -LongMemEval evaluates long-term memory across sessions. OmniMemEval loads -`longmemeval_s_cleaned.json` through a shared loader that removes known bad -special tokens and applies the same cleaned data to ingestion and search. Data -and license notes live in -[data/longmemeval/README.md](../../data/longmemeval/README.md). +Each command below evaluates its benchmark's configured dataset, not a small +smoke subset. Prepare the corresponding data and credentials first. ```bash +./scripts/run_locomo_eval.sh --lib memos --env .env.memos ./scripts/run_lme_eval.sh --lib memos --env .env.memos +./scripts/run_beam_eval.sh --lib memos --env .env.memos +./scripts/run_pmv2_eval.sh --lib memos --env .env.memos +./scripts/run_halumem_eval.sh --lib memos --env .env.memos ``` -Results: `results/lme/{LIB}-{VERSION}/` +Replace `memos` and the env file with the desired backend. Set `--version ` +to isolate run artifacts. Shared defaults and benchmark-specific options are +maintained in the [parameter reference](../../env_examples/PARAMETERS.md). -Replay later stages: + +### Pipeline and scoring -```bash -./scripts/run_lme_eval.sh --lib memos --env .env.memos --replay results/lme/{LIB}-{VERSION}/ +The standard retrieval-based flow is: + +```text +Dataset → Ingest (add) → Search → Answer → Evaluate → Metrics → Report + ↕ ↕ + Memory backend ``` - -### BEAM +- Ingestion and search call the selected adapter's `add()` and `search()`. +- Answer generation normally uses the configured OpenAI-compatible answer model. + Backend-specific direct-answer modes may replace or skip that stage; disclose + the selected mode when reporting results. +- LoCoMo, LongMemEval, and HaluMem use LLM judging, with additional metrics where + implemented. BEAM uses LLM-based nugget scoring. PersonaMem v2 uses option matching. +- Runners support stage checkpoints and replay. A partial run is not a complete + benchmark result. -BEAM evaluates long-term memory at 128K, 500K, 1M, and 10M token scales with -per-nugget LLM-as-Judge scoring. Data and license notes live in -[data/beam/README.md](../../data/beam/README.md). +### Streaming and partial runs -```bash -./scripts/run_beam_eval.sh --lib memos --env .env.memos -``` +LongMemEval, BEAM, PersonaMem v2, and HaluMem support `--streaming 1`. +For each unit, streaming runs add, search, save, and delete before proceeding. +It supports `--start-idx`, `--end-idx`, `--restart-unit`, `--no-resume`, +and `--skip-failed-streaming`; check the selected runner's `--help` for semantics. + +Use `--to-step 2` for ingestion/search checks. This limits stages, not dataset +size; without an explicit supported subset selector, it can still process the +whole dataset. The [quick start](#quick-start) combines both limits for LongMemEval. -Results: `results/beam/{LIB}-{VERSION}/` +### Replay -Replay later stages: +Resume later stages in an existing run by using the same backend/version and +keeping the prerequisite artifacts. For example, to regenerate answers onward: ```bash -./scripts/run_beam_eval.sh --lib memos --env .env.memos --replay results/beam/{LIB}-{VERSION}/ +./scripts/run_lme_eval.sh --lib memos --env .env.memos \ + --version existing_run --from-step 3 ``` - -### PersonaMem v2 +Replace `existing_run` with your saved run version. Explicit `--from-step` +forces that stage and subsequent stages to run again. -PersonaMem v2 evaluates personalized memory and preference-aware multiple-choice -QA. Data and license notes live in -[data/personamem_v2/README.md](../../data/personamem_v2/README.md). +By contrast, `--replay` loads a saved experiment configuration and creates a new +timestamped run version. It does not copy earlier-stage artifacts into the new +directory. To repeat a complete run: ```bash -./scripts/run_pmv2_eval.sh --lib memos --env .env.memos +./scripts/run_lme_eval.sh --env .env.memos \ + --replay results/lme/{LIB}-{VERSION}/ ``` -Results: `results/pmv2/{LIB}-{VERSION}/` +Other runners accept `--replay` with their corresponding result directory. +Replace `{LIB}-{VERSION}` with the saved backend key and run version. Use the +`--env FILE --replay DIR` ordering shown above; the backend key and run settings +are restored from the saved configuration, so do not add `--lib`. +Only use trusted configurations: replay sources `experiment_config.sh` as shell code. +Record any configuration changes and disclose checkpoint reuse or re-evaluation +when submitting scores. -Replay later stages: + +### Cleanup + +Preview the evaluation user IDs before deleting backend memory from a run: ```bash -./scripts/run_pmv2_eval.sh --lib memos --env .env.memos --replay results/pmv2/{LIB}-{VERSION}/ +./scripts/run_memory_clear.sh --lib memos --env .env.memos --version '' --datasets locomo,lme,beam,pmv2,hm --dry-run +./scripts/run_memory_clear.sh --lib memos --env .env.memos --version '' --datasets locomo,lme,beam,pmv2,hm --yes ``` - -### HaluMem +Replace `` with the intended run version. `--dry-run` does not +delete data; actual deletion requires `--yes`. +For non-default BEAM or HaluMem datasets, use repeatable `--beam-scale` +or `--halumem-variant` options as appropriate. Verify the account and run version. -HaluMem evaluates memory hallucination, conflict handling, dynamic updates, and -memory-boundary robustness. Data and license notes live in -[data/halumem/README.md](../../data/halumem/README.md). + +## Outputs and results -```bash -./scripts/run_halumem_eval.sh --lib memos --env .env.memos -``` +Artifacts are written under these directories, using the backend key and run version: -Results: `results/halumem/{LIB}-{VERSION}/` +| Benchmark | Result directory | +|---|---| +| LoCoMo | `results/locomo/{LIB}-{VERSION}/` | +| LongMemEval | `results/lme/{LIB}-{VERSION}/` | +| BEAM | `results/beam/{LIB}-{VERSION}/` | +| PersonaMem v2 | `results/pmv2/{LIB}-{VERSION}/` | +| HaluMem | `results/halumem/{LIB}-{VERSION}/` | -Replay later stages: +See [benchmark results](./results.md) for the public snapshot, evaluation +configuration, aggregation rules, and context-token definitions. +Reproduced Results and Published Reference Results have different evidence +and comparability requirements; do not mix them. -```bash -./scripts/run_halumem_eval.sh --lib memos --env .env.memos --replay results/halumem/{LIB}-{VERSION}/ -``` + +## Extensions and contribution -## Cleanup +The adapter layer provides common add/search operations and user-scoped cleanup. +Supported backend keys are: -To delete backend memory created by a run: - -```bash -./scripts/run_memory_clear.sh --lib memos --env .env.memos --version --datasets locomo,lme,beam,pmv2,hm --dry-run -./scripts/run_memory_clear.sh --lib memos --env .env.memos --version --datasets locomo,lme,beam,pmv2,hm --yes -``` +| `--lib` | Backend | +|---|---| +| `memos` | MemOS | +| `mem0` | Mem0 | +| `zep` | Zep | +| `supermemory` | Supermemory | +| `everos` | EverOS | +| `letta` | Letta | +| `hindsight` | Hindsight | +| `graphiti` | Zep Graphiti local/self-hosted | +| `cognee` | Cognee | +| `viking` | Viking Memory | +| `memori` | Memori | +| `memmachine` | MemMachine | +| `memorylake` | MemoryLake | +| `backboard` | Backboard.io | +| `mem9` | mem9 | -`--dry-run` prints target ids without deleting data. Destructive deletion -requires `--yes`. Use repeatable `--beam-scale` and `--halumem-variant` when -clearing non-default BEAM or HaluMem datasets. +Follow the [adapter checklist](../../CONTRIBUTING.md#adding-a-user-memory-adapter) +for integration, configuration, isolation, and testing, and the +[result submission requirements](../../CONTRIBUTING.md#submitting-benchmark-results) +for published or reproduced scores. -## Project Layout + +The root [repository layout](../../README.md#repository-layout) describes where +code, configuration, and data preparation live. -```text -OmniMemEval/ -├── data/ -│ ├── beam/ -│ ├── halumem/ -│ ├── locomo/ -│ ├── longmemeval/ -│ └── personamem_v2/ -├── docs/ -│ └── user_memory/ -│ ├── README.md -│ ├── README_zh.md -│ └── results.md -├── env_examples/ -├── scripts/ -│ ├── client_factory/ -│ ├── beam/ -│ ├── halumem/ -│ ├── locomo/ -│ ├── longmemeval/ -│ ├── personamem_v2/ -│ ├── tests/ -│ ├── utils/ -│ ├── run_beam_eval.sh -│ ├── run_halumem_eval.sh -│ ├── run_locomo_eval.sh -│ ├── run_lme_eval.sh -│ ├── run_pmv2_eval.sh -│ └── run_memory_clear.sh -├── README.md -├── README_zh.md -├── THIRD_PARTY_NOTICES.md -├── requirements_user_memory.txt # User Memory dependencies -└── requirements_agentbench.txt # AgentBench dependencies -``` + +## Verification and licenses -## Verification +Use a separate development environment with Python 3.11 or 3.12 for offline checks: ```bash -git ls-files -z '*.sh' | xargs -0 -n 1 bash -n -conda run -n omnimemeval python -m compileall -q scripts data -conda run -n omnimemeval python -m pip install -r requirements-dev.txt -conda run -n omnimemeval python -m pytest -q +python -m pip install -r requirements-dev.txt +python -m pytest -q +python scripts/check_doc_links.py ``` -See the [test guide](../../scripts/tests/README.md) for offline suites, CI checks, -and backend integration smoke tests. - -## License +The [test guide](../../scripts/tests/README.md) distinguishes offline tests, +repository checks, and real-backend integration tests. Offline CI does not +validate benchmark scores or call live services. -See [LICENSE](../../LICENSE). Third-party benchmark data keeps its upstream license; -the OmniMemEval code license does not relicense external datasets. See -[THIRD_PARTY_NOTICES.md](../../THIRD_PARTY_NOTICES.md). + +See [LICENSE](../../LICENSE) for the code license. External datasets keep their +upstream licenses; see [THIRD_PARTY_NOTICES.md](../../THIRD_PARTY_NOTICES.md). diff --git a/docs/user_memory/README_zh.md b/docs/user_memory/README_zh.md index c6042c5..1c564a0 100644 --- a/docs/user_memory/README_zh.md +++ b/docs/user_memory/README_zh.md @@ -1,113 +1,67 @@ -# User Memory Evaluation + +# User Memory Evaluation（用户记忆评测） -[English](./README.md) +[English](./README.md) · [文档索引](../README_zh.md) · [评测结果](./results_zh.md) -[文档索引](../README_zh.md) +## 概览 -本文档说明 OmniMemEval 的 User Memory Evaluation 评测线。该评测线通过统一 benchmark pipeline 和 adapter 层评估 memory backend API。用户可以通过 `--lib` 切换不同 memory backend，在同一套 benchmark 流程下对比主流 memory 产品、自托管 memory framework 和自定义 adapter。 +本评测线通过共享评测流程和适配器（adapter）层评估记忆后端（memory backend） +API。通过 `--lib` 选择后端，可以在同一套 benchmark 流程下评估记忆产品、 +自托管框架和自定义适配器。 -adapter 层通过 15 个入口覆盖 14 种主流 memory 方案，包括 MemOS、Mem0、Zep/Graphiti、Supermemory、EverOS、Letta、Hindsight、Cognee、Viking Memory、Memori、MemMachine、MemoryLake、Backboard.io 和 mem9。该评测线支持 LoCoMo、LongMemEval、BEAM、PersonaMem v2 和 HaluMem 5 个评测任务，覆盖长期记忆系统中几类关键能力：对话记忆、跨 session 更新、超长上下文检索、个性化偏好，以及在 -幻觉、冲突和动态更新场景下的鲁棒性。 +目前有 15 个适配器入口，覆盖 14 种记忆方案（Zep 和 Graphiti 使用不同入口）， +支持 LoCoMo、LongMemEval、BEAM、PersonaMem v2 和 HaluMem 五个 benchmark。 -Benchmark 覆盖： +首次运行建议按本文顺序阅读。已有环境可以直接查看 +[进阶运行](#进阶运行)、[输出与结果](#输出与结果)或[扩展与贡献](#扩展与贡献)。 +所有命令均在仓库根目录执行。 -- [LoCoMo](#locomo)：长对话 QA，覆盖多跳召回、时序推理和开放域记忆使用。 -- [LongMemEval](#longmemeval)：跨 session 长期记忆，覆盖知识更新、时序推理和偏好问题。 -- [BEAM](#beam)：128K 到 10M token 上下文规模下的大规模记忆检索评测。 -- [PersonaMem v2](#personamem-v2)：面向用户偏好、敏感信息和个性化行为的记忆评测。 -- [HaluMem](#halumem)：面向记忆幻觉、边界识别、冲突处理、多跳推理和动态更新的鲁棒性评测。 +## 环境与配置 -## 评测流程 + +### 安装依赖 -OmniMemEval 的 benchmark 链路共享同一套阶段： - -```text -┌──────────────────┐ -│ Benchmark 数据 │ -│ 数据集特定格式 │ -└────────┬─────────┘ - │ - ▼ -┌──────────────────┐ add() ┌──────────────────┐ -│ 1. 写入 ├─────────────────▶│ Memory Backend │ -│ 对话内容 │ │ 由 --lib 选择 │ -└────────┬─────────┘ └────────┬─────────┘ - │ │ - ▼ │ search() -┌──────────────────┐ │ -│ 2. 检索 │◀──────────────────────────┘ -│ 召回上下文 │ -└────────┬─────────┘ - │ - ▼ -┌──────────────────┐ ANSWER LLM -│ 3. 生成答案 ├─────────────────▶ 生成答案 -│ ANSWER 模型 │ -└────────┬─────────┘ - │ - ▼ -┌──────────────────┐ EVAL LLM / NLP -│ 4. 评估 ├─────────────────▶ 评估记录 -│ LLM-as-Judge │ -└────────┬─────────┘ - │ - ▼ -┌──────────────────┐ -│ 5. 指标计算 │ -│ 准确率/延迟 │ -└────────┬─────────┘ - │ - ▼ -┌──────────────────┐ -│ 6. 报告 │ -│ Markdown/结果 │ -└──────────────────┘ -``` - -- 写入阶段调用 memory client 的 `add()`。 -- 检索阶段调用 memory client 的 `search()`。 -- 答案生成阶段使用 OpenAI-compatible ANSWER 模型生成答案。 -- 评估阶段使用 OpenAI-compatible EVAL 模型做 LLM-as-Judge，并计算 NLP 指标。 -- 指标和报告写入 `results//-/`。 - -Shell runner 和 Python 阶段都支持 checkpoint/resume。 - -## 快速开始 - -### 1. 创建环境 +为 User Memory 创建独立环境： ```bash conda create -n omnimemeval python=3.12 -y conda activate omnimemeval -pip install -r requirements_user_memory.txt +python -m pip install -r requirements_user_memory.txt ``` -AgentBench 依赖放在 `requirements_agentbench.txt`，应安装在独立的 Agent Memory 环境中。 +Agent Memory 的依赖应安装在另一个环境中，详见 +[Agent Memory 指南](../agent_memory/README_zh.md)。 -### 2. 配置凭证 + +### 配置凭证 -从产品模板开始： +复制待评测后端的模板，再填写配置。例如： ```bash cp env_examples/.env.memos .env.memos ``` -填写 memory 产品凭证和 ANSWER/EVAL LLM 配置： +配置答案模型、适用时的评分模型，以及后端凭证： + +- `ANSWER_MODEL`、`ANSWER_API_KEY`、`ANSWER_BASE_URL` +- 使用 LLM 评分时的 `EVAL_MODEL`、`EVAL_API_KEY`、`EVAL_BASE_URL` +- 后端凭证，例如 `MEMOS_API_KEY` 或 `MEM0_API_KEY` -- `ANSWER_MODEL`, `ANSWER_API_KEY`, `ANSWER_BASE_URL` -- `EVAL_MODEL`, `EVAL_API_KEY`, `EVAL_BASE_URL` -- 产品侧凭证，例如 `MEMOS_API_KEY` 或 `MEM0_API_KEY` +后端特有配置、默认值和各 benchmark 的适用范围见 +[模板索引（英文）](../../env_examples/README.md)和 +[参数说明（英文）](../../env_examples/PARAMETERS.md)。 +PersonaMem v2 使用选项匹配，不使用 LLM 评分。不要提交填写了凭证的文件。 -完整参数见 [env_examples/README.md](../../env_examples/README.md) 和 -[env_examples/PARAMETERS.md](../../env_examples/PARAMETERS.md)。 + +## 数据准备 -### 3. 准备数据 +只下载计划运行的 benchmark： ```bash # LoCoMo python data/locomo/prepare_locomo.py -# LongMemEval S +# LongMemEval-S python data/longmemeval/prepare_longmemeval.py # BEAM 100K @@ -116,231 +70,212 @@ python data/beam/prepare_beam.py # PersonaMem v2 python data/personamem_v2/prepare_personamem.py -# HaluMem Medium +# HaluMem-Medium python data/halumem/prepare_halumem.py ``` -Benchmark 数据按需下载，不提交到仓库。上游数据许可证和再分发说明见 -[THIRD_PARTY_NOTICES.md](../../THIRD_PARTY_NOTICES.md) 和各数据目录 README。 - -### 4. 运行评测 - -```bash -./scripts/run_locomo_eval.sh --lib memos --env .env.memos -./scripts/run_lme_eval.sh --lib memos --env .env.memos -./scripts/run_beam_eval.sh --lib memos --env .env.memos -./scripts/run_pmv2_eval.sh --lib memos --env .env.memos -./scripts/run_halumem_eval.sh --lib memos --env .env.memos -``` - -使用 runner 的 `--help` 查看接受的选项。共享参数默认值和各 benchmark 的适用范围 -统一维护在 [参数说明](../../env_examples/PARAMETERS.md) 中。 - -LongMemEval、BEAM、PersonaMem v2 和 HaluMem 支持 streaming 模式。在 -streaming 模式下，OmniMemEval 会对每个 benchmark unit 依次执行 add、search、 -保存结果和 delete，再进入下一个 unit。对应 runner 均通过 `--streaming 1` 启用： - -```bash -./scripts/run_lme_eval.sh --lib memos --env .env.memos --streaming 1 -./scripts/run_beam_eval.sh --lib memos --env .env.memos --streaming 1 -./scripts/run_pmv2_eval.sh --lib memos --env .env.memos --streaming 1 -./scripts/run_halumem_eval.sh --lib memos --env .env.memos --streaming 1 -``` +数据按需下载，不提交到仓库。数据准备与许可证信息见 +[benchmark 说明](#benchmarks)、各数据目录 README 和 +[第三方声明（英文）](../../THIRD_PARTY_NOTICES.md)。 -Streaming 模式可配合 `--start-idx`、`--end-idx`、`--restart-unit`、 -`--no-resume` 和 `--skip-failed-streaming` 使用。 +## 快速开始 -Ingestion/search 检查（会调用后端；以下只有 LongMemEval 示例限制为一个数据单元）： +配置后端并准备 LongMemEval 数据后，先检查一个对话单元的写入与检索： ```bash -# LoCoMo：只跑 ingestion 和 search -./scripts/run_locomo_eval.sh --lib memos --env .env.memos --version smoke_locomo --to-step 2 - -# LongMemEval：只跑一个 streaming conversation 到 search ./scripts/run_lme_eval.sh --lib memos --env .env.memos --version smoke_lme \ --streaming 1 --start-idx 0 --end-idx 0 --to-step 2 +``` -# BEAM：默认 100K scale，只跑 ingestion 和 search -./scripts/run_beam_eval.sh --lib memos --env .env.memos --version smoke_beam --to-step 2 +该命令调用真实后端，可能产生费用。索引范围按对话单元选择，此处只处理第一个单元。 +流式模式（streaming）会写入、检索、保存结果，再删除该单元的后端记忆。 +`--to-step 2` 在答案生成和评分前停止，因此这不是完整 benchmark 分数。 +请使用独立的评测账号。 -# PersonaMem v2：只跑 ingestion 和 search -./scripts/run_pmv2_eval.sh --lib memos --env .env.memos --version smoke_pmv2 --to-step 2 +完整评分运行见[进阶运行](#进阶运行)。各 runner 的 `--help` 也提供选项说明。 -# HaluMem：只跑 ingestion 和 search -./scripts/run_halumem_eval.sh --lib memos --env .env.memos --version smoke_hm --to-step 2 -``` +## Benchmarks -从已有结果目录 replay 后续阶段： +### LoCoMo -```bash -./scripts/run_locomo_eval.sh --lib memos --env .env.memos --replay results/locomo/{LIB}-{VERSION}/ -./scripts/run_lme_eval.sh --lib memos --env .env.memos --replay results/lme/{LIB}-{VERSION}/ -./scripts/run_beam_eval.sh --lib memos --env .env.memos --replay results/beam/{LIB}-{VERSION}/ -./scripts/run_pmv2_eval.sh --lib memos --env .env.memos --replay results/pmv2/{LIB}-{VERSION}/ -./scripts/run_halumem_eval.sh --lib memos --env .env.memos --replay results/halumem/{LIB}-{VERSION}/ -``` +长对话问答，覆盖多跳召回、时序推理和开放域记忆使用。 +复现结果排除对抗性问题。 +详见[数据与许可证说明（英文）](../../data/locomo/README.md)。 -## Benchmark 结果 +### LongMemEval -公开评测结果见 [benchmark results（英文）](./results.md)。该文档整理了 -当前公开 benchmark 链路在 OmniMemEval 统一评测配置下复现的分数、context token -指标、部署说明、公开参考分数和复现命令。 +跨会话长期记忆，覆盖知识更新、时序推理和偏好。 +共享加载器使用 `longmemeval_s_cleaned.json`，对写入和检索一致地清除已知异常特殊字符。 +详见[数据与许可证说明（英文）](../../data/longmemeval/README.md)。 -## 支持的 Memory Backend +### BEAM -公开 adapter 层统一暴露 `add()` / `search()` / `delete()` 接口，覆盖主流 -memory 产品和自托管 memory framework： +大规模长期记忆问答。Runner 支持 100K（约 128K token）、500K、1M 和 10M +规模，使用 LLM 对原子事实点（nugget）评分。 +详见[数据与许可证说明（英文）](../../data/beam/README.md)。 -通过 `--lib` 可以在不改 benchmark 阶段、prompt 流程和指标计算的前提下，对不同 -memory 方案运行同一套评测。 + +### PersonaMem v2 -| `--lib` | Adapter | -|---------|---------| -| `memos` | MemOS | -| `mem0` | Mem0 | -| `zep` | Zep | -| `supermemory` | Supermemory | -| `everos` | EverOS | -| `letta` | Letta | -| `hindsight` | Hindsight | -| `graphiti` | Zep Graphiti 本地/自托管 | -| `cognee` | Cognee | -| `viking` | Viking Memory | -| `memori` | Memori | -| `memmachine` | MemMachine | -| `memorylake` | MemoryLake | -| `backboard` | Backboard.io | -| `mem9` | mem9 | +个性化记忆与偏好感知多选问答。将所选选项与标准答案匹配，不使用 LLM 评分。 +详见[数据与许可证说明（英文）](../../data/personamem_v2/README.md)。 + +### HaluMem -新增后端时，请遵循 [adapter 接入清单（英文）](../../CONTRIBUTING.md#adding-a-user-memory-adapter)。 +记忆鲁棒性评测，覆盖幻觉、记忆边界、冲突、多跳推理和动态更新。 +详见[数据与许可证说明（英文）](../../data/halumem/README.md)。 -## Benchmarks +## 进阶运行 - -### LoCoMo + +### 完整 benchmark 运行 -LoCoMo 评估长对话记忆、多跳推理和时序记忆。数据说明见 -[data/locomo/README.md](../../data/locomo/README.md)。 +以下每条命令运行对应 benchmark 配置的数据集，不是小规模检查。 +请先准备相应数据和凭证。 ```bash ./scripts/run_locomo_eval.sh --lib memos --env .env.memos +./scripts/run_lme_eval.sh --lib memos --env .env.memos +./scripts/run_beam_eval.sh --lib memos --env .env.memos +./scripts/run_pmv2_eval.sh --lib memos --env .env.memos +./scripts/run_halumem_eval.sh --lib memos --env .env.memos ``` -结果目录：`results/locomo/{LIB}-{VERSION}/` +替换 `memos` 和环境文件即可切换后端。通过 `--version ` 隔离运行产物。 +共享默认值与 benchmark 特有选项统一维护在 +[参数说明（英文）](../../env_examples/PARAMETERS.md)。 - -### LongMemEval + +### 流程与评分 -LongMemEval 评估跨 session 长期记忆。OmniMemEval 通过共享 loader 读取 -`longmemeval_s_cleaned.json`，并对 ingestion 和 search 使用同一份清洗后的数据。 -数据说明见 [data/longmemeval/README.md](../../data/longmemeval/README.md)。 +标准检索模式的流程如下： -```bash -./scripts/run_lme_eval.sh --lib memos --env .env.memos +```text +数据集 → 写入（add）→ 检索 → 答案生成 → 评估 → 指标计算 → 报告 + ↕ ↕ + 记忆后端 ``` -结果目录：`results/lme/{LIB}-{VERSION}/` +- 写入和检索调用所选适配器的 `add()` 和 `search()`。 +- 答案生成通常使用配置的 OpenAI-compatible 答案模型。 + 后端特有的直接回答模式可能替代或跳过该阶段；提交结果时必须披露所选模式。 +- LoCoMo、LongMemEval 和 HaluMem 使用 LLM 评分，并计算已实现的辅助指标； + BEAM 使用基于 LLM 的 nugget 评分；PersonaMem v2 使用选项匹配。 +- Runner 支持阶段检查点和重放（replay）。部分运行不等于完整 benchmark 结果。 - -### BEAM +### 流式与部分运行 + +LongMemEval、BEAM、PersonaMem v2 和 HaluMem 支持 `--streaming 1`， +对每个数据单元依次写入、检索、保存、删除，再处理下一个单元。 +支持 `--start-idx`、`--end-idx`、`--restart-unit`、`--no-resume` 和 +`--skip-failed-streaming`；具体语义以所选 runner 的 `--help` 为准。 + +`--to-step 2` 可用于写入与检索检查，但它只限制阶段，不限制数据量。 +如果没有显式使用受支持的子集选择参数，仍可能处理整个数据集。 +[快速开始](#快速开始)中的 LongMemEval 示例同时限制了阶段和数据范围。 + +### 重放 -BEAM 评估 128K、500K、1M 和 10M token 规模下的长期记忆能力，采用 -per-nugget LLM-as-Judge 评分。数据说明见 -[data/beam/README.md](../../data/beam/README.md)。 +在已有运行中继续后续阶段时，使用相同后端和版本，并保留所需的前序产物。 +例如，从答案生成阶段开始重新执行： ```bash -./scripts/run_beam_eval.sh --lib memos --env .env.memos +./scripts/run_lme_eval.sh --lib memos --env .env.memos \ + --version existing_run --from-step 3 ``` -结果目录：`results/beam/{LIB}-{VERSION}/` - - -### PersonaMem v2 +将 `existing_run` 替换为已有运行版本。显式指定 `--from-step` +会强制重新执行该阶段及后续阶段。 -PersonaMem v2 评估个性化记忆和偏好感知多选问答。数据说明见 -[data/personamem_v2/README.md](../../data/personamem_v2/README.md)。 +相比之下，`--replay` 加载已有实验配置，并创建带时间戳的新运行版本， +不会把前序产物复制到新目录。完整重放示例： ```bash -./scripts/run_pmv2_eval.sh --lib memos --env .env.memos +./scripts/run_lme_eval.sh --env .env.memos \ + --replay results/lme/{LIB}-{VERSION}/ ``` -结果目录：`results/pmv2/{LIB}-{VERSION}/` +其他 runner 也接受 `--replay`，传入其对应结果目录即可。 +请将 `{LIB}-{VERSION}` 替换为已有后端标识和运行版本，使用上面的 +`--env FILE --replay DIR` 顺序；后端标识和运行设置会从已保存配置恢复， +无需额外添加 `--lib`。只使用可信配置：重放会将 `experiment_config.sh` 作为 Shell 代码执行。 +提交分数时应记录配置变化，并披露检查点复用或重新评估的情况。 - -### HaluMem + +### 清理 -HaluMem 评估记忆幻觉、冲突处理、动态更新和记忆边界鲁棒性。数据说明见 -[data/halumem/README.md](../../data/halumem/README.md)。 +先预览评测用户 ID，再删除某次运行写入的后端记忆： ```bash -./scripts/run_halumem_eval.sh --lib memos --env .env.memos +./scripts/run_memory_clear.sh --lib memos --env .env.memos --version '' --datasets locomo,lme,beam,pmv2,hm --dry-run +./scripts/run_memory_clear.sh --lib memos --env .env.memos --version '' --datasets locomo,lme,beam,pmv2,hm --yes ``` -结果目录：`results/halumem/{LIB}-{VERSION}/` +请将 `` 替换为目标运行版本。`--dry-run` 不删除数据；实际删除必须传入 `--yes`。 +清理非默认 BEAM 或 HaluMem 数据时，根据需要使用可重复的 `--beam-scale` +或 `--halumem-variant`。请核对账号和运行版本。 -## 清理 + +## 输出与结果 -删除某次评测写入的后端 memory： +产物按后端标识和运行版本写入以下目录： -```bash -./scripts/run_memory_clear.sh --lib memos --env .env.memos --version --datasets locomo,lme,beam,pmv2,hm --dry-run -./scripts/run_memory_clear.sh --lib memos --env .env.memos --version --datasets locomo,lme,beam,pmv2,hm --yes -``` +| Benchmark | 结果目录 | +|---|---| +| LoCoMo | `results/locomo/{LIB}-{VERSION}/` | +| LongMemEval | `results/lme/{LIB}-{VERSION}/` | +| BEAM | `results/beam/{LIB}-{VERSION}/` | +| PersonaMem v2 | `results/pmv2/{LIB}-{VERSION}/` | +| HaluMem | `results/halumem/{LIB}-{VERSION}/` | -`--dry-run` 只打印目标 id；实际删除必须显式传入 `--yes`。清理非默认 BEAM 或 HaluMem 数据时，可配合可重复的 `--beam-scale` 和 `--halumem-variant` 使用。 +[评测结果](./results_zh.md)提供公开快照、评测配置、汇总规则和上下文 token 定义的入口。 +统一流程复现结果（Reproduced Results）与公开参考结果（Published Reference Results） +的证据要求和可比性不同，不应混用。 -## 项目结构 + +## 扩展与贡献 -```text -OmniMemEval/ -├── data/ -│ ├── beam/ -│ ├── halumem/ -│ ├── locomo/ -│ ├── longmemeval/ -│ └── personamem_v2/ -├── docs/ -│ └── user_memory/ -│ ├── README.md -│ ├── README_zh.md -│ └── results.md -├── env_examples/ -├── scripts/ -│ ├── client_factory/ -│ ├── beam/ -│ ├── halumem/ -│ ├── locomo/ -│ ├── longmemeval/ -│ ├── personamem_v2/ -│ ├── tests/ -│ ├── utils/ -│ ├── run_beam_eval.sh -│ ├── run_halumem_eval.sh -│ ├── run_locomo_eval.sh -│ ├── run_lme_eval.sh -│ ├── run_pmv2_eval.sh -│ └── run_memory_clear.sh -├── README.md -├── README_zh.md -├── THIRD_PARTY_NOTICES.md -├── requirements_user_memory.txt # User Memory 依赖 -└── requirements_agentbench.txt # AgentBench 依赖 -``` +适配器层提供通用写入、检索操作和用户级清理。支持以下后端标识： + +| `--lib` | 后端 | +|---|---| +| `memos` | MemOS | +| `mem0` | Mem0 | +| `zep` | Zep | +| `supermemory` | Supermemory | +| `everos` | EverOS | +| `letta` | Letta | +| `hindsight` | Hindsight | +| `graphiti` | Zep Graphiti 本地/自托管 | +| `cognee` | Cognee | +| `viking` | Viking Memory | +| `memori` | Memori | +| `memmachine` | MemMachine | +| `memorylake` | MemoryLake | +| `backboard` | Backboard.io | +| `mem9` | mem9 | -## 验证 +接入、配置、隔离与测试要求见 +[适配器清单（英文）](../../CONTRIBUTING.md#adding-a-user-memory-adapter)， +分数提交要求见 +[结果提交规范（英文）](../../CONTRIBUTING.md#submitting-benchmark-results)。 + +代码、配置与数据准备脚本的位置见根目录的[仓库结构](../../README_zh.md#目录结构)。 + + +## 验证与许可证 + +在独立的 Python 3.11 或 3.12 开发环境运行离线检查： ```bash -git ls-files -z '*.sh' | xargs -0 -n 1 bash -n -conda run -n omnimemeval python -m compileall -q scripts data -conda run -n omnimemeval python -m pip install -r requirements-dev.txt -conda run -n omnimemeval python -m pytest -q +python -m pip install -r requirements-dev.txt +python -m pytest -q +python scripts/check_doc_links.py ``` -离线测试、CI 检查和后端集成 smoke test 的区别见 -[测试指南](../../scripts/tests/README.md)。 - -## License +[测试指南（英文）](../../scripts/tests/README.md)区分离线测试、仓库检查和真实后端集成测试。 +离线 CI 不验证 benchmark 分数，也不调用真实服务。 -见 [LICENSE](../../LICENSE)。第三方 benchmark 数据保留其上游许可证；OmniMemEval -代码许可证不重新授权外部数据集。详见 -[THIRD_PARTY_NOTICES.md](../../THIRD_PARTY_NOTICES.md)。 + +代码许可证见 [LICENSE](../../LICENSE)。外部数据集保留上游许可证，详见 +[THIRD_PARTY_NOTICES.md（英文）](../../THIRD_PARTY_NOTICES.md)。 diff --git a/docs/user_memory/results.md b/docs/user_memory/results.md index 02870e0..7c03ea7 100644 --- a/docs/user_memory/results.md +++ b/docs/user_memory/results.md @@ -1,7 +1,11 @@ -# Benchmark Results + +# User Memory Evaluation Results -[User Memory Evaluation guide](./README.md) · -[Documentation index](../README.md) +[中文版](./results_zh.md) · [Documentation index](../README.md) · +[Guide](./README.md) + +This page is the canonical source for numeric result tables; the Chinese page +provides translated explanations and links to the same tables. This document provides a public result snapshot for OmniMemEval's current benchmark pipelines. The reproduced scores were generated under one evaluation diff --git a/docs/user_memory/results_zh.md b/docs/user_memory/results_zh.md new file mode 100644 index 0000000..e7cdb52 --- /dev/null +++ b/docs/user_memory/results_zh.md @@ -0,0 +1,46 @@ +# User Memory Evaluation（用户记忆评测）结果 + +[English / 完整结果表](./results.md) · [文档索引](../README_zh.md) · +[运行指南](./README_zh.md) + +本文提供中文阅读说明与结果表入口。数值统一维护在 +[英文结果页](./results.md)，避免中英文表格更新不同步。 + +## 评测设置与可比性 + +结果分为两类： + +- **统一流程复现结果（Reproduced Results）**：同一 benchmark 内使用共享数据、 + 提示词、答案/评分模型与指标逻辑。后端部署和配置差异仍需披露。 +- **公开参考结果（Published Reference Results）**：来自外部公开报告或贡献者自测， + 可能采用不同模型、数据、检索配置或评分方式，不能直接与统一流程复现结果比较。 + +模型标识、后端部署说明和指标定义见 +[评测设置](./results.md#evaluation-setup)，各后端的整体分数见 +[结果汇总](./results.md#result-summary)。横线 `-` 表示当前快照未收录可比的完整复现结果， +不表示零分。 + +## 指标与结果入口 + +| Benchmark | 主要指标 | 结果与范围说明 | +|---|---|---| +| LoCoMo | LLM 评分准确率 | [非对抗性问题子集与分类结果](./results.md#locomo) | +| LongMemEval | LLM 评分准确率 | [LongMemEval-S 与按题数加权的总体分数](./results.md#longmemeval) | +| BEAM | Nugget 分数 | [100K / 10M 的结果、范围与评分定义](./results.md#beam) | +| PersonaMem v2 | 选项匹配准确率 | [偏好类型与多次运行汇总](./results.md#personamem-v2) | +| HaluMem | LLM 评分准确率 | [问题类型与分类结果](./results.md#halumem) | + +`Context Tokens` 是每道问题发送给答案模型的平均输入 token 数， +包含答案提示词和检索上下文，不是单纯的后端返回文本长度。 +在准确率可比时，较低数值表示较高的 token 效率。 +具体定义和各 benchmark 的汇总方式以英文结果页为准。 + +## 复现与提交结果 + +运行准备和命令见[运行指南](./README_zh.md)， +快照复现说明见[Reproduction Notes](./results.md#reproduction-notes)。 + +提交结果时，请附上运行作者、版本、数据范围、配置、指标汇总方式、 +失败/跳过情况，以及可重算分数的公开逐题产物。 +适配器接入并不自动证明其分数可收录。 +完整要求见[结果提交规范（英文）](../../CONTRIBUTING.md#submitting-benchmark-results)。