Malena-yy · Oct 19, 2026 · Oct 19, 2026
diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml
@@ -0,0 +1,30 @@
+name: Bug report
+description: Report a reproducible code or documentation problem.
+body:
+  - type: textarea
+    id: problem
+    attributes:
+      label: Expected and actual behavior
+      description: Explain the difference. Include file/section links for documentation issues.
+    validations:
+      required: true
+  - type: textarea
+    id: reproduce
+    attributes:
+      label: Steps to reproduce
+      description: Include the command and a minimal input/configuration with credentials removed.
+    validations:
+      required: true
+  - type: textarea
+    id: environment
+    attributes:
+      label: Environment
+      description: Repository commit, OS, Python version, benchmark/backend, and relevant dependency versions.
+    validations:
+      required: true
+  - type: textarea
+    id: logs
+    attributes:
+      label: Relevant output
+      description: Remove API keys, private endpoints, and private conversation data before posting.
+      render: text
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
@@ -0,0 +1,21 @@
+## Problem and change
+
+<!-- Explain the problem and resulting behavior. Link related issues. -->
+
+## Validation
+
+<!-- List commands actually run and their results. Note untested integrations. -->
+
+- [ ] The change is focused, and relevant tests/documentation are updated.
+- [ ] Credentials, private endpoints, and generated benchmark data are excluded.
+
+## Adapter or benchmark results (if applicable)
+
+<!-- Delete this section for unrelated changes. See CONTRIBUTING.md. -->
+
+- Contribution type: adapter / Published Reference Results / Reproduced Results
+- Backend, harness, and dataset versions:
+- Run author, evaluated scope, and failure/skip accounting:
+- Commands, configuration, scoring, and aggregation:
+- Stable public result/artifact links:
+- Backend integration validation and any limitations:
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -0,0 +1,125 @@
+# Contributing to OmniMemEval
+
+Contributions to adapters, evaluation pipelines, documentation, and public
+benchmark results are welcome. Use an issue to report a reproducible problem
+or discuss a substantial change, and a pull request to propose a focused change.
+
+## Development workflow
+
+Create a branch from the latest `main` (use a fork if you do not have write
+access). Names such as `fix/search-status`, `chore/test-ci`, or
+`docs/quickstart` describe the purpose; no specific prefix is required.
+Keep one PR focused on one goal. All changes to `main` go through a PR.
+
+For offline development, use Python 3.11 or 3.12 in a virtual or conda environment:
+
+```bash
+python -m pip install -r requirements-dev.txt
+python -m pytest -q
+python scripts/check_doc_links.py
+git diff --check
+```
+
+See the [test guide](scripts/tests/README.md) for targeted suites and repository
+checks. Install [User Memory](docs/user_memory/README.md) or
+[Agent Memory](docs/agent_memory/README.md) runtime dependencies separately when
+running real evaluations. Record the dependency versions used for a run;
+the requirements files are dependency lists, not locked reproduction environments.
+
+Match the surrounding code style, avoid unrelated formatting changes, and add
+a regression test for a behavior fix. Documentation-only changes should pass
+the link checks; include a minimal before/after example when explaining a bug.
+
+## Adding a User Memory adapter
+
+The public entry points use `add(messages, user_id, ...)` and
+`search(query, user_id, top_k, ...)`, plus a user-scoped cleanup method.
+Inspect an existing adapter with similar behavior and its tests; product-specific
+results are normalized by the search dispatch helpers.
+
+1. Add the client under `scripts/client_factory/` and register its class in
+   [registry.py](scripts/client_factory/registry.py). Keep optional SDK imports
+   lazy so other backends remain usable without that SDK.
+2. Register its search behavior in
+   [search_helpers.py](scripts/utils/search_helpers.py) and the LoCoMo-specific
+   dispatch in [locomo_search.py](scripts/locomo/locomo_search.py).
+3. Check timestamp and session handling in
+   [ingest_helpers.py](scripts/utils/ingest_helpers.py), along with any
+   benchmark-specific ingestion behavior. Registering the client class alone
+   does not define these behaviors.
+4. Expose cleanup through a method supported by
+   [delete_user_data](scripts/utils/streaming.py), such as `delete(user_id)`.
+   Verify that deleting one evaluation user leaves all other users intact.
+5. Add a credential-free template under `env_examples/`, dependency/setup
+   instructions, and entries in the template index and both backend guide tables.
+   Document the backend version, deployment mode, required settings, and
+   non-default settings used by any submitted results.
+6. Add offline tests for configuration, registration, add/search/cleanup,
+   user isolation, and failures. Document a small real-backend smoke run and
+   its version/configuration when integration has been tested.
+
+For Agent Memory extensions, follow the separate
+[architecture and extension contracts](docs/agent_memory/architecture.md).
+
+## Submitting benchmark results
+
+The result tables distinguish **Published Reference Results** from
+**Reproduced Results**. Adding an adapter does not automatically add its scores.
+Only submit benchmarks whose supporting materials are ready; other rows can
+follow in another PR. Missing or unrun benchmarks use `-`.
+
+For every proposed result, provide a stable public report/artifact link and:
+
+- Run author and date; backend/deployment version; OmniMemEval or other harness
+  commit; dataset revision or checksum, split, answer-key version, and any corrections.
+- Evaluated question counts and coverage; failed, skipped, timed-out, and excluded
+  items; the denominator used for the reported metric.
+- Answer/judge model identifiers, prompts or prompt versions, retrieval settings,
+  exact commands, and credential-free configuration.
+- Scoring and aggregation rules, repetition counts, and how multiple runs are
+  combined. Explain checkpoint reuse and any re-evaluation; do not combine retained
+  high scores with score-selected reruns into an unexplained aggregate.
+- Per-question records sufficient to check coverage and recompute the aggregate,
+  along with summary files. Provide answer prompt-token totals and call counts
+  when submitting `Context Tokens`; otherwise use `-`.
+
+Link large artifacts in a public repository or release rather than adding them
+to this repository's ignored `results/` directory. Share the records needed for
+verification with credentials and private endpoints removed. Respect the
+upstream dataset licenses listed in [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).
+
+### Published Reference Results
+
+Add rows to the relevant sections of
+[docs/user_memory/results.md](docs/user_memory/results.md), with a direct source
+link. These are externally reported results and may use different data,
+models, prompts, or budgets. State such differences next to the result or in
+the linked methodology. For the LoCoMo four-category table, identify the
+non-adversarial subset; report any full-set score separately.
+
+An adapter contribution and a new shared-baseline run are not prerequisites
+for submitting published reference results.
+
+### Reproduced Results
+
+Use an available/reviewable OmniMemEval adapter and the shared pipeline,
+dataset scope, prompts, metrics, and baseline answer/judge settings documented
+in [Evaluation Setup](docs/user_memory/results.md#evaluation-setup).
+PersonaMem v2 uses option matching rather than an LLM judge. Backend-specific
+settings must be disclosed, including modes that generate answers directly.
+
+Contributor-run results are eligible for review when these conditions and the
+artifact requirements are met. Identify who ran them. The review should record
+whether verification covered artifact arithmetic, replay of answer/judge stages,
+or a complete rerun; these establish different levels of evidence.
+
+## Review and merge
+
+Describe the problem, change, and validation in the PR. If a command was not
+run, say so. Separate blocking issues from optional suggestions and reply to
+review comments when updating the PR. New commits may require another review.
+
+CI checks offline tests, Python/Shell syntax, and relative Markdown file links.
+It does not establish benchmark validity or run live integrations. Maintainers
+review evidence and test results before merging; squash merging is the normal
+choice for a focused PR. Follow-up fixes should use another branch and PR.
diff --git a/README.md b/README.md
@@ -9,24 +9,6 @@ OmniMemEval is an evaluation framework for memory systems and memory-augmented a
 | User Memory Evaluation | Memory backend APIs exposed through `add()` and `search()` | LoCoMo, LongMemEval, BEAM, PersonaMem v2, HaluMem | [docs/user_memory/README.md](./docs/user_memory/README.md) |
 | Agent Memory Evaluation | Agent runtimes equipped with memory plugins | AgentBench domains: reasoning, information retrieval, knowledge work, code implementation, software engineering | [docs/agent_memory/README.md](./docs/agent_memory/README.md) |
 
-## Evaluation Tracks
-
-### User Memory Evaluation
-
-User Memory Evaluation measures the capability of memory backend systems through a standardized API adapter layer. The same benchmark pipeline can be run against mainstream memory products, self-hosted memory frameworks, or custom adapters by selecting a backend with `--lib`.
-
-The pipeline covers ingestion, retrieval, answer generation, LLM-as-Judge evaluation, metric aggregation, and report generation. Results are written under `results/<benchmark>/<LIB>-<VERSION>/`.
-
-Start here: [docs/user_memory/README.md](./docs/user_memory/README.md)
-
-### Agent Memory Evaluation
-
-Agent Memory Evaluation measures the task performance of an agent runtime after a memory plugin is installed. The current implementation is based on AgentBench and evaluates OpenClaw across five task domains. The memory-plugin protocol includes memory cleanup, training, memory settling, backup, restore, and test execution.
-
-Results are written under `results/agentbench/`.
-
-Start here: [docs/agent_memory/README.md](./docs/agent_memory/README.md)
-
 ## Installation
 
 User Memory Evaluation and Agent Memory Evaluation use separate environments. For User Memory Evaluation, create the base Python environment from the repository root:
@@ -59,6 +41,13 @@ Each evaluation track may require additional dependencies:
 
 ```bash
 cp env_examples/.env.memos .env.memos
+```
+
+Fill in the memory backend credentials and `ANSWER_*` / `EVAL_*` model settings
+in `.env.memos` before continuing. Then prepare data and run the full LoCoMo
+evaluation (this calls the configured backend and model APIs):
+
+```bash
 python data/locomo/prepare_locomo.py
 ./scripts/run_locomo_eval.sh --lib memos --env .env.memos
 ```
@@ -67,14 +56,22 @@ python data/locomo/prepare_locomo.py
 
 ```bash
 cp env_examples/.env.agent .env.agent
+```
+
+Complete the OpenClaw setup and credentials in the
+[Agent Memory guide](./docs/agent_memory/README.md), then run one reasoning
+task as a smoke check:
+
+```bash
 mkdir -p data/agentbench
-huggingface-cli download EverMind-AI/EvoAgentBench \
+hf download EverMind-AI/EvoAgentBench \
   --repo-type dataset \
   --local-dir ./data/agentbench
 ./scripts/run_agent_eval.sh \
   --agent openclaw \
   --domain reasoning \
   --protocol test_only \
+  --test-split 1 \
   --version smoke_agentbench \
   --trials 1 \
   --parallel 1
@@ -109,8 +106,15 @@ scripts/
   utils/                      # Shared utilities
 requirements_user_memory.txt  # User Memory Evaluation dependencies
 requirements_agentbench.txt   # Agent Memory Evaluation dependencies
+requirements-dev.txt          # Offline tests and development checks
 ```
 
+## Contributing
+
+See [CONTRIBUTING.md](./CONTRIBUTING.md) for the branch/PR workflow, adapter
+integration, and benchmark-result submission requirements. The
+[test guide](./scripts/tests/README.md) documents offline checks and CI.
+
 ## License
 
 See [LICENSE](./LICENSE). Third-party benchmark data keeps its upstream license; the OmniMemEval code license does not relicense external datasets. See [THIRD_PARTY_NOTICES.md](./THIRD_PARTY_NOTICES.md).
diff --git a/README_zh.md b/README_zh.md
@@ -9,24 +9,6 @@ OmniMemEval 是面向记忆系统和记忆增强 Agent 的评测框架，当前
 | User Memory Evaluation | 通过 `add()` 和 `search()` 暴露能力的 memory backend API | LoCoMo、LongMemEval、BEAM、PersonaMem v2、HaluMem | [docs/user_memory/README_zh.md](./docs/user_memory/README_zh.md) |
 | Agent Memory Evaluation | 安装记忆插件后的 Agent Runtime | AgentBench domains：reasoning、information retrieval、knowledge work、code implementation、software engineering | [docs/agent_memory/README_zh.md](./docs/agent_memory/README_zh.md) |
 
-## 评测线
-
-### User Memory Evaluation
-
-User Memory Evaluation 通过统一 adapter 层评估 memory backend 系统能力。使用 `--lib` 可以在同一套 benchmark pipeline 下切换不同 memory 产品、自托管 memory framework 或自定义 adapter。
-
-该流程覆盖写入、检索、答案生成、LLM-as-Judge 评估、指标聚合和报告生成。结果写入 `results/<benchmark>/<LIB>-<VERSION>/`。
-
-入口文档：[docs/user_memory/README_zh.md](./docs/user_memory/README_zh.md)
-
-### Agent Memory Evaluation
-
-Agent Memory Evaluation 评估安装记忆插件后的 Agent Runtime 在任务域中的实际表现。当前实现基于 AgentBench，评估 OpenClaw 在五个任务域上的能力。记忆插件协议包含记忆清理、训练、沉淀、备份、恢复和测试执行。
-
-结果写入 `results/agentbench/`。
-
-入口文档：[docs/agent_memory/README_zh.md](./docs/agent_memory/README_zh.md)
-
 ## 安装
 
 User Memory Evaluation 和 Agent Memory Evaluation 使用两套独立环境。User Memory Evaluation 可在仓库根目录创建基础 Python 环境：
@@ -59,6 +41,12 @@ OpenClaw、系统包和各 domain 的额外安装步骤详见 [docs/agent_memory
 
 ```bash
 cp env_examples/.env.memos .env.memos
+```
+
+先在 `.env.memos` 中填写后端凭证和 `ANSWER_*` / `EVAL_*` 模型配置，再准备数据并
+运行完整 LoCoMo 评测。以下命令会调用配置的后端和模型 API：
+
+```bash
 python data/locomo/prepare_locomo.py
 ./scripts/run_locomo_eval.sh --lib memos --env .env.memos
 ```
@@ -67,14 +55,21 @@ python data/locomo/prepare_locomo.py
 
 ```bash
 cp env_examples/.env.agent .env.agent
+```
+
+按 [Agent Memory 指南](./docs/agent_memory/README_zh.md) 完成 OpenClaw 安装和凭证
+配置后，使用一个 reasoning 任务进行 smoke 检查：
+
+```bash
 mkdir -p data/agentbench
-huggingface-cli download EverMind-AI/EvoAgentBench \
+hf download EverMind-AI/EvoAgentBench \
   --repo-type dataset \
   --local-dir ./data/agentbench
 ./scripts/run_agent_eval.sh \
   --agent openclaw \
   --domain reasoning \
   --protocol test_only \
+  --test-split 1 \
   --version smoke_agentbench \
   --trials 1 \
   --parallel 1
@@ -109,8 +104,15 @@ scripts/
   utils/                      # 通用工具
 requirements_user_memory.txt  # User Memory Evaluation 依赖
 requirements_agentbench.txt   # Agent Memory Evaluation 依赖
+requirements-dev.txt          # 离线测试和开发检查依赖
 ```
 
+## 参与贡献
+
+分支和 PR 流程、adapter 接入及评测结果提交要求见
+[贡献指南（英文）](./CONTRIBUTING.md)。离线检查和 CI 用法见
+[测试指南（英文）](./scripts/tests/README.md)。
+
 ## License
 
 见 [LICENSE](./LICENSE)。第三方 benchmark 数据保留其上游许可证；OmniMemEval 代码许可证不重新授权外部数据集。详见 [THIRD_PARTY_NOTICES.md](./THIRD_PARTY_NOTICES.md)。
diff --git a/docs/README.md b/docs/README.md
@@ -20,4 +20,5 @@ Related documentation remains next to the files it describes:
   [env_examples/README.md](../env_examples/README.md) and
   [env_examples/PARAMETERS.md](../env_examples/PARAMETERS.md)
 - Test commands: [scripts/tests/README.md](../scripts/tests/README.md)
+- Contribution and result submission rules: [CONTRIBUTING.md](../CONTRIBUTING.md)
 - Third-party attribution: [THIRD_PARTY_NOTICES.md](../THIRD_PARTY_NOTICES.md)
diff --git a/docs/README_zh.md b/docs/README_zh.md
@@ -19,4 +19,5 @@ OmniMemEval 包含两条互补的评测线。请根据待评测对象选择对
   [env_examples/README.md](../env_examples/README.md) 和
   [env_examples/PARAMETERS.md](../env_examples/PARAMETERS.md)
 - 测试命令：[scripts/tests/README.md](../scripts/tests/README.md)
+- 开发与结果提交规范：[CONTRIBUTING.md（英文）](../CONTRIBUTING.md)
 - 第三方归属：[THIRD_PARTY_NOTICES.md](../THIRD_PARTY_NOTICES.md)
diff --git a/docs/agent_memory/README.md b/docs/agent_memory/README.md
@@ -119,7 +119,7 @@ Run the following commands from the repository root:
 
 ```bash
 mkdir -p data/agentbench
-huggingface-cli download EverMind-AI/EvoAgentBench \
+hf download EverMind-AI/EvoAgentBench \
   --repo-type dataset \
   --local-dir ./data/agentbench
 ```
@@ -135,6 +135,8 @@ data/agentbench/
   swebench/
 ```
 
+The `hf` CLI is provided by `huggingface_hub`; see the
+[upstream CLI guide](https://huggingface.co/docs/huggingface_hub/en/guides/cli).
 Domain data paths are configured under `configs/agentbench/domains/*.yaml`. If a custom data location is used, update the relevant domain yaml fields.
 
 LiveCodeBench requires its official verifier package:
@@ -164,19 +166,18 @@ systemctl start docker
 
 ## Quick Start
 
-Reasoning smoke run:
+After completing environment, credentials, and data preparation above, run one
+reasoning task. `--test-split 1` selects the first test question for this domain;
+omit it to evaluate the complete test split. `--parallel` controls concurrency,
+not the number of selected tasks.
 
 ```bash
-cp env_examples/.env.agent .env.agent
-mkdir -p data/agentbench
-huggingface-cli download EverMind-AI/EvoAgentBench \
-  --repo-type dataset \
-  --local-dir ./data/agentbench
 ./scripts/run_agent_eval.sh \
   --agent openclaw \
   --domain reasoning \
   --protocol test_only \
   --version smoke_agentbench \
+  --test-split 1 \
   --trials 1 \
   --parallel 1
 ```