Skip to content

feat: add MEILISEARCH_COURSE_INDEXING to scope Studio course indexing - #39041

Open
blarghmatey wants to merge 1 commit into
openedx:masterfrom
mitodl:tmacey/meilisearch-course-indexing-mode
Open

feat: add MEILISEARCH_COURSE_INDEXING to scope Studio course indexing#39041
blarghmatey wants to merge 1 commit into
openedx:masterfrom
mitodl:tmacey/meilisearch-course-indexing-mode

Conversation

@blarghmatey

Copy link
Copy Markdown
Contributor

What are the relevant tickets?

N/A

Description (What does it do?)

openedx/core/djangoapps/content/search is gated only on MEILISEARCH_ENABLED, so an operator running Meilisearch for Libraries V2 necessarily also indexes every course XBlock into studio_content. There is currently no way to have one without the other.

This adds MEILISEARCH_COURSE_INDEXING, which takes three values:

  • all — every course XBlock. The default, so behaviour is unchanged for everyone who does not set it.
  • library_downstream_only — only course blocks carrying an upstream link.
  • none — no course content, and no indexing task is even enqueued.

Library content (Libraries V2 blocks, collections, containers) is indexed in every mode.

library_downstream_only is the interesting one, and it lines up with an existing invariant rather than inventing a new rule.

frontend-app-authoring's course-libraries Review tab lists library components in a course that have upstream updates ready to sync. It gets the authoritative list from the entity-links REST API, then queries this index purely to hydrate those blocks for display:

// src/course-libraries/ReviewTabContent.tsx
extraFilter={[`context_key = "${courseId}"`, `usage_key IN ["${downstreamKeys?.join('","')}"]`]}

where downstreamKeys is outOfSyncItems.map(link => link.downstreamUsageKey). Every one of those keys belongs to a PublishableEntityLink row, and create_or_update_xblock_upstream_link (cms/djangoapps/contentstore/utils.py) opens with if not xblock.upstream: return None. So a link exists only where xblock.upstream is set — which is exactly what the gate keeps. That holds for the use_top_level_parents=True case the Review tab passes too, since the substituted parents are themselves ContainerLink rows.

What it gives up is the Studio course content search modal and the block-type breakdown in the course outline info sidebar. LMS-side courseware search and discovery are untouched: openedx.core.djangoapps.content.search is in cms/envs/common.py's INSTALLED_APPS only, not the LMS's.

Details worth noting in review:

  • The gate is should_index_course_block(block) in api.py, testing getattr(block, "upstream", None). upstream is a String field on UpstreamSyncMixin (cms/lib/xblock/upstream_sync.py), which is in the CMS XBLOCK_MIXINS, so the test needs no extra DB round trip.
  • index_course and upsert_xblock_index_doc filter each block but recurse regardless, since an unlinked section can contain linked children.
  • rebuild_index skips the Courses section entirely (including the CourseOverview count) only when the mode is none.
  • Deletes are deliberately left ungated — xblock_deleted_handler and listen_for_course_delete — so that widening the setting later does not inherit index documents for content that was removed while the scope was narrow.
  • Switching modes takes full effect after ./manage.py cms reindex_studio, so the scope is reversible by configuration rather than by reverting a commit.
  • Not gated: the tag/collection stub-doc path (upsert_content_object_tags_index_doc and friends). It is driven by manual tagging at human scale, and gating it would require a modulestore load inside a signal handler for negligible benefit.

What motivated it, from one production instance on 2026-08-28: studio_content held 1,319,937 documents at 11.91 GiB against a 4 GiB pod memory limit. 966 of those documents were library content. The rest were course XBlocks, written by ordinary publishes plus a batch of course reruns and imports. That instance runs Meilisearch solely for Libraries V2; Studio course search is not a feature it uses. The index size was in turn making library component creation slow enough to hit a gateway timeout.

How can this be tested?

Automated. New tests in openedx/core/djangoapps/content/search/tests/test_api.py and tests/test_handlers.py cover: none writing nothing; library_downstream_only skipping an unlinked block while still indexing one whose upstream is set, reached through a skipped ancestor; a rebuild under none emitting library_block docs and no course_block docs; and an unrecognised setting value falling back to all. These have not been executed — I have no local edx-platform environment, so they have had a compile and line-length check only. CI needs to run them.

Manual, against a Studio with Meilisearch enabled and at least one V2 library:

  1. Leave the setting unset. Publish a course block and confirm it still reaches the index — this is the no-op path that protects existing operators.
  2. Set MEILISEARCH_COURSE_INDEXING = "library_downstream_only", restart, and run ./manage.py cms reindex_studio --experimental. Confirm the index drops to library content plus downstream course blocks.
  3. Publish, import and rerun a course. Confirm no document writes for blocks with no upstream.
  4. Create, edit, publish and delete a V2 library component. Each should still reach the index.
  5. On a course with a library component that has a pending upstream update, confirm the course-libraries Review tab still lists it and renders its display data. This is the criterion the design exists to satisfy.
  6. Confirm the Studio course content search modal and the outline block-type facets are empty, which is the accepted cost.
  7. Set the value back to all, restart, re-run reindex_studio, and confirm course documents fully repopulate.

Note for step 7: use the non-incremental form. Meilisearch's LMDB store never returns freed pages to the filesystem, so narrowing the scope only reclaims disk through the temp-index-and-swap path.

content/search is gated only on MEILISEARCH_ENABLED, so an operator running
Meilisearch for Libraries V2 also gets every course XBlock written to
studio_content. On one production instance that was 1,318,971 of 1,319,937
documents and 11.91 GiB against a 4Gi pod, with slow index batches pushing
library component creation past the gateway timeout.

MEILISEARCH_COURSE_INDEXING takes "all" (the default, today's behaviour),
"library_downstream_only" or "none". library_downstream_only keeps course
blocks that carry an `upstream` link, which is the set the Authoring MFE's
course-libraries Review tab queries by usage_key to hydrate components with
pending upstream updates; the Studio course content search modal and the
course outline block-type facets go empty. The recursive walks still descend
through skipped blocks, since an unlinked section can contain linked children.

Deletes stay ungated so that re-enabling a mode does not inherit documents
for content removed while indexing was narrowed. Switching modes takes full
effect after ./manage.py cms reindex_studio, so the scope is reversible
without a code change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QKxn25wDBC4k5HouhQDBrx
@openedx-webhooks openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label Aug 28, 2026
@openedx-webhooks

Copy link
Copy Markdown

Thanks for the pull request, @blarghmatey!

This repository is currently maintained by @openedx/wg-maintenance-openedx-platform-oncall.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

  • If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
    • This process (including the steps you'll need to take) is documented here.
  • If it doesn't, simply proceed with the next step.
🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

  • Dependencies

    This PR must be merged before / after / at the same time as ...

  • Blockers

    This PR is waiting for OEP-1234 to be accepted.

  • Timeline information

    This PR must be merged by XX date because ...

  • Partner information

    This is for a course on edx.org.

  • Supporting documentation
  • Relevant Open edX discussion forum threads
🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details
Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

  • The size and impact of the changes that it introduces
  • The need for product review
  • Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

open-source-contribution PR author is not from Axim or 2U

Projects

Status: Needs Triage

Development

Successfully merging this pull request may close these issues.

2 participants