Skip to content

feat(release): add /rhdh-kb-theme-analysis for public KB theme analysis - #159

Closed
mattreid wants to merge 6 commits into
redhat-developer:mainfrom
mattreid:codex/kb-theme-analysis
Closed

mattreid wants to merge 6 commits into
redhat-developer:mainfrom
mattreid:codex/kb-theme-analysis

Conversation

@mattreid

@mattreid mattreid commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

What this adds

/rhdh-kb-theme-analysis answers "what are customers hitting with RHDH this quarter?" from
public Red Hat KB data instead of from memory.

Ask a question and it pulls every RHDH Solution Article from the Customer Portal knowledge
base, then reports two views that are never mixed: what support published in a window
(createdDate), and what customers hit, weighted by each article's linked support cases
across the whole corpus. It groups articles into support themes, reports each theme's share,
per-day rate, and case weight, and links every article it names back to its KB page so the
answer can be checked.

Typical asks:

  • "What support themes are growing for RHDH this quarter?"
  • "Which themes should we look at before 1.10 ships?"
  • "Group these KB articles under the taxonomy I care about."

It is read-only and public

The skill reads the public KB search API and nothing else. No Jira, no case records, no
private data. Release questions that need Jira stay with /rhdh-release-status, and writing
documentation stays with /rhdh-feature-docs.

What the PR does

  • Adds the skill, built on one dependency-free Node 18+ script: no install step, no build,
    no package.json.
  • Makes the taxonomy a parameter. The built-in clusters are the default, --themes PATH
    applies a taxonomy written for a specific question, and --themes none returns the
    articles for the agent to group itself.
  • Separates authoring from demand. The rolling windows keep measuring publication; a
    separate demand view weights the whole corpus by caseCount_365 and ignores publication
    date, because that API count cannot be re-windowed below a year.
  • Divides shared articles across their products. A 9-product article no longer counts as nine
    times the demand of a product-specific one: every theme and article carries casesRaw,
    casesAttributed, and casesDedicated.
  • Classifies theme movement against a baseline that excludes the recent window, on share
    rather than absolute rate, and states corpusRateRatio so a corpus-wide slowdown is not
    reported as eight separate theme trends.
  • Adds KCS state and a refresh list for stale, high-touch articles, component folding with an
    optional --component-aliases file, and emergent vocabulary mined from titles and tags
    with floors that survive a multiple-comparisons check.
  • Adds three references: how to interpret the output, how the KB API behaves, and when a
    taxonomy needs replacing.
  • Registers the skill in the catalog, the /ask-rhdh routing table, the README, and the
    install manifest.
  • Replaces three per-skill tests with one generic installable-group test, and adds 64 script
    tests.

What the output gives you

Two views, deliberately kept apart. The rolling windows measure what support published; the
demand view weights the whole corpus by caseCount_365 and ignores publication date. Case
weight is annual by construction — the API publishes a fixed trailing-year count — so the
skill says so rather than implying a quarterly case number.

Per theme and per article the report carries three case figures: casesRaw (plain sum),
casesAttributed (each article divided across the products it is tagged to), and
casesDedicated (single-product articles only).

Against the live corpus that changes the story. Of 172 RHDH articles, 81 carry any linked
cases, summing to 186 raw and 110.2 attributed. A single 9-product CVE article holds 44.6% of
raw case weight (83 cases) but only 8.4% attributed; it is now named under "Corpus
concentration" instead of quietly setting the tone of a theme total. support-version-clarity
reads 102 raw cases and 28.2 attributed — the difference is that same article.

The report also adds KCS state (154 verified, 13 unverified, 5 work-in-progress) and a refresh
list of 11 articles that carry cases and have gone unedited for over a year; theme movement
classified on share with the corpus rate stated alongside, which is what separates a real
trend from a corpus-wide slowdown; and emergent vocabulary mined from titles and tags, floored
so that chance words do not appear as findings.

The built-in taxonomy is versioned as built-in-v2, and the report names the version it used.
Version 1 matched bare product nouns — plugins?, catalogs?, ui, custom, entity —
which took upgrade-plugin-reliability to 66% of a window and made
catalog-database-scale mostly find articles about the software catalog. Version 2 requires
plugin-lifecycle and catalog or data-layer context; catalog-database-scale fell from 39
articles and 22 attributed cases to 9 and 10.

Verification

  • node --test "skills/**/tests/*.test.mjs": 132 pass, 64 for this skill.
  • uv run pytest: 991 pass. The 6 failures in test_base_images_and_rpms.py and
    test_check_trusted_tasks.py reproduce on the base commit and are macOS-only.
  • python3 scripts/validate_skill_catalog.py --root .: valid.
  • Run against access.redhat.com over the default 90/365 window, a calendar quarter, 120,
    150 and 180-day windows, a 730-day comparison, and a 180/360 halves comparison.

Adds a read-only skill that pulls Red Hat Customer Portal KB Solution
articles for Red Hat Developer Hub and compares a rolling 90-day window
against a rolling 365-day window.

The taxonomy is data rather than code: the built-in clusters stay the
default, --themes PATH applies a taxonomy proposed for the investigation,
and --themes none hands the article corpus to the agent to group.

The report and JSON carry the signals needed to judge a taxonomy:
per-window unmatched counts, and a warning when a single article supplies
most of a cluster's caseCount365.

- dependency-free Node 18+ script; no build step, no dependencies
- catalog entry, /ask-rhdh routing table, README row, install manifest
- one generic installable-group test replaces per-skill assertions
- 34 script tests covering windows, taxonomy modes, and report shapes

Assisted-By: Codex
@mattreid

Copy link
Copy Markdown
Contributor Author

@rm3l @schultzp2020 I just put up a PR for a new skill.

I couldn't find an MCP for KB articles (sounds like one may be coming later though), so had Codex write up a script and skill that hits the public site. If this proves to be something we use long term, and we figure out the JEV style classifiers and model access, my assumption is I could then refactor to utilize that for better and cheaper analysis.

It doesn't totally fit with the other release skills, but that felt like the best current categorization. Maybe in the future it should move to a planning or operations bucket?

It was written with GLM and DeepSeek so I could try out ChatGPT on EnMaaS models.

@rm3l rm3l left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, works great. Thanks @mattreid !!

The first cut answered "what did support publish recently" while reading
as "what are customers hitting". Of 171 RHDH articles, the 16 created in
the last 90 days carry 10 linked cases, and one 9-product CVE article
holds 83 of the 187 corpus cases. Both distortions are now explicit.

- demand view: whole corpus weighted by caseCount_365, not windowed by
  createdDate, since that count cannot be re-windowed below a year
- casesRaw, casesAttributed (divided across tagged products), and
  casesDedicated (single-product only) per theme and per article
- corpus concentration: largest article named, share of raw and
  attributed weight, warning above 25%
- theme movement classified against a baseline that excludes the recent
  window, with a five-article guard that prints insufficient rather than
  a ratio
- KCS state mix and refresh candidates for stale high-touch articles
- component folding with --component-aliases for real synonyms
- emergent terms mined from titles and tags, with ubiquitous-term and
  singleton floors
- schemaVersion 2; existing JSON keys and report sections preserved
- 53 script tests (was 34)

Assisted-By: Codex
…vered

An explicit --recent-since/--recent-until pair ends before the default
comparison window does, so a calendar quarter's comparison silently
included the following month. The caveat now names the uncovered span,
counts its days, and points at --comparison-until.

Assisted-By: Codex
With 16 recent articles, any word used twice in the year has a 4.6% chance
of landing only in the recent window, so a hundred candidate terms yield
about four false 'new' hits per run. Raise the floor to three recent
articles, drop symptom vocabulary from titles, and state the baseline
length on rising rows so the rate claim is visible.

Assisted-By: Codex
The corpus published 46 articles in the prior half-year and 25 in the
recent half. Rating themes on absolute rate turned that single
corpus-wide slowdown into six 'declining' themes, two of which had
gained share. Classes now read share ratio (1.25x / 0.8x) and the report
states movement.corpusRateRatio so the absolute shift stays visible.

Assisted-By: Codex
Bare product nouns were doing the theme work. plugins? alone appeared in
40% of the corpus, taking upgrade-plugin-reliability to 66% of a window
and inverting the demand ranking; catalogs? found articles about the
software catalog rather than catalog scale, so catalog-database-scale
claimed 39 articles and 22 attributed cases.

Version 2 requires plugin-lifecycle context and catalog or data-layer
context, and drops ui, custom, entity, large, and many. Shares now top out
at 41%, unmatched rises from 1 to 5 of 73 comparison articles. The report
names the version so runs are never compared across sets.

Assisted-By: Codex
@rm3l

rm3l commented Nov 20, 2026

Copy link
Copy Markdown
Member

Moved to internal GL instead: https://gitlab.cee.redhat.com/rhidp/rhdh-internal-skills/-/merge_requests/5

/close

@openshift-ci openshift-ci Bot closed this Nov 20, 2026
@openshift-ci

openshift-ci Bot commented Nov 20, 2026

Copy link
Copy Markdown

@rm3l: Closed this PR.

Details

In response to this:

Moved to internal GL instead: https://gitlab.cee.redhat.com/rhidp/rhdh-internal-skills/-/merge_requests/5

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants