keyword-extract gives an archival document two kinds of keyword, kept apart. Statistical keywords (KeyBERT by default, YAKE, or the legacy KER) describe the document and each of its pages. Controlled keywords assign each line a term of the AMČR keyword lists and the TEATER thesaurus, chosen by a language model, with the records behind the term. Every keyword names the method that produced it, with its score and rank. Use it on the ATRIUM document record after nlp-enrich; the controlled kind needs a language-model service and is research output.
Releases · MIT licence · ATRIUM documentation
This repository was assembled on 1 October 2026 from the keyword extraction of
atrium-nlp-enrich (keywords.py) and of atrium-llm-enrich (the LLM
engine and the vocabulary code), after the meeting of 30 September
(#1); nlp-enrich keeps the LINDAT calls (UDPipe,
NameTag). The images of those two repositories that carried these stages stay published and get no new tags.
| Input | Output |
|---|---|
the ATRIUM document record after nlp-enrich: its lines[].text, lemma, upos |
statistical keywords of the document and of each page; the record with its keywords block |
| the same record, with a language-model service configured | per line, an AMČR/TEATER term with its ids, the Czech and English keywords of the line, entity links; the enrichment block |
a plain text (/extract_keywords_text) |
its statistical keywords |
| for the batch command line: nlp-enrich's CoNLL-U or TEITOK files | a summary CSV and one CSV per document (outputs) |
The record follows the hub's document contract;
lines labelled Trash, Garbage, Inverted or Empty are left out.
The service from the published image, on a sentence, with YAKE (no model to download):
docker run --rm -p 8000:8000 ghcr.io/ufal/atrium-keyword-extract-api:<version> # the release without its v
curl --fail-with-body -sS -X POST localhost:8000/extract_keywords_text -H 'Content-Type: application/json' \
-d '{"text": "Archeologický výzkum odkryl zahloubený objekt se sídlištní keramikou.", "method": "yake", "kind": "statistical", "num_keywords": 5}'Expected result (abridged):
{"kind": "statistical", "method_used": "yake", "words": 8,
"keywords": [{"keyword": "Archeologický výzkum odkryl", "score": 1.0, "method": "yake", "rank": 1}, "…"],
"kinds": [{"kind": "statistical", "status": "ok", "detail": null}]}POST /extract_keywords takes a record (-F "document_json=@<doc>.document.json;type=application/json") and also
returns per-page keywords and the record with its blocks. KeyBERT, the default method, loads its embedding model from
the Hugging Face Hub at start-up. The controlled kind (kind=controlled or both) calls OpenRouter or a local
Ollama, set by LLM_BACKEND; without one it answers 501, or reports itself skipped under kind=both. Every
option: service/README.md.
The statistical kind runs one of three methods over the record's text, for the whole document and page by page; scores compare only within a method. The controlled kind sends each line, with a prompt that carries the whole vocabulary (about 58k tokens), to a language model whose answer is constrained to the vocabulary's 4,711 terms, and writes the chosen term with its AMČR and TEATER ids. The details, and the batch and research paths: docs/how-it-works.md.
keyword-extract comes after nlp-enrich, which gives the record its lemmas and parts of speech, and on the
born-digital route after digital-convert. It writes the record's keywords and enrichment blocks, which
nlp-enrich's /project_record can carry into the TEITOK header. The vocabulary build is also released as
atrium-vocabulary-<version>.zip, which the translator and the end-to-end tests read. The whole chain: the
hub's workflow page.
- The controlled kind is not evaluated. The rubric is decided, but the annotated gold set does not exist yet, so treat its output as research output (#2).
- A constrained answer stays in the vocabulary, not on topic: a model can be forced onto an unrelated term that shares the start of the word it began (Known limits).
- The prompt is about 58k tokens. A model served with a smaller window sees only part of the vocabulary; the paradata records how many terms were cut.
- YAKE is AGPL-3.0: a run that uses it is declared AGPL-3.0 (licences).
- The research path (
llm_run.py) needs a GPU and local weights; the service needs neither.
| To | Read |
|---|---|
| set it up, run the service, the batch CLI, the images | docs/usage.md |
| deploy and operate the service, its options, limits, errors | service/README.md |
| see the API contract | service/openapi.json |
| understand the two kinds and the vocabulary | docs/how-it-works.md |
| read the batch CLI's outputs and scores | docs/reference/outputs.md |
| see the licences of a run | docs/reference/licences.md |
| build or review the vocabulary | data_samples/vocab/RUNBOOK.md |
| change the controlled kind's prompt | prompts/RUNBOOK.md |
| contribute, test, release | CONTRIBUTING.md |
The documentation follows the ATRIUM documentation style.
The code is MIT (LICENSE). It uses KeyBERT with sentence-transformers (MIT, Apache-2.0), YAKE (AGPL-3.0) and KER (MIT), and the AMČR and TEATER vocabularies (CC0); the language model of the controlled kind is the deployment's choice, and every run records its id. The inventory is para_config.txt; what a run records is in docs/reference/licences.md. Cite with CITATION.cff.
Developed by ÚFAL, Charles University. Funded by ATRIUM (GA 101132163), with the support of the LINDAT/CLARIAH-CZ research infrastructure and the AIS CR infrastructure (LM2023031). Contact: the issue tracker.