Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion docs/conllu-conventions.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,9 +137,19 @@ Participle tags (`v.prap.`, `v.prpp.`, `v.pfap.`, `v.pfpp.`) map to `VerbForm=Pa

## MISC

The column is typed key by key in `@interslavic/conllu` (`Misc`): an
attribute not listed there is a type error, not a new string in the column.

| Key | Meaning | Set by |
|---|---|---|
| `SID` | Steenbergen id from source row | `inflect()`, passed from caller |
| `SID` | Identity of the entry: the Steenbergen id of the source row, or `<id>-<suffix>` for a lexeme `derive()` produced | `inflect()`, passed from caller; `derive()` |
| `LID` | Steenbergen id of the row a derived lexeme descends from | `derive()`, when given a `lid` |
| `Derivation` | Which rule produced a derived lexeme, as a `DerivationCode` (`v_prap`, `adj_adv`) | `derive()` |
| `Declension` | `Adj` on a noun that is an adjective used as one (`učeny`, `prěměnna`). UPOS says `NOUN`, so nothing else records which tables the forms came from | `inflect()` |
| `SpaceAfter` | `No` where the original text had no space | `splitTokens()` |

The golden dump carries sid, lemma, form and FEATS, not MISC, so a change to
an attribute above moves snapshots and nothing else.

## Normalization

Expand Down
9 changes: 7 additions & 2 deletions packages/conllu/src/format.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -129,15 +129,20 @@ describe('formatTokenAsConllu', () => {
);
});

test('drops an attribute that arrived as an explicit undefined', () => {
const token: Token = { form: 'a', misc: { SID: '1', LID: undefined } };
assert.equal(columns(formatTokenAsConllu(token))[9], 'SID=1');
});

// `Key=` is not a legal MISC entry, so an empty value drops out entirely.
test('empty values are dropped', () => {
const token: Token = { form: 'a', misc: { SID: '1', Gloss: '' } };
const token: Token = { form: 'a', misc: { SID: '1', LID: '' } };
assert.equal(columns(formatTokenAsConllu(token))[9], 'SID=1');
});

test('a bag of nothing but empty values is _', () => {
assert.equal(
columns(formatTokenAsConllu({ form: 'a', misc: { Gloss: '' } }))[9],
columns(formatTokenAsConllu({ form: 'a', misc: { LID: '' } }))[9],
'_',
);
});
Expand Down
7 changes: 5 additions & 2 deletions packages/conllu/src/format.ts
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
import type { Token } from './schema.ts';
import type { Misc } from './misc.ts';

// An empty value would print a bare `Key=`, which is not a legal MISC entry.
function formatMisc(misc?: Record<string, string>): string {
// An optional property spread from another bundle can also arrive as an
// explicit `undefined`; that is an absent attribute, not an empty one.
function formatMisc(misc?: Misc): string {
const pairs = Object.entries(misc ?? {})
.filter(([, value]) => value !== '')
.filter(([, value]) => value != null && value !== '')
.map(([key, value]) => `${key}=${value}`);

return pairs.length > 0 ? pairs.join('|') : '_';
Expand Down
1 change: 1 addition & 0 deletions packages/conllu/src/index.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
export * from './features-atomic.ts';
export * from './features-pos.ts';
export * from './format.ts';
export * from './misc.ts';
export * from './schema.ts';
export * from './split-tokens.ts';
export * from './upos.ts';
60 changes: 60 additions & 0 deletions packages/conllu/src/misc.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
/**
* The MISC column, spelled out key by key.
*
* MISC takes any annotation with no home in the standard columns, so left
* untyped it accumulates keys nobody declared. Naming every key it may carry
* makes adding one a deliberate edit, next to what is already there, and makes
* a collision with an existing key a type error.
*
* @see https://universaldependencies.org/format.html
*/
export type Misc = {
// Attributes the CoNLL-U format itself defines.

/** No whitespace between this token and the next one in the original text. */
SpaceAfter?: 'No';

// Attributes of this project.

/** Steenbergen id of the dictionary entry the token was inflected from. */
LID?: string;
/**
* Identifier of the entry a token belongs to: the `LID` itself for a word
* out of the dictionary, and `<LID>-<suffix>` for one `derive` produced.
*/
SID?: string;
/**
* Which rule produced a derived token. Absent on tokens that are dictionary
* entries in their own right, however they inflect.
*/
Derivation?: DerivationCode;
/**
* `Adj` on a lemma that is an adjective used as a noun. UPOS says `NOUN`,
* as the lexicon means it to for `učeny` or `prěměnna`, so this is the only
* place the adjectival declension is visible.
*/
Declension?: 'Adj';
};

/**
* What `Derivation` may say: one code per rule that turns a dictionary entry
* into a lexeme of its own. The codes live next to the key they fill rather
* than beside the rules that use them, so a rule added without a code is a
* type error.
*/
export type DerivationCode =
// Verb derivations
| 'v_prap' // Present active participle (dělajųći)
| 'v_prpp' // Present passive participle (dělajemy)
| 'v_pfap' // Past active participle (sdělavši)
| 'v_pfpp' // Past passive participle (dělany)
| 'v_noun' // Verbal noun (dělanje)
// Adjective derivations
| 'adj_cmp' // Comparative adjective, synthetic (dobrějši)
| 'adj_sup' // Superlative adjective, synthetic (najdobrějši)
| 'adj_sup_a' // Superlative adjective, analytical (najdobry)
| 'adj_adv' // Adverb positive (dobro)
// Adverb derivations
| 'adv_cmp' // Comparative adverb, synthetic (bolje)
| 'adv_sup' // Superlative adverb, synthetic (najbolje)
| 'adv_sup_a'; // Superlative adverb, analytical (najdobro)
5 changes: 4 additions & 1 deletion packages/conllu/src/schema.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import type { Misc } from './misc.ts';
import type * as FeaturesPOS from './features-pos.ts';
import type { UPOS } from './upos.ts';

Expand Down Expand Up @@ -41,7 +42,7 @@ export interface Token<POS extends UPOS = UPOS> {
head?: number;
deprel?: string;
deps?: Array<{ head: number; deprel: string }>;
misc?: Record<string, string>;
misc?: Misc;
}

export type TokenFeatures = FeaturesPOS.Any;
Expand All @@ -54,4 +55,6 @@ export type AdapterResult<POS extends UPOS = UPOS> = Array<{
form: string;
upos: POS;
feats: POSFeatures[POS];
/** Annotation with no home in FEATS, merged into the token's MISC column. */
misc?: Misc;
}>;
5 changes: 4 additions & 1 deletion packages/conllu/src/split-tokens.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import type { Token } from './schema.ts';
import type { Misc } from './misc.ts';

type TokenClass = 'word' | 'punct' | 'space' | 'number' | 'dash';

Expand Down Expand Up @@ -113,7 +114,9 @@ export function splitTokens(str: string): Token[] {

return rawTokens.map((raw, index) => {
const id = index + 1;
const misc = raw.hasSpaceAfter ? undefined : { SpaceAfter: 'No' };
const misc: Misc | undefined = raw.hasSpaceAfter
? undefined
: { SpaceAfter: 'No' };

if (raw.class === 'punct') {
return {
Expand Down
4 changes: 2 additions & 2 deletions packages/morphology/src/derive/derive.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import type { Token } from '@interslavic/conllu';
import type { Misc, Token } from '@interslavic/conllu';
import type { Lemma } from '../schema.ts';
import type { KnownForm } from '../inflect/inflect.ts';
import { normalize, type SteenInput } from '../normalize.ts';
Expand Down Expand Up @@ -155,7 +155,7 @@ export function derive(
feats.Gender = 'Neut';
}

const misc: Record<string, string> = {
const misc: Misc = {
Derivation: mapping.code,
};

Expand Down
25 changes: 4 additions & 21 deletions packages/morphology/src/derive/virtual-id.ts
Original file line number Diff line number Diff line change
@@ -1,24 +1,7 @@
/**
* Short canonical derivation identifiers for CoNLL-U misc.Derivation.
*/
export type DerivationCode =
// Verb derivations
| 'v_prap' // Present active participle (dělajųći)
| 'v_prpp' // Present passive participle (dělajemy)
| 'v_pfap' // Past active participle (sdělavši)
| 'v_pfpp' // Past passive participle (dělany)
| 'v_noun' // Verbal noun (dělanje)
// Adjective derivations
| 'adj_cmp' // Comparative adjective, synthetic (dobrějši)
| 'adj_sup' // Superlative adjective, synthetic (najdobrějši)
| 'adj_sup_a' // Superlative adjective, analytical (najdobry)
| 'adj_adv' // Adverb positive (dobro)
// Adverb derivations
| 'adv_cmp' // Comparative adverb, synthetic (bolje)
| 'adv_sup' // Superlative adverb, synthetic (najbolje)
| 'adv_sup_a' // Superlative adverb, analytical (najdobro)
// General word formation
| string;
import type { DerivationCode } from '@interslavic/conllu';

// The codes themselves are declared beside the MISC key they fill.
export type { DerivationCode };

export type DerivationType =
| 'prap'
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ exports[`conllufy - nouns 20837 1`] = `
exports[`conllufy - nouns 22830 1`] = `
"znajema znajema NOUN f. Case=Nom|Gender=Fem|Number=Sing
znajeme znajema NOUN f. Case=Nom|Gender=Fem|Number=Plur
znajemo znajema NOUN f. Case=Acc|Gender=Fem|Number=Sing
znajemų znajema NOUN f. Case=Acc|Gender=Fem|Number=Sing
znajeme znajema NOUN f. Case=Acc|Gender=Fem|Number=Plur
znajemoj znajema NOUN f. Case=Gen|Gender=Fem|Number=Sing
znajemyh znajema NOUN f. Case=Gen|Gender=Fem|Number=Plur
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,9 @@ exports[`conllufy - numerals 2 1`] = `
četvŕtině četvŕtina NOUN num.fract. Case=Dat|Gender=Fem|NumType=Frac|Number=Sing
četvŕtinam četvŕtina NOUN num.fract. Case=Dat|Gender=Fem|NumType=Frac|Number=Plur
četvŕtinojų četvŕtina NOUN num.fract. Case=Ins|Gender=Fem|NumType=Frac|Number=Sing
četvŕtinami četvŕtina NOUN num.fract. Case=Ins|Gender=Fem|NumType=Frac|Number=Plur"
četvŕtinami četvŕtina NOUN num.fract. Case=Ins|Gender=Fem|NumType=Frac|Number=Plur
četvŕtino četvŕtina NOUN num.fract. Case=Voc|Gender=Fem|NumType=Frac|Number=Sing
četvŕtiny četvŕtina NOUN num.fract. Case=Voc|Gender=Fem|NumType=Frac|Number=Plur"
`;

exports[`conllufy - numerals 3 1`] = `
Expand Down Expand Up @@ -95,7 +97,9 @@ exports[`conllufy - numerals 4 1`] = `
četverkě četverka NOUN num.subst. Case=Dat|Gender=Fem|Number=Sing
četverkam četverka NOUN num.subst. Case=Dat|Gender=Fem|Number=Plur
četverkojų četverka NOUN num.subst. Case=Ins|Gender=Fem|Number=Sing
četverkami četverka NOUN num.subst. Case=Ins|Gender=Fem|Number=Plur"
četverkami četverka NOUN num.subst. Case=Ins|Gender=Fem|Number=Plur
četverko četverka NOUN num.subst. Case=Voc|Gender=Fem|Number=Sing
četverky četverka NOUN num.subst. Case=Voc|Gender=Fem|Number=Plur"
`;

exports[`conllufy - numerals 6 1`] = `
Expand Down
11 changes: 6 additions & 5 deletions packages/morphology/src/inflect/inflect-word.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { parsePos } from '../partOfSpeech/index.ts';
import type { AdapterResult, Token, XPOS } from '@interslavic/conllu';
import type { AdapterResult, Misc, Token, XPOS } from '@interslavic/conllu';
import { etymologify } from '../etymologify.ts';
import { parseXPos } from '../steen/index.ts';
import {
Expand All @@ -15,7 +15,7 @@ function composeTokens(
results: AdapterResult,
lemma: string,
xpos: XPOS,
misc?: Record<string, string>,
misc?: Misc,
): Token[] {
return results.map((result) => ({
form: etymologify(result.form),
Expand All @@ -24,8 +24,9 @@ function composeTokens(
xpos,
feats: result.feats,
// A copy per token: one shared object would let a consumer annotating one
// form of the paradigm annotate all the others behind its back.
misc: misc && { ...misc },
// form of the paradigm annotate all the others behind its back. What the
// adapter has to say comes first, so a caller can override it.
misc: misc || result.misc ? { ...result.misc, ...misc } : undefined,
}));
}

Expand All @@ -34,7 +35,7 @@ export function inflectWord(
xpos: XPOS,
lemma: string,
irregular?: string,
misc?: Record<string, string>,
misc?: Misc,
): Token[] {
if (!word || typeof word !== 'string' || word.includes(' ')) {
return [];
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -148264,7 +148264,7 @@ exports[`noun feminine 21351 1`] = `
"acc": [
[
{
"form": "narěčeno",
"form": "narěčenų",
},
],
[
Expand Down Expand Up @@ -155079,7 +155079,7 @@ exports[`noun feminine 22830 1`] = `
"acc": [
[
{
"form": "znajemo",
"form": "znajemų",
},
],
[
Expand Down Expand Up @@ -156126,7 +156126,7 @@ exports[`noun feminine 23050 1`] = `
"acc": [
[
{
"form": "doråstlo",
"form": "doråstlų",
},
],
[
Expand Down Expand Up @@ -156215,7 +156215,7 @@ exports[`noun feminine 23051 1`] = `
"acc": [
[
{
"form": "vȯzråstlo",
"form": "vȯzråstlų",
},
],
[
Expand Down Expand Up @@ -159526,7 +159526,7 @@ exports[`noun feminine 23429 1`] = `
"acc": [
[
{
"form": "zapęto",
"form": "zapętų",
},
],
[
Expand Down Expand Up @@ -186832,7 +186832,7 @@ exports[`noun feminine 26879 1`] = `
"acc": [
[
{
"form": "krivo",
"form": "krivų",
},
],
[
Expand Down Expand Up @@ -235832,7 +235832,7 @@ exports[`noun feminine 35526 1`] = `
"acc": [
[
{
"form": "prěměnno",
"form": "prěměnnų",
},
],
[
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40694,7 +40694,7 @@ exports[`noun neuter 12566 1`] = `
"acc": [
[
{
"form": "srědnji",
"form": "srědnje",
},
],
[
Expand Down Expand Up @@ -72614,7 +72614,7 @@ exports[`noun neuter 23160 1`] = `
"acc": [
[
{
"form": "šampansky",
"form": "šampansko",
},
],
[
Expand Down
Loading