Geometric Characterization of Embedding Spaces Across Chunking Strategies and Embedding Models for Slovenian Text
Authors: Anja Jodl, Mitko Nikov
This repository contains the code and data from the paper "Geometric Characterization of Embedding Spaces Across Chunking Strategies and Embedding Models for Slovenian". The study investigates how different chunking strategies and embedding models affect the geometric properties of embedding spaces for the Slovenian language.
To be published.
The code is based on the semora python package, which is developed by the authors of the paper, as an extensible tool for semantic processing, analysis and retrieval of text.
The ./scripts/main.sh script is the main entry point that does everything from processing the newspapers in the ./data/newspapers directory, to generating the embeddings.
The data is stored in the ./data directory. It includes the statistical data and results from the experiments conducted in the study. Unfortunately, we cannot provide all of the original Slovenian newspapers due to copyright restrictions, however, we provided couple of samples of the newspaper markdown files in the ./samples/newspapers directory.
Results Table: Combined evaluation results by chunking strategy and embedding model.
| Model | Chunking | MNCS ↓ | MPCS ↑ | MTCS ↑ | SDR ↓ | DBI ↓ | MRD ↓ | PC1 ↓ | PC5 ↓ | PC10 ↓ | GNB N/P ↑ | GNB N/T ↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| E5 | P | 0.7770 | 0.8154 | 0.8286 | 10.44% | 4.5847 | 0.0529 | 0.0450 | 0.1156 | 0.1668 | 72.60% | 78.10% |
| E5 | S1/0 | 0.7808 | 0.8002 | 0.8171 | 15.49% | 8.1291 | 0.0745 | 0.0585 | 0.1262 | 0.1803 | 62.44% | 72.16% |
| E5 | S4/0 | 0.7842 | 0.8265 | 0.8469 | 3.53% | 4.3856 | 0.0521 | 0.0305 | 0.1026 | 0.1547 | 75.82% | 86.46% |
| E5 | S4/1 | 0.7854 | 0.8307 | 0.8811 | 0.62% | 4.2962 | 0.0509 | 0.0299 | 0.1025 | 0.1548 | 76.57% | 94.55% |
| E5 | T32/0 | 0.8091 | 0.8286 | 0.8445 | 7.54% | 7.7732 | 0.0796 | 0.0294 | 0.0993 | 0.1511 | 65.59% | 77.09% |
| E5 | T128/0 | 0.7967 | 0.8402 | 0.8589 | 2.69% | 4.2328 | 0.0484 | 0.0275 | 0.1033 | 0.1567 | 77.74% | 87.91% |
| E5 | T128/16 | 0.7980 | 0.8432 | 0.8743 | 0.39% | 4.1806 | 0.0479 | 0.0277 | 0.1026 | 0.1557 | 78.47% | 93.62% |
| E5 | T256/0 | 0.8041 | 0.8605 | 0.8729 | 3.83% | 3.2015 | 0.0471 | 0.0330 | 0.1172 | 0.1751 | 82.87% | 88.27% |
| E5 | T256/32 | 0.8065 | 0.8654 | 0.8866 | 0.38% | 3.1684 | 0.0465 | 0.0340 | 0.1145 | 0.1731 | 84.18% | 93.46% |
| E5 | T256/64 | 0.8078 | 0.8692 | 0.9010 | 0.04% | 3.1413 | 0.0455 | 0.0348 | 0.1146 | 0.1735 | 84.42% | 97.01% |
| E5 | T256/128 | 0.8109 | 0.8739 | 0.9295 | 0.00% | 3.1672 | 0.0415 | 0.0363 | 0.1164 | 0.1756 | 84.36% | 99.23% |
| E5 | T256/192 | 0.8136 | 0.8782 | 0.9578 | 0.00% | 3.1550 | 0.0384 | 0.0375 | 0.1179 | 0.1777 | 84.35% | 99.76% |
| E5 | R400/0 | 0.7794 | 0.8169 | 0.8327 | 10.44% | 4.8714 | 0.0509 | 0.0316 | 0.1048 | 0.1577 | 72.93% | 79.95% |
| E5 | R400/64 | 0.7821 | 0.8230 | 0.8493 | 4.83% | 4.6690 | 0.0508 | 0.0255 | 0.1005 | 0.1539 | 75.20% | 85.39% |
| E5 | R1000/0 | 0.7920 | 0.8423 | 0.8557 | 6.19% | 3.3331 | 0.0485 | 0.0310 | 0.1101 | 0.1667 | 78.46% | 84.62% |
| E5 | R1000/100 | 0.7933 | 0.8461 | 0.8625 | 3.56% | 3.2629 | 0.0485 | 0.0315 | 0.1093 | 0.1656 | 79.45% | 86.95% |
| SloVlo | P | 0.6668 | 0.7383 | 0.7553 | 9.70% | 4.5019 | 0.0895 | 0.0501 | 0.1518 | 0.2203 | 74.18% | 78.17% |
| SloVlo | S1/0 | 0.7052 | 0.7381 | 0.7617 | 18.28% | 7.9990 | 0.1315 | 0.0800 | 0.1849 | 0.2541 | 61.58% | 69.62% |
| SloVlo | S4/0 | 0.6617 | 0.7340 | 0.7707 | 3.20% | 4.2893 | 0.0885 | 0.0346 | 0.1321 | 0.2030 | 75.26% | 85.90% |
| SloVlo | S4/1 | 0.6617 | 0.7396 | 0.8218 | 0.52% | 4.2035 | 0.0857 | 0.0349 | 0.1314 | 0.2024 | 75.64% | 93.16% |
| SloVlo | T32/0 | 0.7254 | 0.7576 | 0.7820 | 9.93% | 7.7314 | 0.1525 | 0.0410 | 0.1395 | 0.2114 | 64.53% | 74.10% |
| SloVlo | T128/0 | 0.6702 | 0.7462 | 0.7802 | 2.36% | 4.1717 | 0.0907 | 0.0367 | 0.1307 | 0.2007 | 76.89% | 87.20% |
| SloVlo | T128/16 | 0.6707 | 0.7504 | 0.8039 | 0.39% | 4.1203 | 0.0894 | 0.0371 | 0.1305 | 0.2008 | 77.62% | 92.02% |
| SloVlo | T256/0 | 0.6718 | 0.7723 | 0.7970 | 2.68% | 3.1663 | 0.0845 | 0.0424 | 0.1393 | 0.2113 | 82.04% | 88.52% |
| SloVlo | T256/32 | 0.6732 | 0.7799 | 0.8215 | 0.26% | 3.1122 | 0.0844 | 0.0435 | 0.1381 | 0.2104 | 83.27% | 93.29% |
| SloVlo | T256/64 | 0.6740 | 0.7851 | 0.8431 | 0.03% | 3.0946 | 0.0826 | 0.0444 | 0.1386 | 0.2109 | 83.22% | 96.19% |
| SloVlo | T256/128 | 0.6774 | 0.7917 | 0.8858 | 0.00% | 3.1325 | 0.0744 | 0.0461 | 0.1406 | 0.2132 | 83.20% | 98.81% |
| SloVlo | T256/192 | 0.6815 | 0.7990 | 0.9319 | 0.00% | 3.1137 | 0.0665 | 0.0478 | 0.1430 | 0.2162 | 83.03% | 99.65% |
| SloVlo | R400/0 | 0.6655 | 0.7298 | 0.7560 | 9.20% | 4.7972 | 0.0898 | 0.0386 | 0.1373 | 0.2067 | 72.63% | 79.39% |
| SloVlo | R400/64 | 0.6668 | 0.7372 | 0.7789 | 5.18% | 4.6042 | 0.0901 | 0.0360 | 0.1325 | 0.2033 | 74.56% | 84.19% |
| SloVlo | R1000/0 | 0.6600 | 0.7544 | 0.7796 | 5.04% | 3.2675 | 0.0821 | 0.0419 | 0.1369 | 0.2082 | 79.18% | 85.76% |
| SloVlo | R1000/100 | 0.6610 | 0.7593 | 0.7901 | 3.41% | 3.1964 | 0.0823 | 0.0423 | 0.1362 | 0.2075 | 80.20% | 87.53% |
| BGE | P | 0.3461 | 0.4442 | 0.4916 | 11.16% | 4.6069 | 0.1597 | 0.0289 | 0.1074 | 0.1655 | 72.46% | 79.08% |
| BGE | S1/0 | 0.3624 | 0.4168 | 0.4701 | 13.68% | 8.0167 | 0.2127 | 0.0402 | 0.1191 | 0.1805 | 63.07% | 74.33% |
| BGE | S4/0 | 0.3645 | 0.4778 | 0.5512 | 3.37% | 4.3839 | 0.1635 | 0.0303 | 0.1072 | 0.1663 | 75.52% | 87.77% |
| BGE | S4/1 | 0.3667 | 0.4900 | 0.6548 | 0.52% | 4.2885 | 0.1601 | 0.0306 | 0.1082 | 0.1675 | 75.77% | 95.05% |
| BGE | T32/0 | 0.3723 | 0.4342 | 0.4974 | 6.19% | 7.5657 | 0.2277 | 0.0246 | 0.0989 | 0.1620 | 66.53% | 79.94% |
| BGE | T128/0 | 0.3824 | 0.5089 | 0.5770 | 2.06% | 4.2126 | 0.1692 | 0.0300 | 0.1094 | 0.1690 | 78.98% | 89.99% |
| BGE | T128/16 | 0.3838 | 0.5179 | 0.6289 | 0.21% | 4.1400 | 0.1680 | 0.0303 | 0.1103 | 0.1697 | 79.14% | 95.12% |
| BGE | T256/0 | 0.3854 | 0.5443 | 0.5950 | 2.35% | 3.2251 | 0.1721 | 0.0315 | 0.1140 | 0.1738 | 82.80% | 90.24% |
| BGE | T256/32 | 0.3884 | 0.5593 | 0.6482 | 0.18% | 3.1521 | 0.1716 | 0.0323 | 0.1159 | 0.1755 | 83.83% | 95.61% |
| BGE | T256/64 | 0.3907 | 0.5698 | 0.6949 | 0.02% | 3.1278 | 0.1684 | 0.0328 | 0.1169 | 0.1768 | 83.87% | 98.00% |
| BGE | T256/128 | 0.3949 | 0.5810 | 0.7850 | 0.00% | 3.1661 | 0.1543 | 0.0335 | 0.1177 | 0.1779 | 83.86% | 99.69% |
| BGE | T256/192 | 0.3985 | 0.5937 | 0.8758 | 0.00% | 3.1389 | 0.1409 | 0.0342 | 0.1179 | 0.1786 | 83.82% | 99.92% |
| BGE | R400/0 | 0.3633 | 0.4673 | 0.5222 | 7.70% | 4.8806 | 0.1637 | 0.0281 | 0.1073 | 0.1678 | 73.58% | 82.31% |
| BGE | R400/64 | 0.3653 | 0.4798 | 0.5669 | 5.49% | 4.6830 | 0.1659 | 0.0287 | 0.1065 | 0.1666 | 75.46% | 86.54% |
| BGE | R1000/0 | 0.3725 | 0.5174 | 0.5649 | 4.91% | 3.3556 | 0.1693 | 0.0309 | 0.1133 | 0.1727 | 79.31% | 86.62% |
| BGE | R1000/100 | 0.3741 | 0.5255 | 0.5824 | 3.64% | 3.2920 | 0.1700 | 0.0313 | 0.1142 | 0.1736 | 80.17% | 88.37% |
| Qwen3 | P | 0.3179 | 0.4060 | 0.4572 | 15.19% | 4.8259 | 0.1274 | 0.0398 | 0.1321 | 0.1973 | 66.83% | 74.08% |
| Qwen3 | S1/0 | 0.2941 | 0.3409 | 0.3835 | 22.61% | 8.9303 | 0.2005 | 0.0555 | 0.1361 | 0.1954 | 59.60% | 67.30% |
| Qwen3 | S4/0 | 0.3474 | 0.4540 | 0.5207 | 6.78% | 4.6226 | 0.1321 | 0.0363 | 0.1211 | 0.1861 | 71.16% | 81.84% |
| Qwen3 | S4/1 | 0.3514 | 0.4658 | 0.6171 | 2.28% | 4.5609 | 0.1318 | 0.0367 | 0.1213 | 0.1863 | 71.58% | 90.42% |
| Qwen3 | T32/0 | 0.3348 | 0.3864 | 0.4275 | 14.59% | 8.5227 | 0.2099 | 0.0291 | 0.0986 | 0.1556 | 61.93% | 70.36% |
| Qwen3 | T128/0 | 0.3673 | 0.4862 | 0.5469 | 4.22% | 4.4442 | 0.1317 | 0.0358 | 0.1210 | 0.1852 | 74.58% | 84.94% |
| Qwen3 | T128/16 | 0.3702 | 0.4945 | 0.5900 | 1.14% | 4.4065 | 0.1317 | 0.0362 | 0.1216 | 0.1855 | 74.58% | 90.22% |
| Qwen3 | T256/0 | 0.3718 | 0.5303 | 0.5805 | 4.50% | 3.3189 | 0.1110 | 0.0382 | 0.1280 | 0.1951 | 78.92% | 86.47% |
| Qwen3 | T256/32 | 0.3766 | 0.5453 | 0.6301 | 0.83% | 3.2641 | 0.1125 | 0.0390 | 0.1298 | 0.1967 | 79.95% | 92.08% |
| Qwen3 | T256/64 | 0.3784 | 0.5559 | 0.6771 | 0.22% | 3.2401 | 0.1135 | 0.0396 | 0.1310 | 0.1982 | 80.43% | 95.33% |
| Qwen3 | T256/128 | 0.3813 | 0.5680 | 0.7698 | 0.01% | 3.2739 | 0.1095 | 0.0407 | 0.1321 | 0.1997 | 80.58% | 98.84% |
| Qwen3 | T256/192 | 0.3827 | 0.5803 | 0.8666 | 0.00% | 3.2512 | 0.1068 | 0.0416 | 0.1325 | 0.2003 | 80.70% | 99.72% |
| Qwen3 | R400/0 | 0.3414 | 0.4377 | 0.4864 | 13.23% | 5.1583 | 0.1312 | 0.0345 | 0.1198 | 0.1852 | 69.26% | 76.54% |
| Qwen3 | R400/64 | 0.3483 | 0.4539 | 0.5304 | 7.33% | 4.9700 | 0.1335 | 0.0355 | 0.1202 | 0.1845 | 70.85% | 82.07% |
| Qwen3 | R1000/0 | 0.3615 | 0.5063 | 0.5552 | 7.58% | 3.4347 | 0.1061 | 0.0382 | 0.1294 | 0.1971 | 75.42% | 82.88% |
| Qwen3 | R1000/100 | 0.3648 | 0.5161 | 0.5740 | 4.84% | 3.3719 | 0.1067 | 0.0386 | 0.1305 | 0.1983 | 76.02% | 85.10% |
| Gemma | P | 0.3445 | 0.4783 | 0.5034 | 13.33% | 4.7226 | 0.1206 | 0.0644 | 0.1675 | 0.2313 | 71.29% | 74.41% |
| Gemma | S1/0 | 0.4068 | 0.4670 | 0.4936 | 23.88% | 8.6883 | 0.2461 | 0.0716 | 0.1657 | 0.2277 | 60.00% | 64.49% |
| Gemma | S4/0 | 0.3528 | 0.4845 | 0.5439 | 6.60% | 4.5065 | 0.1160 | 0.0388 | 0.1353 | 0.2027 | 71.86% | 81.42% |
| Gemma | S4/1 | 0.3556 | 0.4970 | 0.6494 | 1.49% | 4.3987 | 0.1179 | 0.0395 | 0.1331 | 0.2004 | 72.76% | 90.69% |
| Gemma | T32/0 | 0.4473 | 0.5087 | 0.5410 | 18.07% | 8.2395 | 0.2732 | 0.0359 | 0.1183 | 0.1820 | 61.36% | 67.19% |
| Gemma | T128/0 | 0.3767 | 0.5240 | 0.5803 | 4.55% | 4.3104 | 0.1077 | 0.0426 | 0.1405 | 0.2094 | 75.00% | 83.58% |
| Gemma | T128/16 | 0.3799 | 0.5337 | 0.6299 | 1.17% | 4.2433 | 0.1092 | 0.0433 | 0.1411 | 0.2082 | 75.33% | 88.96% |
| Gemma | T256/0 | 0.3773 | 0.5691 | 0.6127 | 4.51% | 3.2290 | 0.0768 | 0.0451 | 0.1495 | 0.2202 | 80.38% | 86.30% |
| Gemma | T256/32 | 0.3837 | 0.5873 | 0.6676 | 0.67% | 3.1499 | 0.0786 | 0.0467 | 0.1498 | 0.2182 | 81.60% | 91.66% |
| Gemma | T256/64 | 0.3862 | 0.5989 | 0.7155 | 0.09% | 3.1156 | 0.0796 | 0.0477 | 0.1514 | 0.2204 | 81.53% | 95.24% |
| Gemma | T256/128 | 0.3927 | 0.6120 | 0.8028 | 0.00% | 3.1475 | 0.0779 | 0.0493 | 0.1539 | 0.2236 | 81.53% | 98.51% |
| Gemma | T256/192 | 0.3979 | 0.6255 | 0.8901 | 0.00% | 3.1102 | 0.0820 | 0.0508 | 0.1557 | 0.2259 | 81.65% | 99.63% |
| Gemma | R400/0 | 0.3602 | 0.4756 | 0.5175 | 12.59% | 5.0388 | 0.1141 | 0.0390 | 0.1430 | 0.2099 | 69.85% | 75.55% |
| Gemma | R400/64 | 0.3633 | 0.4897 | 0.5634 | 7.66% | 4.8275 | 0.1166 | 0.0386 | 0.1332 | 0.2010 | 71.17% | 80.78% |
| Gemma | R1000/0 | 0.3759 | 0.5398 | 0.5837 | 7.64% | 3.3759 | 0.0746 | 0.0423 | 0.1429 | 0.2114 | 75.66% | 82.04% |
| Gemma | R1000/100 | 0.3794 | 0.5505 | 0.6033 | 5.41% | 3.3065 | 0.0752 | 0.0429 | 0.1427 | 0.2096 | 76.20% | 83.80% |
To be published.