debugging.md

Debugging

nikos doctor

Run this first for any issue:

nikos doctor

It checks:

• ansible, dialog, gh, tmux, Ollama, VS Code, distrodeck, conda, Claude Code, Gemini CLI
• llama.cpp, shell-gpt, glances, and mkcert when present
• Nordic GTK theme files
• Papirus icon files
• Git setup wizard completion flag

Any [!!] items indicate missing or broken components. Run nikos update to attempt repair.

Reading the install summary

The installer prints a summary when it finishes, and writes the same numbers to the log:

[SUMMARY] rc=0 ok=189 changed=15 failed=0 unreachable=0
[DONE] Install complete

An install runs the playbook twice: once for the main run, and once more for any optional bundles that were selected. Each run prints its own PLAY RECAP, and the summary totals them, so ok= will not match any single recap in the log. To see the runs separately:

grep -A2 '^PLAY RECAP' ~/.config/nikos/logs/install-latest.log | grep localhost

failed= counts both runs. Before 0.6.0 the summary read only one number per field, which made the count a multi-line value: the box printed the remainder on its own line, and the comparison that decides whether to report a failure raised a shell error and evaluated false. A bundle could fail while the installer reported a clean install. If you are on an older version and a bundle looks like it did not install, check the log directly:

grep -c '^fatal:' ~/.config/nikos/logs/install-latest.log
awk '/^TASK \[/{t=$0} /^fatal: \[/{print t}' ~/.config/nikos/logs/install-latest.log

The digest at the end of every log

Since 0.8.0 every install log and nikos update playbook log ends with a digest, so the end of the file says what went wrong:

=== NikOS log digest ===
NikOS ref: 1.0.0   OS: Ubuntu 24.04.5 LTS   Kernel: ...
Ansible: ansible-playbook [core 2.16.3]   Python: 3.12.3
Free disk: / 43G, home 43G
Failed tasks (1):
  - [dev-tools : Clone ai-runner] Local modifications exist in the destination ...
Warnings (1 distinct):
  - (1x) [WARNING]: ...
Errors (1 distinct):
  - (1x) [ERROR]: ...
=== end of digest ===

A failed task also prints its file and line (task path: roles/.../main.yml:42).

Where downloads and keys live

Nothing is downloaded to a fixed /tmp path. Root steps use /var/lib/nikos/downloads, user steps ~/.cache/nikos/downloads (both 0700), and apt signing keys are fetched and checked in /var/lib/nikos/keys. Cached archives there can be deleted; the next run downloads and re-checks them.

A download or key check fails

Every download is checked against the sha256 pinned in vars/versions.yml, and every apt key against its pinned fingerprint. A mismatch stops the run: an upstream release was replaced or a key changed, and that needs a person, not a retry. scripts/bump-versions.py --check shows what upstream has now and --verify re-checks every pin (see docs/development.md).

Common issues

Xfce doesn't start after install

LightDM may not be set as the default display manager.

sudo systemctl status lightdm
sudo systemctl enable --now lightdm
cat /etc/X11/default-display-manager   # should contain /usr/sbin/lightdm

Nordic theme not applied

ls /usr/share/themes/Nordic          # should exist
ls /usr/share/icons/Papirus-Dark     # should exist

If missing, re-run the playbook through the CLI:

nikos setup

The playbook is idempotent, and nikos setup is what to reach for rather than ansible-playbook directly, for three reasons. It collects the sudo password site.yml needs for become: true; a bare ansible-playbook site.yml asks for nothing and fails on the first privileged task unless sudo is already passwordless. It re-applies the --skip-tags saved in ~/.config/nikos/selected-options.env, which ansible-playbook knows nothing about, so a hand-run playbook, even with --ask-become-pass, installs every bundle that was declined at install time. And it sets nikos_update_mode explicitly: the variable defaults to true in vars/main.yml so that the 0.6.4 CLI, which runs the newly checked-out playbook without passing it, still refreshes dependencies on its first 0.6.5 update. A hand-run playbook inherits that default and performs update-only work — including a full system package upgrade — unless it passes -e nikos_update_mode=false, which every direct invocation below therefore does.

Not --tags theming. The theming role carries no tag, in site.yml or on its tasks, and Ansible does not give a role an implicit tag named after it, so --tags theming selects the always tasks and nothing else:

$ ansible-playbook site.yml -i inventory/local --tags theming --list-tasks -e nikos_update_mode=false
      Check whether local override vars exist	TAGS: [always]
      Load local override vars	TAGS: [always]
      ...

It exits cleanly having done nothing, which reads like theming ran and failed to fix the problem.

Ollama model not found

ollama list
ollama pull qwen3.5:4b

Check the service:

systemctl --user status ollama
systemctl --user restart ollama

Git setup wizard re-run

nikos-git-setup --reset

VS Code extensions not installed

code --list-extensions
code --install-extension Continue.continue --force

conda / nikos-ai or nikos-aider env missing or broken

nikos update creates a missing env and installs its packages at the pins in vars/versions.yml. To rebuild one, remove it and update:

~/miniforge3/bin/conda env list
~/miniforge3/bin/conda env remove -n nikos-aider -y   # or nikos-ai
nikos update

aider is a launcher in ~/.local/bin that runs from the nikos-aider env.

image-view not found

image-view installs to ~/.local/bin/. Ensure that's in your PATH:

echo $PATH | grep -q ".local/bin" || echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc
image-view --help

If the playbook reports that Cargo is too old for image-view, update Rust via distrodeck and rerun nikos update. The package dependencies require Cargo 1.85+ for Rust 2024 edition support.

vellum not found

vellum installs to ~/.local/bin/ like image-view (see above for the PATH line). It needs Rust 1.90 or newer; when the playbook reports Cargo is too old, run rustup update stable and then nikos update.

NikOS boot splash logo missing

The Plymouth logo is embedded into initramfs. If the install was interrupted or a later task failed before handlers ran, rerun:

nikos update
sudo plymouth-set-default-theme nikos
sudo update-initramfs -u -k all

Then reboot and check the early boot splash again.

Still booting into GNOME after installing on Ubuntu

The switch only takes effect after a reboot. If a reboot did not help, check which unit owns the display manager alias:

readlink -f /etc/systemd/system/display-manager.service

It must resolve to lightdm.service. If it still points at gdm3.service:

sudo systemctl disable gdm3
sudo ln -sf /usr/lib/systemd/system/lightdm.service \
  /etc/systemd/system/display-manager.service
sudo systemctl daemon-reload

If the greeter is LightDM but the desktop is still GNOME, the account has a session recorded from before the migration:

grep -i session /var/lib/AccountsService/users/"$USER"
ls /usr/share/xsessions

Both Session and XSession should name an entry from /usr/share/xsessions, normally xubuntu. /etc/lightdm/lightdm.conf.d/60-nikos.conf holds the system-wide default. You can also pick the session from the gear menu on the login screen.

git-lantern / lantern not found

lantern is installed per-user, not system-wide: a root-owned /usr/local/bin/lantern pointing into the user's writable home would let that user hand root-controlled code to anyone else who ran the command.

which lantern                      # expect ~/.local/bin/lantern
ls ~/.local/bin/lantern
# Reinstall if missing (no sudo - everything here is user-owned):
~/.local/share/nikos-tools/git-lantern/install --prefix ~/.local/opt/git-lantern --bin-link ~/.local/bin/lantern

If which lantern finds nothing but the file exists, ~/.local/bin is not on PATH for that shell.

bitnet-cli not found, or fails to load a library

which bitnet-cli                          # expect /usr/local/bin/bitnet-cli
ls /usr/local/lib/nikos/bitnet/llama-cli  # the binary it runs

bitnet-cli is a small wrapper. The binary and the shared libraries it loads are installed together under /usr/local/lib/nikos/bitnet, so the command does not depend on the BitNet build tree: ~/.local/share/nikos-tools/bitnet.cpp is only needed to rebuild, and can be deleted to reclaim disk.

Two things to check if it misbehaves:

# A copy from an older NikOS shadowing the real one. ~/.local/bin comes first
# on a default PATH, and that copy read its libraries out of the build tree.
ls ~/.local/bin/bitnet-cli && rm ~/.local/bin/bitnet-cli

# Nothing installed at all: the bundle is optional and may never have run.
nikos add bitnet

Building the CLI needs LLAMA_BUILD_TOOLS. llama.cpp turns it off by default when it is built as a subdirectory, which BitNet does, so a build configured by an older NikOS produced only the shared libraries and no llama-cli. nikos add bitnet re-configures an existing build tree rather than skipping it.

Installer or nikos update cannot move the checkout

You are not currently on a branch.
Please specify which branch you want to merge with.
Failed to pull updates in /home/you/.local/share/nikos

Installing a release tag leaves ~/.local/share/nikos on a detached HEAD, which is normal. Versions before 0.5.0 then tried to git pull it on the next run; a detached HEAD has no upstream to merge, so the installer exited before it reached the code that would have switched refs.

0.5.0 fetches first and only fast-forwards when HEAD is on a branch with an upstream, so this resolves itself. On an installer older than that, move the checkout onto a branch by hand:

git -C ~/.local/share/nikos switch --track origin/main

Checking which version is installed

cat ~/.local/share/nikos/VERSION
git -C ~/.local/share/nikos describe --tags --exact-match   # release installs
git -C ~/.local/share/nikos branch --show-current           # branch installs

An empty branch --show-current with a tag from describe is a release install and is expected.

Disk full: nikos clean

A full disk fails quietly: Elasticsearch, for one, blocks writes above 95% and indexing just stops. nikos clean lists what nothing uses, with sizes, and removes it only with --apply (it asks first; --yes for no terminal):

What	Kept
Docker build cache not used in --keep-days (7), of every builder (docker buildx ls)	cache used this week
Docker images with no tag; with --unused-images, tagged ones no container uses	images any container, running or stopped, uses
node_modules in git repositories under --projects (~/Projects) with no commit for --idle-days (30) and nothing uncommitted, beside a package.json	repositories with recent commits or uncommitted changes, one a process of yours runs in or a running container bind-mounts, tracked or nested ones
the whole uv cache (downloads and unpacked packages; environments keep their files); pnpm and npm entries no project uses; pip files not used in --keep-days	pnpm and npm references; npm and pnpm are skipped while an npm, pnpm or yarn process runs

Never removed: Docker volumes (databases, indexes and models, also of projects not being worked on), containers, virtual environments, build output, tracked files. --only docker,node,packages limits it. Which repositories processes run in is read from /proc, or from lsof where there is no /proc (macOS); with neither, node_modules is left alone.

nikos clean                 # what would go, and how much
nikos clean --apply         # remove it, after a yes
nikos clean --apply --keep-days 0 --unused-images   # all build cache, and images no container uses

--keep-days 0 removes the whole build cache (the next build of each image is slower). A uv cache a uv process holds is skipped after 10 seconds, naming the processes: a uvx tool (an MCP server, for one) holds it for as long as it runs. --uv-force clears it anyway; restart those tools afterwards. Each step reports what it freed; other programs write at the same time, so the figures are approximate.

Ansible logs

Run the playbook directly with verbose output:

ansible-playbook ~/nikos/site.yml -i ~/nikos/inventory/local -v -e nikos_update_mode=false

Use -vvv for full debug output including module arguments.

Check mode (dry-run)

Preview what would change without applying:

ansible-playbook ~/nikos/site.yml -i ~/nikos/inventory/local --check -e nikos_update_mode=false

Re-running a single role

ansible-playbook ~/nikos/site.yml -i ~/nikos/inventory/local --tags theming -e nikos_update_mode=false
ansible-playbook ~/nikos/site.yml -i ~/nikos/inventory/local --tags ai-stack -e nikos_update_mode=false

Note: role tags must be explicitly set in site.yml. The optional roles (network, music, education, neovim, java, podman) are tag-gated by default.

Security hooks: nikos hooks

nikos hooks drives the script-helpers hooks through git-lantern (git hooks per repository) and agentvault (AI tool rules and hooks). Only clones whose origin is under git config --global script-helpers.owners are changed; hooks a clone already runs (an employer's included) keep running, and remove brings back what was there. nikos update installs gitleaks and foxguard so the hooks can run the credential and advisory static-analysis scans instead of reporting a missing tool.

git config --global script-helpers.owners <user-or-org>   # once per machine
nikos hooks                                    # tools, owners, every clone
nikos hooks apply --dry-run                    # the plan
nikos hooks apply --batch 10 [--agents all]    # ten clones at a time
nikos hooks repair                             # after husky took over, a missing hooks dir, or a bad Claude hook
nikos hooks remove [--agents codex]            # take ours out
nikos hooks --global [--uninstall]             # machine-wide; install/update never run it
nikos hooks export ~/carry                     # lantern-hooks.json + agentvault.avbundle
nikos hooks import ~/carry                     # on the other computer

--root DIR (or NIKOS_HOOKS_ROOT) says where the clones are; the default is ~/Projects. script-helpers comes from script-helpers.home, else the copy NikOS pins. An unknown option exits 2 and changes nothing. import sets the owners from the export only when this machine has none.

The Claude Code hook is per user, not per clone. nikos install and nikos update install it with setup-hooks.sh --claude, which copies it to ~/.local/share/script-helpers/claude-hooks; the settings entry runs that copy, never a checkout. nikos hooks shows its state (ok, NOT INSTALLED, BROKEN, STALE), and repair or apply (without --only) reinstalls it. Point script-helpers.home at a copy that stays on a release, not a checkout you switch branches in: status and repair run its code.