Updating the Icinga agent causes the Icinga PowerShell service to terminate quickly

[smarsching]

We are currently experiencing a problem on all of our Windows systems, where the Icinga PowerShell service terminates shortly after being started. The Windows event log only contains the following message:

The Icinga PowerShell Service service terminated unexpectedly.

Initially, I thought it was related to updating the Icinga PowerShell Framework from version 1.14.2 to 1.15.0 and the Icinga PowerShell plugins from version 1.14.1 to 1.15.0, but after rebooting systems, we now have this problem on all Windows systems (a mix of Windows Server 2002 and Windows 11), even those where we never installed the update.

Due to all systems being affected, a common cause seems very likely, but I don’t know what this cause might be or how I could get additional information. I tried running Start-IcingaWindowsRESTApi from an interactive PowerShell, but doing so didn’t yield anything useful (the cmdlet does not print anything and quits after a few seconds).

Is there any way how I can turn on debug logging in order to find out more?

[LordHepipud]

Hello,

Thank you for the issue.
You can enable the debug mode with Enable-IcingaFrameworkDebugMode.

Can you please share the following on an affected system:

Show-Icinga;
Get-ExecutionPolicy -List;
Test-IcingaForWindows;

[smarsching]

Thanks for reacting so quickly. I didn’t expect this, in particular not on a Sunday. 🙂

Here is the output of the commands:

PS C:\Windows\system32> Show-Icinga
Icinga for Windows environment:
-----------


Environment configuration:

PowerShell Root                 => C:\Program Files\WindowsPowerShell\Modules\
Icinga for Windows Service Path => C:\Program Files\icinga-framework-service\
Icinga for Windows Service User => .\icinga
Icinga for Windows Service Pid  => 3424
Icinga for Windows JEA Pid      => 5656
Icinga Agent Path               => C:\Program Files\ICINGA2\
Icinga Agent User               => NT AUTHORITY\NetworkService
Defined Default User            => WINTER\icinga
Icinga Managed User             => True
PowerShell Version              => 5.1.20348.4294
Operating System                => Microsoft Windows Server 2022 Standard
Operating System Version        => 10.0.20348
JEA Context                     => IcingaForWindows
JEA Session File                => C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\RoleCapabilities\IcingaForWindows.psrc
Api Check Forwarder             => True
Debug Mode                      => False

Icinga for Windows Certificate:

Issuer  => CN=Icinga CA
Subject => CN=winter.office-bad.ad.aquenos.com

List of configured background daemons on this system:

Start-IcingaWindowsRESTApi
-----------
No arguments defined

List of configured background service checks on this system:
=> https://icinga.com/docs/icinga-for-windows/latest/doc/110-Installation/06-Collect-Metrics-over-Time/

No background service checks configured

List of configured repositories on this system. The list order matches the apply order:

Icinga Stable
-----------
CloneSource  =>
Enabled      => True
LocalPath    =>
Order        => 0
RemotePath   => https://packages.icinga.com/IcingaForWindows/stable/ifw.repo.json
UseSCP       => False

Installed components on this system:

Component    Version   Available
---          ---       ---
agent        2.16.2    2.16.3
framework    1.14.2    1.15.0
plugins      1.14.1    1.15.0
service      1.3.0     1.3.0

Available versions flagged with "*" mean that this component is locked to this version

PS C:\Windows\system32> Get-ExecutionPolicy -List

        Scope ExecutionPolicy
        ----- ---------------
MachinePolicy       Undefined
   UserPolicy       Undefined
      Process       Undefined
  CurrentUser       Undefined
 LocalMachine    RemoteSigned

PS C:\Windows\system32> Test-IcingaForWindows
[Notice]: Collecting Icinga for Windows environment information
[Passed]: The Icinga Agent service and the Icinga Agent are installed on the system
[Passed]: The Icinga for Windows service is installed on the system
[Passed]: The Icinga for Windows service binary does exist: "C:\Program Files\icinga-framework-service\icinga-service.exe"
[Passed]: Your service installation is not affected by IWKB000009
[Passed]: Your service installation is properly referring to "icinga-powershell-framework.psd1" for module imports.
[Warning]: The Icinga Agent service user "NT AUTHORITY\NetworkService" is not matching the Icinga for Windows service user ".\icinga"
[Passed]: It seems the provided SID "S-1-5-20" is a system SID. Skipping permission check
[Passed]: Directory "C:\ProgramData\icinga2\etc" is accessible and writable by the Icinga Service User "NT AUTHORITY\NetworkService"
[Passed]: Directory "C:\ProgramData\icinga2\var" is accessible and writable by the Icinga Service User "NT AUTHORITY\NetworkService"
[Passed]: Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\cache" is accessible and writable by the Icinga Service User "NT AUTHORITY\NetworkService"
[Passed]: Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\config" is accessible and writable by the Icinga Service User "NT AUTHORITY\NetworkService"
[Passed]: Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\certificate" is accessible and writable by the Icinga Service User "NT AUTHORITY\NetworkService"
[Passed]: The Icinga Agent state file is healthy
[Passed]: Icinga Agent configuration is valid
[Passed]: Icinga Agent debug log is disabled
[Passed]: The Icinga for Windows REST-Api is configured to start with the daemon
[Passed]: The Icinga for Windows REST-Api is configured to allow API checks
[Passed]: The Icinga for Windows certificate is installed on the system and possibly signed by a valid CA
[Passed]: Icinga for Windows is configured with a JEA-Profile
[Failed]: The Icinga for Windows service is currently not running
[Failed]: The Icinga for Windows REST-Api responded with an error on this machine: "Unable to connect to the remote server"

Indeed, the icinga2 service was configured to use the Network Service user, while the icingapowershell service was configured to use the .\icinga user. I guess this happened when the Icinga agent was updated a few days ago, but it didn’t have any effect until the PowerShell service was restarted (due to updating the PowerShell modules or rebooting the whole operating system).

After changing the icinga2 service to use the .\icinga user, the icinga2 service wouldn’t start either, and I got the following output:

PS C:\Windows\system32> Test-IcingaForWindows
[Notice]: Collecting Icinga for Windows environment information
[Passed]: The Icinga Agent service and the Icinga Agent are installed on the system
[Passed]: The Icinga for Windows service is installed on the system
[Passed]: The Icinga for Windows service binary does exist: "C:\Program Files\icinga-framework-service\icinga-service.exe"
[Passed]: Your service installation is not affected by IWKB000009
[Passed]: Your service installation is properly referring to "icinga-powershell-framework.psd1" for module imports.
[Passed]: The Icinga Agent service user ".\icinga" is matching the Icinga for Windows service user ".\icinga"
[Passed]: The specified user "icinga" is allowed to run as service
[Failed]: Directory "C:\ProgramData\icinga2\etc" is not accessible by the Icinga Service User "icinga"
\_ Please run the following command to fix this issue: Set-IcingaAcl -Directory 'C:\ProgramData\icinga2\etc'
[Failed]: Directory "C:\ProgramData\icinga2\var" is not accessible by the Icinga Service User "icinga"
\_ Please run the following command to fix this issue: Set-IcingaAcl -Directory 'C:\ProgramData\icinga2\var'
[Failed]: Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\cache" is not accessible by the Icinga Service User "icinga"
\_ Please run the following command to fix this issue: Set-IcingaAcl -Directory 'C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\cache'
[Failed]: Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\config" is not accessible by the Icinga Service User "icinga"
\_ Please run the following command to fix this issue: Set-IcingaAcl -Directory 'C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\config'
[Failed]: Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\certificate" is not accessible by the Icinga Service User "icinga"
\_ Please run the following command to fix this issue: Set-IcingaAcl -Directory 'C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\certificate'
[Passed]: The Icinga Agent state file is healthy
[Passed]: Icinga Agent configuration is valid
[Passed]: Icinga Agent debug log is disabled
[Passed]: The Icinga for Windows REST-Api is configured to start with the daemon
[Passed]: The Icinga for Windows REST-Api is configured to allow API checks
[Passed]: The Icinga for Windows certificate is installed on the system and possibly signed by a valid CA
[Passed]: Icinga for Windows is configured with a JEA-Profile
[Failed]: The Icinga for Windows service is currently not running
[Failed]: The Icinga for Windows REST-Api responded with an error on this machine: "Unable to connect to the remote server"

I ran the the Set-IcingaAcl commands as instructed, and this did indeed resolve the issues. Now, both services are running again as the icinga user.

So, I now wonder whether there is way of keeping the installer for the Icinga agent from messing up the service configuration and ACLs.

BTW, I updated the issue title to reflect the fact that the Icinga PowerShell service termating was only a symptom and not the underlying cause.

[LordHepipud]

Thank you very much - how did you update the Agent? The best approach is to use the Icinga for Windows Repository and use Update-Icinga for applying updates. This will update all required components and Icinga for Windows will handle everything.

You can also update specific components only, like

Update-Icinga 'agent' -Confirm;

When you install the Agent standalone, the permissions of the files in "ProgramData" are ensured to match the Icinga Agents users, in this case the NetworkService. In general that shouldn't cause any problems, because Icinga for Windows is barely accessing this directory. But it could be a cause. I will have a look on that.

[smarsching]

We update the Icinga agent through our regular software management system, which uninstalls the MSI and then installs the new version. It’s possible that this process removes the icinga2 service and readds it using the Network Service user, also changing the ACLs of some of the directories in the process.

We could change to using the PowerShell for installing and updating the Icinga agent through our configuration management system (Salt), the problem with the Update-Icinga cmdlet is that as far as I know, it does not allow explicitly specifying component versions, and there is no call for checking whether any changes would be made either. We need both to ensure that we fully control which version is installed (even if a newer version is available) and to be able to see whether a configuration update would make changes to a system.

At the moment, we call Get-Module in order to determine whether a certain component has to be updated, and then call either Install-IcingaFrameworkUpdate -FrameworkUrl … for updating the Icinga PowerShell framework or Install-IcingaComponent -Name … -Version … for the Icinga PowerShell plugins and service.

I wonder whether this approach might also work for the Icinga agent (though we would have to use something different than Get-Module to determine the currently installed version). Do you think using Install-IcingaComponent for updating the agent would preserve the service configuration and ACLs?

At the moment, we also see the problem that updating the framework and plugins seems to mess up the JEA profile, but I have to investigate this more closely and will probably open a separate issue, because I think that this might be a separate problem.

[smarsching]

After having a closer look, I was able to verify that the problem with the JEA profile was actually a problem in our configuration logic: After updating components, Install-IcingaJEAProfile / Update-IcingaJEAProfile should have been called, but due to a flaw in the logic it wasn’t. After fixing this, the update works as expected.

On a related note, I realized that calling Install-IcingaSecurity after updating the agent would probably fix the issues that we experienced. The only problem with that is that it also resets the passwor of the service user, which is a problem on Active Directory domain controllers, because there is no notion of “local” users there, so changing the password on one domain controller affects all domain controllers, which means that the password needs to be managed manually.

Maybe it could help to add an optional parameter to Install-IcingaSecurity, so that the password can be supplied instead of using a randomly generated password. I think that unlike Linux, Windows does not allow regular users to see the command line arguments of any process, so this should be safe.