Ubuntu 24 chicken-egg problem: sshd config can not be validated due to missing /run/sshd

[timdeluxe]

We are looking for Ubuntu 24 installations and experience a problem:
On a fresh install, the openssh-server is not started yet. Therefore the "/run/sshd" directory is missing, which is required for the validation of the config:

err | Puppet | Execution of '/usr/sbin/sshd -tf /etc/ssh/sshd_config20241106-2171-1prqvcn' returned 255: Missing privilege separation directory: /run/sshd
err | /Stage[main]/Ssh::Server::Config/Concat[/etc/ssh/sshd_config]/File[/etc/ssh/sshd_config]/content | change from '{sha256}9ce65293b15513398541fad559786ef310a9045bdda398005477fda6ebf3c0f0' to '{sha256}8e740c5c770cf9c254e52e80004d078f329b8411fc4aff04ad723a5d92638c31' failed: Execution of '/usr/sbin/sshd -tf /etc/ssh/sshd_config20241106-2171-1prqvcn' returned 255: Missing privilege separation directory: /run/sshd
warning | /Stage[main]/Ssh::Server::Service/Service[ssh] | Skipping because of failed dependencies

(Output taken from Foreman, which explains why it looks a bit different as usual)

The service comes later in the dependency try of this module, thats why it never will succeed without manual interaction.

I have no good idea how to fix this, an idea could be to add a mkdir /run/sshd && at the beginning of the validate-command?

[saz]

We might just manage the /run/sshd dir as a file resource, on some operating systems. Might be the cleanest fix for this issue.

[bwitt]

maybe i can add this to my PR at #418 ? just manage /run/sshd?

[saz]

@timdeluxe Can you tell me how to reproduce this issue? All my Ubuntu 24.04 are working without any issues.

[timdeluxe]

Hi @saz
we implemented a workaround with this puppet snippet:

  # workaround for issue describe in https://github.com/saz/puppet-ssh/issues/398
  if ($facts['os']['name'] == 'Ubuntu' and versioncmp($facts['os']['release']['full'], '24.04') == 0) {
    file { '/run/sshd':
      ensure => 'directory',
    }
    File['/run/sshd'] -> Class['ssh::server']
  }

and had no further issues since then.

As said, it has to do with fresh minimal installations. We perform our installs with Foreman, which executes puppet at pretty early stages.

If you don't want to spend time into this, i am fine with closing the issue, as the workaround works fine for us.

[saz]

@timdeluxe I'm happy to spend time on it :-)

I'm just trying to understand, how to reproduce this issue, because chances are high, that this is affecting all Debian based distros, not only Ubuntu.

As you mention minimal installations: there's no SSH running, when puppet is running the first time, right?

[timdeluxe]

Exactly. Thats how i assume the problem arises. Basically Foreman performs a typical "network installation" of the distros. With Ubuntu that is nowadays the "cloudinit" based way.
I think it should be reproducable with a minimal install or by deselecting openssh(-server) on install and then using your module via a puppet install (requires direct terminal access to the install of course). Due to lot of project-work i miss the time to give you more details (f.e. helpful logs) from our Foreman installs, i am sorry.