Merge in v4.1.4

.github/workflows/tests.yaml

@@ -196,7 +196,7 @@ jobs:
 
     - uses: jakebailey/pyright-action@v2
       with:
-        version: 1.1.408
+        version: 1.1.411
 
   ydiff:
     name: Test compatibility with the latest version of ydiff

docs/patroni_configuration.rst

@@ -45,7 +45,7 @@ Some of the PostgreSQL parameters **must hold the same values on the primary and
 - **max_worker_processes**: default value 8, minimal value 2
 - **max_prepared_transactions**: default value 0, minimal value 0
 - **wal_level**: default value hot_standby, accepted values: hot_standby, replica, logical
-- **track_commit_timestamp**: default value off 
+- **track_commit_timestamp**: default value off
 
 For the parameters below, PostgreSQL does not require equal values among the primary and all the replicas. However, considering the possibility of a replica to become the primary at any time, it doesn't really make sense to set them differently; therefore, **Patroni restricts setting their values to the** :ref:`dynamic configuration <dynamic_configuration>`.
 
@@ -136,7 +136,7 @@ Also the following Patroni configuration options **can be changed only dynamical
 
 - **ttl**: 30
 - **loop_wait**: 10
-- **retry_timeouts**: 10
+- **retry_timeout**: 10
 - **maximum_lag_on_failover**: 1048576
 - **max_timelines_history**: 0
 - **check_timeline**: false
@@ -198,7 +198,7 @@ Patroni configuration for a running instance
 Description
 """""""""""
 
-Generate a Patroni configuration in ``yaml`` format for the locally running PostgreSQL instance. 
+Generate a Patroni configuration in ``yaml`` format for the locally running PostgreSQL instance.
 Either the provided DSN (takes precedence) or PostgreSQL `environment variables <https://www.postgresql.org/docs/current/libpq-envars.html>`__ will be used for the PostgreSQL connection. If the password is not provided, it should be entered via prompt.
 
 All the non-internal GUCs defined in the source Postgres instance, independently if they were set through a configuration file, through the postmaster command-line, or through environment variables, will be used as the source for the following Patroni configuration parameters:

docs/patronictl.rst

@@ -64,7 +64,7 @@ This is the synopsis for running a command from the ``patronictl``:
 .. code:: text
 
     patronictl [ { -c | --config-file } CONFIG_FILE ]
-      [ { -d | --dcs-url | --dcs } DCS_URL ] 
+      [ { -d | --dcs-url | --dcs } DCS_URL ]
       [ { -k | --insecure } ]
       SUBCOMMAND
 
@@ -82,6 +82,75 @@ This is the synopsis for running a command from the ``patronictl``:
 
 In the following sub-sections you can find a description of each command implemented by ``patronictl``. For sake of example, we will use the configuration files present in the GitHub repository of Patroni (files ``postgres0.yml``, ``postgres1.yml`` and ``postgres2.yml``).
 
+.. _patronictl_demote_cluster:
+
+patronictl demote-cluster
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. _patronictl_demote_cluster_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    demote-cluster
+      [ CLUSTER_NAME ]
+      [ --host HOST ]
+      [ --port PORT ]
+      [ --restore-command RESTORE_COMMAND ]
+      [ --primary-slot-name PRIMARY_SLOT_NAME ]
+      [ --force ]
+
+.. _patronictl_demote_cluster_description:
+
+Description
+"""""""""""
+
+``patronictl demote-cluster`` converts a regular Patroni cluster into a :ref:`standby cluster <standby_cluster>`.
+
+The command patches the dynamic configuration with a ``standby_cluster`` section built from the provided remote primary connection options, then waits until the leader is running as a standby leader. It prints the current cluster topology before changing the configuration and asks for confirmation unless ``--force`` is used.
+
+At least one of ``--host``, ``--port`` or ``--restore-command`` must be specified.
+
+.. _patronictl_demote_cluster_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+    If not given, ``patronictl`` will attempt to fetch that from the ``scope`` configuration, if it exists.
+
+``--host``
+    Address of the remote node.
+
+``--port``
+    Port of the remote node.
+
+``--restore-command``
+    Command to restore WAL records from the remote primary.
+
+``--primary-slot-name``
+    Name of the replication slot on the remote node to use for replication.
+
+``--force``
+    Flag to skip confirmation prompts when demoting the cluster.
+
+    Useful for scripts.
+
+.. _patronictl_demote_cluster_examples:
+
+Examples
+""""""""
+
+Demote the cluster to a standby cluster that follows a remote primary endpoint:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml demote-cluster batman --host 192.0.2.10 --port 5432 --primary-slot-name batman --force
+
 .. _patronictl_dsn:
 
 patronictl dsn
@@ -202,7 +271,7 @@ Parameters
 
 ``--group``
     Change dynamic configuration of the given Citus group.
-    
+
     If not given, ``patronictl`` will attempt to fetch that from the ``citus.group`` configuration, if it exists.
 
     ``CITUS_GROUP`` is the ID of the Citus group.
@@ -566,7 +635,7 @@ Parameters
     Show history of events from the given Citus group.
 
     ``CITUS_GROUP`` is the ID of the Citus group.
-    
+
     If not given, ``patronictl`` will attempt to fetch that from the ``citus.group`` configuration, if it exists.
 
 ``-f`` / ``--format``
@@ -921,7 +990,7 @@ Parameters
     Pause the given Citus group.
 
     ``CITUS_GROUP`` is the ID of the Citus group.
-    
+
     If not given, ``patronictl`` will attempt to fetch that from the ``citus.group`` configuration, if it exists.
 
 ``--wait``
@@ -940,6 +1009,57 @@ Put the cluster in maintenance mode, and wait until all nodes have been paused:
     'pause' request sent, waiting until it is recognized by all nodes
     Success: cluster management is paused
 
+.. _patronictl_promote_cluster:
+
+patronictl promote-cluster
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. _patronictl_promote_cluster_synopsis:
+
+Synopsis
+""""""""
+
+.. code:: text
+
+    promote-cluster
+      [ CLUSTER_NAME ]
+      [ --force ]
+
+.. _patronictl_promote_cluster_description:
+
+Description
+"""""""""""
+
+``patronictl promote-cluster`` converts a standby cluster into a regular Patroni cluster.
+
+The command removes the ``standby_cluster`` section from the dynamic configuration and waits until the leader is running as the primary. It prints the current cluster topology before changing the configuration and asks for confirmation unless ``--force`` is used.
+
+.. _patronictl_promote_cluster_parameters:
+
+Parameters
+""""""""""
+
+``CLUSTER_NAME``
+    Name of the Patroni cluster.
+
+    If not given, ``patronictl`` will attempt to fetch that from the ``scope`` configuration, if it exists.
+
+``--force``
+    Flag to skip confirmation prompts when promoting the cluster.
+
+    Useful for scripts.
+
+.. _patronictl_promote_cluster_examples:
+
+Examples
+""""""""
+
+Promote the standby cluster to run as a regular Patroni cluster:
+
+.. code:: bash
+
+    $ patronictl -c postgres0.yml promote-cluster batman --force
+
 .. _patronictl_query:
 
 patronictl query
@@ -1452,7 +1572,7 @@ Parameters
 ``--pending``
     Select only members which are flagged as ``Pending restart``.
 
-``timeout``
+``--timeout``
     Abort the restart if it takes more than the specified timeout, and fail over to a replica if the issue is on the primary.
 
     ``TIMEOUT`` is the amount of seconds to wait before aborting the restart.
@@ -1556,7 +1676,7 @@ Parameters
     Resume the given Citus group.
 
     ``CITUS_GROUP`` is the ID of the Citus group.
-    
+
     If not given, ``patronictl`` will attempt to fetch that from the ``citus.group`` configuration, if it exists.
 
 ``--wait``
@@ -1612,7 +1732,7 @@ Parameters
     Show dynamic configuration of the given Citus group.
 
     ``CITUS_GROUP`` is the ID of the Citus group.
-    
+
     If not given, ``patronictl`` will attempt to fetch that from the ``citus.group`` configuration, if it exists.
 
 .. _patronictl_show_config_examples:
@@ -1653,6 +1773,7 @@ Synopsis
       [ --group CITUS_GROUP ]
       [ { --leader | --primary } LEADER_NAME ]
       --candidate CANDIDATE_NAME
+      [ --scheduled TIMESTAMP ]
       [ --force ]
 
 .. _patronictl_switchover_description:

docs/releases.rst

@@ -3,6 +3,86 @@
 Release notes
 =============
 
+Version 4.1.4
+-------------
+
+Released 2026-07-07
+
+**Bugfixes**
+
+- Check ``NOTIFY_SOCKET`` environment variable before using ``systemd`` package (Polina Bungina)
+
+  Only try to import and use the package when the ``NOTIFY_SOCKET`` environment variable is set to avoid ``FileNotFoundError: [Errno 2] No such file or directory`` exception.
+
+- Unify ``pg_replication_slots`` query (Polina Bungina)
+
+  Incorrect handling of the ``failover`` and ``synced`` values was resulting in ``KeyError`` exceptions during removal of the incorrect logical replication slots.
+
+- Consider version-specific authentication parameters in configuration generation (Polina Bungina)
+
+  In the ``patroni --generate-config`` command, remove all inapplicable authentication parameters that were accidentally picked up from the environment, based on the version retrieved from a PostgreSQL connection.
+
+- Handle ``pg_rewind`` while a PostgreSQL instance is starting as a standby (Alexander Kukushkin)
+
+  Fallback to ``pg_controldata`` information when a PostgreSQL instance is running but is not yet accepting connections.
+
+- Fix Prometheus metric type for ``patroni_postgres_timeline`` (Huseyin Demir)
+
+  Declare the ``patroni_postgres_timeline`` metric as ``gauge`` instead of ``counter``, as it is not always monotonically increasing (e.g., it can be reset to 0 if a PostgreSQL instance is not running).
+
+- Don't stop watchdog until client backends are fully stopped (Alexander Kukushkin)
+
+  Previously, if ``primary_stop_timeout`` was shorter than the minimum watchdog timeout, and the stop timeout actually expired, Patroni disabled the watchdog before all client backends had exited.
+
+- Handle statement timeout error for monitoring query (Alexander Kukushkin)
+
+  In case of a statement timeout error, use the cached role as a fallback to avoid demoting the primary. Additionally, forcibly set ``pg_stat_statements.track`` to ``none`` for the monitroing query to avoid expensive ``pg_stat_statements`` GC calls.
+
+- Drop Patroni-managed replication slots with ``wal_status=lost`` (Alexander Kukushkin)
+
+  Replication slots with ``wal_status=lost`` are no longer usable. Patroni will now drop such slots and recreate them if needed.
+
+- Fix role representation in patronictl member validation error (Polina Bungina)
+
+  Ensures the correct string representation is used within the exception message, preventing errors from being formatted like ``Error: No CtlPostgresqlRole.REPLICA among provided members``.
+
+
+Version 4.1.3
+-------------
+
+Released 2026-05-05
+
+**Stability improvements**
+
+- Properly handle mislabeled Etcd error (Ants Aasma)
+
+  Current Etcd versions raise ``Unknown`` error when Etcd leader is lost while updating the lease. Patroni will now override the reported error code to ``Unavailable``.
+
+**Bugfixes**
+
+- Use binary version when ``PG_VERSION`` file does not exist (Polina Bungina)
+
+  In some cases, for example when using custom bootstrap, the ``PG_VERSION`` file may not be present in the data directory. In this case, Patroni was treating the version as 0.0, which was causing issues with some of the version-specific logic. With this fix, Patroni will try to get the version from the binary in such cases.
+
+- Refactor logger intialization to avoid missing early log messages (Alexander Kukushkin)
+
+  Create ``PatroniLogger`` before loading ``Config`` to capture early log messages.
+
+- Include ``MONOTONIC_USEC`` in ``RELOADING=1`` systemd notification (Alexander Kukushkin)
+
+  systemd 257+ requires ``MONOTONIC_USEC`` alongside ``RELOADING=1`` for ``Type=notify-reload`` services. Without it, ``systemctl reload`` hangs indefinitely.
+
+**Improvements**
+
+- Skip single-user crash recovery when ``backup_label`` exists (Vadim Ponomarev)
+
+  Skip single-user crash recovery and let PostgreSQL handle it during normal startup when starting a replica restored from an external backup (not using a custom bootstrap method).
+
+- Warn when running under ``systemd`` without ``python-systemd`` package (Alexander Kukushkin)
+
+  Instead of logging "systemd integration is not supported" at startup, check for ``NOTIFY_SOCKET`` and warn only when actually running under ``systemd`` without the ``python-systemd`` package installed.
+
+
 Version 4.1.2
 -------------
 

docs/rest_api.rst

@@ -342,7 +342,7 @@ Retrieve the Patroni metrics in Prometheus format through the ``GET /metrics`` e
 	# TYPE patroni_postgres_timeline counter
 	patroni_failsafe_mode_is_active{scope="batman",name="patroni1"} 0
 	# HELP patroni_postgres_timeline Postgres timeline of this node (if running), 0 otherwise.
-	# TYPE patroni_postgres_timeline counter
+	# TYPE patroni_postgres_timeline gauge
 	patroni_postgres_timeline{scope="batman",name="patroni1"} 24
 	# HELP patroni_dcs_last_seen Epoch timestamp when DCS was last contacted successfully by Patroni.
 	# TYPE patroni_dcs_last_seen gauge

patroni/__main__.py

@@ -13,12 +13,14 @@
 from typing import Any, Dict, List, Optional, TYPE_CHECKING
 
 from patroni import global_config, MIN_PSYCOPG2, MIN_PSYCOPG3, parse_version
+from patroni.collections import EMPTY_DICT
 from patroni.daemon import abstract_main, AbstractPatroniDaemon, get_base_arg_parser
 from patroni.tags import Tags
 
 if TYPE_CHECKING:  # pragma: no cover
     from .config import Config
     from .dcs import Cluster
+    from .log import PatroniLogger
 
 logger = logging.getLogger(__name__)
 
@@ -39,7 +41,7 @@ class Patroni(AbstractPatroniDaemon, Tags):
         * ``postmaster_start_time``: timestamp when Postgres was last started.
     """
 
-    def __init__(self, config: 'Config') -> None:
+    def __init__(self, config: 'Config', patroni_logger: 'PatroniLogger') -> None:
         """Create a :class:`Patroni` instance with the given *config*.
 
         Get a connection to the DCS, configure watchdog (if required), set up Patroni interface with Postgres, configure
@@ -49,6 +51,7 @@ def __init__(self, config: 'Config') -> None:
             Expected to be instantiated and run through :func:`~patroni.daemon.abstract_main`.
 
         :param config: Patroni configuration.
+        :param patroni_logger: the logging handler for this daemon.
         """
         from patroni import thread_pool
         from patroni.api import RestApiServer
@@ -68,7 +71,7 @@ def __init__(self, config: 'Config') -> None:
         logger.info('Patroni global thread_pool_size = %d', thread_pool_size)
         thread_pool.configure_global_pool(thread_pool_size)
 
-        super(Patroni, self).__init__(config)
+        super(Patroni, self).__init__(config, patroni_logger)
 
         self.version = __version__
         self.dcs = get_dcs(self.config)
@@ -144,15 +147,15 @@ def ensure_unique_name(self, cluster: 'Cluster') -> None:
         member = cluster.get_member(self.config['name'], False)
         if not isinstance(member, Member):
             return
+        # Silence annoying WARNING: Retrying (...) messages when Patroni is quickly restarted.
+        configured_loggers: Dict[str, Any] = (self.config.get('log') or EMPTY_DICT).get('loggers') or {}
         try:
-            # Silence annoying WARNING: Retrying (...) messages when Patroni is quickly restarted.
-            # At this moment we don't have custom log levels configured and hence shouldn't lose anything useful.
-            self.logger.update_loggers({'urllib3.connectionpool': 'ERROR'})
+            self.logger.update_loggers({**configured_loggers, 'urllib3.connectionpool': 'ERROR'})
             _ = self.request(member, endpoint="/liveness", timeout=3)
             logger.fatal("Can't start; there is already a node named '%s' running", self.config['name'])
             sys.exit(1)
         except Exception:
-            self.logger.update_loggers({})
+            self.logger.update_loggers(configured_loggers)
 
     def _get_tags(self) -> Dict[str, Any]:
         """Get tags configured for this node, if any.