wqueue: Support custom user work queues.

Documentation/reference/os/wqueue.rst

@@ -149,10 +149,28 @@ and the user-mode work queue is functionally equivalent to the high
 priority work queue. It differs in that its implementation does not
 depend on internal, kernel-space facilities.
 
+**Custom User Work Queues**. Applications can use
+``work_queue_create()`` to create additional user-mode queues with a
+configurable priority and worker pool. The returned handle is passed to
+the ``*_wq()`` interfaces and to ``work_queue_free()``. The predefined
+``USRWORK`` queue remains available through the queue-ID interfaces. Custom
+user-mode queues require pthread support; the predefined ``USRWORK`` queue
+does not require pthread support in a protected build.
+
+**Execution Context**. The user-mode implementation uses mutexes and
+semaphores for synchronization. Its queue, cancel, create, priority, and
+destroy interfaces must therefore only be called from task context and
+must not be called from an interrupt handler. Kernel-mode and flat-build
+``work_queue()``, ``work_queue_wq()``, ``work_cancel()``, and
+``work_cancel_wq()`` remain safe for interrupt handlers. Creation,
+destruction, and synchronous cancellation are task-context operations in
+all build modes.
+
 **Configuration Options**.
 
 -  ``CONFIG_LIBC_USRWORK``. If CONFIG_LIBC_USRWORK is also defined
-   then the user-mode work queue will be enabled.
+   then the user-mode work queue will be enabled. Dynamically allocated
+   user-mode work queues require pthread support.
 -  ``CONFIG_LIBC_USRWORKPRIORITY``. The execution priority of the
    user-mode priority worker thread. Default: 100
 -  ``CONFIG_LIBC_USRWORKSTACKSIZE``. The stack size allocated for
@@ -202,7 +220,7 @@ Work Queue Interfaces
 ---------------------
 
 .. c:function:: int work_queue(int qid, FAR struct work_s *work, worker_t worker, \
-               FAR void *arg, uint32_t delay)
+               FAR void *arg, clock_t delay)
 
   Queue work to be performed at a later time. All
   queued work will be performed on the worker thread of execution
@@ -230,6 +248,56 @@ Work Queue Interfaces
 
   :return: Zero is returned on success; a negated errno is returned on failure.
 
+.. c:function:: FAR struct kwork_wqueue_s *work_queue_create( \
+               FAR const char *name, int priority, FAR void *stack_addr, \
+               int stack_size, int nthreads)
+
+  Create a custom work queue containing ``nthreads`` workers. All
+  workers use the requested name, priority, and stack size. If
+  ``stack_addr`` is ``NULL``, each worker stack is allocated by the
+  thread creation logic. Otherwise, ``stack_addr`` must identify storage
+  for ``nthreads * stack_size`` bytes.
+
+  This interface must only be called from task context.
+
+  :return: A work queue handle on success; ``NULL`` on failure.
+
+.. c:function:: int work_queue_free(FAR struct kwork_wqueue_s *wqueue)
+
+  Destroy a custom queue, discard pending work, and wait for all running
+  callbacks and worker threads to finish. Pending work structures become
+  available for reuse before the function returns. The predefined
+  ``HPWORK``, ``LPWORK``, and ``USRWORK`` queues cannot be destroyed.
+
+  This interface must only be called from task context and cannot be
+  called from one of the queue's own callbacks.
+
+  :return: Zero on success, ``-EINVAL`` for an invalid or predefined
+    queue, or ``-EDEADLK`` when called by one of the queue's workers.
+
+.. c:function:: int work_queue_wq(FAR struct kwork_wqueue_s *wqueue, \
+               FAR struct work_s *work, worker_t worker, FAR void *arg, \
+               clock_t delay)
+
+  Queue work on a custom queue. If the work structure is already pending
+  on the same queue, the pending instance is replaced. A work structure
+  must be cancelled before it is moved to another queue.
+
+  :return: Zero on success, ``-EINVAL`` for invalid arguments, or
+    ``-ESHUTDOWN`` after queue destruction starts.
+
+.. c:function:: int work_queue_next_wq( \
+               FAR struct kwork_wqueue_s *wqueue, \
+               FAR struct work_s *work, worker_t worker, FAR void *arg, \
+               clock_t delay)
+
+  Queue the next invocation relative to the work structure's previous
+  expiration time. This avoids accumulating callback execution time in a
+  periodic schedule. It is normally called from the work callback.
+
+  :return: Zero on success, ``-EINVAL`` for invalid arguments, or
+    ``-ESHUTDOWN`` after queue destruction starts.
+
 .. c:function:: int work_cancel(int qid, FAR struct work_s *work)
 
   Cancel previously queued work. This removes work
@@ -240,11 +308,37 @@ Work Queue Interfaces
   :param work: The previously queued work structure to cancel.
 
   :return: Zero is returned on success; a negated ``errno`` is returned on
-    failure.
+    failure. Cancelling work that is not queued is a successful no-op.
 
-    -  ``ENOENT``: There is no such work queued.
     -  ``EINVAL``: An invalid work queue was specified.
 
+.. c:function:: int work_cancel_wq(FAR struct kwork_wqueue_s *wqueue, \
+               FAR struct work_s *work)
+
+  Cancel pending work on a custom queue. Cancelling work that is not
+  queued is a successful no-op.
+
+  :return: Zero on success or ``-EINVAL`` for an invalid argument.
+
+.. c:function:: int work_cancel_sync_wq( \
+               FAR struct kwork_wqueue_s *wqueue, \
+               FAR struct work_s *work)
+
+  Cancel pending work and wait for callbacks already using the same work
+  structure to finish. If called from that work's own callback, the caller
+  is excluded from the wait to avoid self-deadlock.
+
+  This interface must only be called from task context.
+
+  :return: Zero on success or ``-EINVAL`` for an invalid argument.
+
+.. c:function:: int work_queue_priority_wq( \
+               FAR struct kwork_wqueue_s *wqueue)
+
+  Return the common scheduling priority of a custom queue's worker pool.
+
+  :return: The worker priority on success or a negated errno on failure.
+
 .. c:function:: int work_signal(int qid)
 
   Signal the worker thread to process the work
@@ -295,4 +389,3 @@ Work Queue Interfaces
 
   :param reqprio: Previously requested minimum worker thread
     priority to be "unboosted".
-

include/nuttx/wqueue.h

@@ -74,11 +74,15 @@
  *   priority worker thread.  Default: 2048.
  *
  * The user-mode work queue is only available in the protected or kernel
- * builds.  This those configurations, the user-mode work queue provides the
- * same (non-standard) facility for use by applications.
+ * builds.  In those configurations, the user-mode work queue provides the
+ * same (non-standard) facility for use by applications.  User-mode work
+ * queue APIs use blocking synchronization and must only be called from task
+ * context.  They must not be called from an interrupt handler.
  *
  * CONFIG_LIBC_USRWORK. If CONFIG_LIBC_USRWORK is also defined then the
- *   user-mode work queue will be created.
+ *   user-mode work queue will be created.  Dynamically allocated user-mode
+ *   work queues require pthread support.  The predefined protected-build
+ *   USRWORK queue does not require pthread support.
  * CONFIG_LIBC_USRWORKPRIORITY - The minimum execution priority of the lower
  *   priority worker thread.  Default: 100
  * CONFIG_LIBC_USRWORKSTACKSIZE - The stack size allocated for the lower
@@ -332,18 +336,18 @@ int work_usrstart(void);
  * Name: work_queue_create
  *
  * Description:
- *   Create a new work queue. The work queue is identified by its work
- *   queue ID, which is used to queue works to the work queue and to
- *   perform other operations on the work queue.
- *   This function will create a work thread pool with nthreads threads.
- *   The work queue ID is returned on success.
+ *   Create a custom work queue and return its handle.  The handle is used
+ *   to queue and cancel work, query the worker priority, and destroy the
+ *   queue.  This function creates a pool containing nthreads workers.
+ *   This function must only be called from task context.
+ *   User-mode custom queues require pthread support.
  *
  * Input Parameters:
  *   name       - Name of the new task
  *   priority   - Priority of the new task
  *   stack_addr - Stack buffer of the new task
  *   stack_size - size (in bytes) of the stack needed
- *   nthreads   - Number of work thread should be created
+ *   nthreads   - Number of worker threads to create
  *
  * Returned Value:
  *   The work queue handle returned on success.  Otherwise, NULL
@@ -359,16 +363,22 @@ FAR struct kwork_wqueue_s *work_queue_create(FAR const char *name,
  * Name: work_queue_free
  *
  * Description:
- *   Destroy a work queue. The work queue is identified by its work queue ID.
- *   All worker threads will be destroyed and the work queue will be freed.
- *   The work queue ID is invalid after this function returns.
+ *   Destroy a custom work queue.  All worker threads are stopped and the
+ *   queue is freed.  The handle is invalid after this function returns.
+ *   Only a custom queue returned by work_queue_create() may be destroyed;
+ *   the predefined HPWORK, LPWORK, and USRWORK queues cannot be destroyed.
+ *   This function must only be called from task context and must not be
+ *   called by one of the queue's own worker threads.
  *
  * Input Parameters:
  *  wqueue - The work queue handle
  *
  * Returned Value:
  *   Zero on success, a negated errno value on failure.
  *
+ *   -EDEADLK - Called by one of the queue's own worker threads.
+ *   -EINVAL  - The handle is NULL or does not identify a custom queue.
+ *
  ****************************************************************************/
 
 int work_queue_free(FAR struct kwork_wqueue_s *wqueue);
@@ -384,8 +394,14 @@ int work_queue_free(FAR struct kwork_wqueue_s *wqueue);
  *   the caller.  Otherwise, the work structure is completely managed by the
  *   work queue logic.  The caller should never modify the contents of the
  *   work queue structure directly.  If work_queue() is called before the
- *   previous work has been performed and removed from the queue, then any
- *   pending work will be canceled and lost.
+ *   previous work has been performed and removed from the same queue, then
+ *   any pending work will be canceled and replaced.  A queued work structure
+ *   must be cancelled before it is moved to a different work queue.
+ *
+ *   work_queue_wq() may be called from interrupt context for a kernel-mode
+ *   or flat-build queue.  A user-mode custom queue uses blocking
+ *   synchronization, so work_queue_wq() must only be called from task
+ *   context in user space.
  *
  * Input Parameters:
  *   qid    - The work queue ID (must be HPWORK or LPWORK)
@@ -399,7 +415,10 @@ int work_queue_free(FAR struct kwork_wqueue_s *wqueue);
  *            is invoked. Zero means to perform the work immediately.
  *
  * Returned Value:
- *   Zero on success, a negated errno on failure
+ *   Zero on success, a negated errno on failure.
+ *
+ *   -EINVAL    - An argument or delay is invalid.
+ *   -ESHUTDOWN - The custom work queue is being destroyed.
  *
  ****************************************************************************/
 
@@ -420,6 +439,10 @@ int work_queue_wq(FAR struct kwork_wqueue_s *wqueue,
  *   Note that calling this function outside the work callback requires
  *   the work->qtime being set.
  *
+ *   A user-mode custom queue uses blocking synchronization, so
+ *   work_queue_next_wq() must only be called from task context in user
+ *   space.
+ *
  * Input Parameters:
  *   qid    - The work queue ID (must be HPWORK or LPWORK)
  *   wqueue - The work queue handle
@@ -432,7 +455,10 @@ int work_queue_wq(FAR struct kwork_wqueue_s *wqueue,
  *            is invoked. Zero means to perform the work immediately.
  *
  * Returned Value:
- *   Zero on success, a negated errno on failure
+ *   Zero on success, a negated errno on failure.
+ *
+ *   -EINVAL    - An argument or delay is invalid.
+ *   -ESHUTDOWN - The custom work queue is being destroyed.
  *
  ****************************************************************************/
 
@@ -466,7 +492,12 @@ int work_queue_priority_wq(FAR struct kwork_wqueue_s *wqueue);
  * Description:
  *   Cancel previously queued work.  This removes work from the work queue.
  *   After work has been cancelled, it may be requeued by calling
- *   work_queue() again.
+ *   work_queue() again.  Cancelling work that is not queued is a successful
+ *   no-op.
+ *
+ *   work_cancel_wq() may be called from interrupt context for a kernel-mode
+ *   or flat-build queue.  It must only be called from task context for a
+ *   user-mode custom queue.
  *
  * Input Parameters:
  *   qid    - The work queue ID (must be HPWORK or LPWORK)
@@ -476,8 +507,7 @@ int work_queue_priority_wq(FAR struct kwork_wqueue_s *wqueue);
  * Returned Value:
  *   Zero on success, a negated errno on failure
  *
- *   -ENOENT - There is no such work queued.
- *   -EINVAL - An invalid work queue was specified
+ *   -EINVAL - An invalid work queue was specified.
  *
  ****************************************************************************/
 
@@ -489,23 +519,24 @@ int work_cancel_wq(FAR struct kwork_wqueue_s *wqueue,
  * Name: work_cancel_sync/work_cancel_sync_wq
  *
  * Description:
- *   Blocked cancel previously queued user-mode work.  This removes work
- *   from the user mode work queue.  After work has been cancelled, it may
- *   be requeued by calling work_queue() again.
+ *   Synchronously cancel previously queued work.  This removes work from
+ *   the queue and waits for callbacks that are already running.  After work
+ *   has been cancelled, it may be requeued by calling work_queue() again.
+ *   Cancelling work that is not queued is a successful no-op.
+ *   This function must only be called from task context.
  *
  * Input Parameters:
  *   qid    - The work queue ID (must be HPWORK or LPWORK)
  *   wqueue - The work queue handle
  *   work   - The previously queued work structure to cancel
  *
  * Returned Value:
- *   Zero means the work was successfully cancelled.
- *   One means the work was not cancelled because it is currently being
- *   processed by work thread, but wait for it to finish.
- *   A negated errno value is returned on any failure:
+ *   Zero means that queued work was cancelled and all callbacks using the
+ *   work structure have finished, except for a callback running in the
+ *   caller's own worker thread.  A negated errno value is returned on any
+ *   failure:
  *
- *   -ENOENT - There is no such work queued.
- *   -EINVAL - An invalid work queue was specified
+ *   -EINVAL - An invalid work queue was specified.
  *
  ****************************************************************************/
 

libs/libc/wqueue/Kconfig

@@ -9,9 +9,12 @@ menu "User Work Queue Support"
 config LIBC_USRWORK
 	bool "User mode worker thread"
 	default n
+	depends on BUILD_PROTECTED || !DISABLE_PTHREAD
 	---help---
-		User space work queues can also be made available for deferred
-		processing in the NuttX kernel build.
+		User-space work queues provide deferred processing in protected and
+		kernel builds.  Dynamically allocated user-mode work queues require
+		pthread support.  The predefined protected-build USRWORK queue does
+		not require pthread support.
 
 if LIBC_USRWORK