Implement ExecTask and ExecTaskStreaming

[ritesh-harihar]

Fixes: #5

Summary

Before ExecTask and ExecTaskStreaming were stubs that returned "not yet implemented". This meant:

• nomad alloc exec always failed with HTTP 500.
• Consul/Nomad script checks could not run inside the task.
• Interactive shells (nomad alloc exec -t ... /bin/sh) were impossible.

This PR implements all three exec entry-points — ExecTask, ExecTaskStreaming, and ExecTaskStreamingRaw — so the driver fully supports both script checks and nomad alloc exec, including interactive PTY sessions.

Command	Before (main)	After (this PR)
nomad alloc exec /bin/sh -i	500: ExecTaskStreaming is not yet implemented	Interactive shell inside task namespaces
nomad alloc exec /bin/echo hi	500: ExecTaskStreaming is not yet implemented	hi
nomad alloc exec /bin/false; echo $?	error, exit code meaningless	exit code 1 propagated correctly
echo x | nomad alloc exec -i cat	error	x (stdin piped through)
nomad alloc exec ip addr	500: ExecTaskStreaming is not yet implemented	Bridge-assigned IP (task's net namespace)
Consul/Nomad script checks	always failed	work correctly via ExecTask

How it works — namespace entry

Every exec path enters the running task's Linux namespaces by invoking nsenter targeting the shim's PID. The namespaces entered are:

Namespace	Flag	Why
mount	--mount	sees the task's unveiled filesystem view
pid	--pid	process tree isolation; exec'd process sees task PIDs only
ipc	--ipc	shared memory isolation
network	--net=	only when task uses bridge mode; skipped for host network

The resulting command prefix built by nsenterArgs():

nsenter --no-fork --target=<PID> --mount --pid --ipc [--net=<netns>] -- <command>

Three exec entry-points

Nomad calls different interfaces depending on context:

Interface	Caller	Use case	Implementation
ExecTask	Nomad server (script checks)	Blocking, buffered stdout/stderr, timeout	driver.go
ExecTaskStreamingRaw	Nomad gRPC server (preferred)	Streaming exec; live gRPC stream; supports PTY for -t	exec_raw_linux.go
ExecTaskStreaming	Nomad gRPC server (fallback only)	Streaming exec via io.Pipe wrappers; no PTY	driver.go

Nomad's gRPC server type-asserts ExecTaskStreamingRawDriver first. Because this driver implements it, ExecTaskStreaming is never called at runtime on Linux — it is kept as a documented fallback.

Request flow — before this PR

nomad alloc exec <id> /bin/sh
        │
        ▼
  Nomad gRPC server
        │  type-assert ExecTaskStreamingRawDriver? → NO
        │  type-assert ExecTaskStreamingDriver?    → YES (method existed as stub)
        │
        ▼
  ExecTaskStreaming()  ←── called
        │
        └──▶ return nil, errors.New("ExecTaskStreaming is not yet implemented")
                      │
                      ▼
               HTTP 500 to client

Request flow — after this PR

Path A — interactive shell (nomad alloc exec -t)

Details

nomad alloc exec -t <id> /bin/sh -i
        │
        ▼
  Nomad gRPC server
        │  type-assert ExecTaskStreamingRawDriver? → YES
        │
        ▼
  ExecTaskStreamingRaw(tty=true)
        │
        ├─ h.ExecInfo()          reads pid + netns under RLock
        ├─ nsenterArgs(pid,ns)   builds nsenter prefix
        ├─ exec.CommandContext   wraps full command
        │
        ▼
  execRawTTY(cmd, stream)
        │
        ├─ pty.Open()            allocates PTY pair (ptm master / pts slave)
        ├─ cmd.Stdin/Stdout/Stderr = pts
        ├─ SysProcAttr{Setsid, Setctty}   process becomes session leader
        ├─ cmd.Start()           forks into task namespaces via nsenter
        ├─ pts.Close()           parent closes slave; child owns it
        │
        ├── goroutine: stream.Recv() → ptm.Write()   (stdin + resize events)
        ├── goroutine: ptm.Read()  → stream.Send()   (stdout; tracked by WaitGroup)
        │
        ├─ cmd.Wait()            blocks until process exits
        ├─ ptm.Close()           unblocks stdout goroutine
        ├─ wg.Wait()             drain stdout goroutine
        └─ stream.Send(exitCode) sends final exit result to Nomad

Path B — non-interactive command (nomad alloc exec without -t)

Details

nomad alloc exec <id> /bin/echo hello
        │
        ▼
  Nomad gRPC server → ExecTaskStreamingRaw(tty=false)
        │
        ▼
  execRawNoTTY(cmd, stream)
        │
        ├─ io.Pipe() × 3        in-memory stdin / stdout / stderr pipes
        ├─ cmd.Start()           forks into task namespaces via nsenter
        │
        ├── goroutine: stream.Recv() → stdinW.Write()
        ├── goroutine: stdoutR.Read() → stream.Send(Stdout)   (WaitGroup)
        ├── goroutine: stderrR.Read() → stream.Send(Stderr)   (WaitGroup)
        │                        ↑ both use sync.Mutex to serialise Send()
        │
        ├─ cmd.Wait()            blocks until process exits
        ├─ stdinR/stdoutW/stderrW.Close()
        ├─ wg.Wait()             drain stdout+stderr goroutines
        └─ stream.Send(exitCode)

Path C — script checks (ExecTask)

Details

Nomad server (script check)
        │
        ▼
  ExecTask(taskID, cmd, timeout)
        │
        ├─ context.WithTimeout(timeout)
        ├─ h.ExecInfo()  →  nsenterArgs()  →  exec.CommandContext
        ├─ bytes.Buffer for stdout + stderr
        ├─ command.Run()   (blocking)
        └─ return ExecTaskResult{Stdout, Stderr, ExitCode}

Testing

Details

job "exec-test" {
  type = "service"

  constraint {
    attribute = "${attr.kernel.name}"
    value     = "linux"
  }

  group "group" {
    reschedule {
      attempts  = 0
      unlimited = false
    }

    restart {
      attempts = 0
      mode     = "fail"
    }

    task "sleep" {
      driver = "exec2"

      config {
        command = "sleep"
        args    = ["infinity"]
      }

      resources {
        cpu    = 100
        memory = 64
      }
    }
  }
}

Result Before:

riteshharihar@podman-dev:/Users/riteshharihar/Desktop/Src/nomad-driver-exec2$ nomad alloc exec -t $ALLOC /bin/sh -i
failed to exec into task: rpc error: code = Unknown desc = ExecTaskStreaming is not yet implemented

riteshharihar@podman-dev:/Users/riteshharihar/Desktop/Src/nomad-driver-exec2$ nomad alloc exec $ALLOC /bin/echo "hello from exec"
failed to exec into task: rpc error: code = Unknown desc = ExecTaskStreaming is not yet implemented

Result After:

riteshharihar@podman-dev:/Users/riteshharihar/Desktop/Src/nomad-driver-exec2$ nomad alloc exec $ALLOC /bin/echo "hello from exec"
hello from exec
             
riteshharihar@podman-dev:/Users/riteshharihar/Desktop/Src/nomad-driver-exec2$ nomad alloc exec $ALLOC /bin/cat /proc/1/cmdline
/tmp/nomad-plugins/nomad-driver-exec2exec2-shimtrue/tmp/NomadClient2727962377/911c7c71-f018-dd13-4767-a55ae04c3a8e-sleep/alloc/logs/.sleep.stdout.fifo/tmp/NomadClient2727962377/911c7c71-f018-dd13-4767-a55ae04c3a8e-sleep/alloc/logs/.sleep.stderr.fifo8042080420r:/etc/mime.typesr:/sys/fs/cgroup/nomad.slice/share.slice/911c7c71-f018-dd13-4767-a55ae04c3a8e.sleep.scoperwxc:/tmp/NomadClient2727962377/911c7c71-f018-dd13-4767-a55ae04c3a8e-sleep/localrwxc:/tmp/NomadClient2727962377/911c7c71-f018-dd13-4767-a55ae04c3a8e-sleep/allocrx:/tmp/NomadClient2727962377/911c7c71-f018-dd13-4767-a55ae04c3a8e-sleep/alloc/logsrwxc:/tmp/NomadClient2727962377/911c7c71-f018-dd13-4767-a55ae04c3a8e-sleep/secretsrwxc:/tmp/NomadClient2727962377/911c7c71-f018-dd13-4767-a55ae04c3a8e-sleep/tmp--sleepinfinity

riteshharihar@podman-dev:/Users/riteshharihar/Desktop/Src/nomad-driver-exec2$ nomad alloc exec $ALLOC /bin/false ; echo "exit: $?"
exit: 1

riteshharihar@podman-dev:/Users/riteshharihar/Desktop/Src/nomad-driver-exec2$ nomad alloc exec -t $ALLOC /bin/sh -i
# exit
riteshharihar@podman-dev:/Users/riteshharihar/Desktop/Src/nomad-driver-exec2$ nomad alloc exec -t $ALLOC /bin/sh
# ps aux
Error, do this: mount -t proc proc /proc
# hostname
podman-dev
# cat /proc/1/cmdline | tr '\0' ' '
/tmp/nomad-plugins/nomad-driver-exec2 exec2-shim true /tmp/NomadClient2265242779/67cf44fa-4fba-b8c3-8b41-6a8e272241bf-sleep/alloc/logs/.sleep.stdout.fifo /tmp/NomadClient2265242779/67cf44fa-4fba-b8c3-8b41-6a8e272241bf-sleep/alloc/logs/.sleep.stderr.fifo 85203 85203  r:/etc/mime.types r:/sys/fs/cgroup/nomad.slice/share.slice/67cf44fa-4fba-b8c3-8b41-6a8e272241bf.sleep.scope rwxc:/tmp/NomadClient2265242779/67cf44fa-4fba-b8c3-8b41-6a8e272241bf-sleep/local rwxc:/tmp/NomadClient2265242779/67cf44fa-4fba-b8c3-8b41-6a8e272241bf-sleep/alloc rx:/tmp/NomadClient2265242779/67cf44fa-4fba-b8c3-8b41-6a8e272241bf-sleep/alloc/logs rwxc:/tmp/NomadClient2265242779/67cf44fa-4fba-b8c3-8b41-6a8e272241bf-sleep/secrets rwxc:/tmp/NomadClient2265242779/67cf44fa-4fba-b8c3-8b41-6a8e272241bf-sleep/tmp -- sleep infinity # 
# 
# ls /proc | grep -E '^[0-9]+$'
1
11
# 

• If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

[ritesh-harihar]

Should we keep interactive shell support (-t)? Can remove it if not needed, as it would reduce significant amount of code.

Current PR includes exec_raw_linux.go + creack/pty which enables real PTY sessions (nomad alloc exec -t <id> /bin/sh — prompt, line editing, resize). Without it, -t still works but falls back to pipes — no prompt, no tab completion.

Dropping exec_raw_linux.go means nomad alloc exec will work — single commands, script checks, stdin piping, exit codes all function correctly. The only thing lost is the interactive shell experience: nomad alloc exec -t still runs but the shell has no prompt, no line editing, and no tab completion because it gets a pipe instead of a real terminal.

[tgross]

Made a first pass review but there's a lot of editing to do here before this is really ready

[tgross]

How?

[ritesh-harihar]

I rechecked the logic, it didn't actually guarantee that.

The resize goroutine keeps running until the client stream ends, so the Lock/Unlock only confirmed no Setsize was in progress at that exact instant, and nothing stopped a new Setsize from starting right after. The earlier code only caught the in-progress case at that instant, it did nothing to stop a new Setsize from starting afterward.

Added a closed flag under the same lock. Once that's set, the stdin goroutine sees it and backs off before calling Setsize, so nothing touches the PTY while it's being closed.

[tgross]

How do these differ from the stdoutDataMsg/stdoutCloseMsg functions? Does the close message even need to be a function rather than var declaration, given that it returns a literal?

[ritesh-harihar]

There is not much difference (only by the field) these can be refactored, yeah the close message didn't need to be a function.

Added two wrappers, wrapStdout/wrapStderr for these.

[tgross]

I keep coming back to this because it's not clear to me why we want the mutex to guard ptm in the first place. Isn't *os.File safe for concurrent use? drivers.ExecTaskStream isn't, but I thought the pty itself was?

[ritesh-harihar]

Yes *os.File is safe for concurrent use, so the mutex was never needed for the PTY itself. The one exception was pty.Setsize: creack/pty implements it as an ioctl on the raw fd from File.Fd(), which bypasses the poller and can race Close. So the mutex wasn't guarding the PTY, it was just a workaround for Setsize's unsafe raw-fd path.

Removed the mutex(which was confusing) and added new setPTYSize(), which keeps the fd valid for the call and is safe against a concurrent Close.

this is the exact race the pipeline caught before the fix: here.