CFE-4074, ENT-14368: Now supports "file" and "string-multiline" as list subtypes in module input

JSON.md

@@ -706,6 +706,22 @@ The entered lines are joined with `\n` into a single string response, same as fo
   }
 ```
 
+A `"string-multiline"` can also be used as the `"subtype"` of a `"list"` input, to collect several blocks of text:
+
+```json
+        {
+          "type": "list",
+          "variable": "notes",
+          "label": "Notes",
+          "subtype": {
+            "type": "string-multiline",
+            "label": "Note",
+            "question": "What should the note say?"
+          },
+          "while": "Do you want to write another note?"
+        }
+```
+
 ### Create multiple files example
 
 Sometimes we would like a module to support taking an arbritary number of inputs.
@@ -900,18 +916,40 @@ $ cat ./out/masterfiles/def.json
 ### Referencing a file example
 
 The `"file"` input type lets a module ask the user for the path to an existing file (a script, playbook, etc.) instead of a value typed in directly - the file must already exist, `cfbs` doesn't generate its contents.
-Two optional attributes are available to the `"file"` input-type: `filetype` restricts which file extension(s) are accepted (a string or list of strings; any extension is accepted if omitted), and `while`, just like for `"list"`, lets the user supply any number of files instead of just one.
+One optional attribute is available to the `"file"` input-type: `filetype` restricts which file extension(s) are accepted (a string or list of strings; any extension is accepted if omitted).
+
+A `"file"` input asks for exactly one file, and its `"response"` is a single path:
 
 ```json
       "input": [
         {
           "type": "file",
-          "variable": "scripts",
+          "variable": "script",
           "namespace": "my_namespace",
           "bundle": "my_bundle",
           "label": "Script",
           "question": "Which script should be run?",
-          "filetype": ".sh",
+          "filetype": ".sh"
+        }
+      ]
+```
+
+To ask for several files, use a `"list"` input with a `"file"` subtype, exactly as you would for several strings - the `"while"` prompt of the list is what asks for each file after the first:
+
+```json
+      "input": [
+        {
+          "type": "list",
+          "variable": "scripts",
+          "namespace": "my_namespace",
+          "bundle": "my_bundle",
+          "label": "Scripts",
+          "subtype": {
+            "type": "file",
+            "label": "Script",
+            "question": "Which script should be run?",
+            "filetype": ".sh"
+          },
           "while": "Do you want to add another script?"
         }
       ]
@@ -929,20 +967,63 @@ Do you want to add another script? no
 $ cat ./run-scripts/input.json
 [
   {
-    "type": "file",
+    "type": "list",
     "variable": "scripts",
     "namespace": "my_namespace",
     "bundle": "my_bundle",
-    "label": "Script",
-    "question": "Which script should be run?",
-    "filetype": ".sh",
+    "label": "Scripts",
+    "subtype": {
+      "type": "file",
+      "label": "Script",
+      "question": "Which script should be run?",
+      "filetype": ".sh"
+    },
     "while": "Do you want to add another script?",
     "response": ["./run-scripts/deploy.sh", "./run-scripts/rollback.sh"]
   }
 ]
 ```
 
-Without `"while"`, `"response"` is a single path instead of a list.
+A `"file"` can also be one key among several in a list `"subtype"`, which is how a module asks for a file together with some options describing it:
+
+```json
+      "input": [
+        {
+          "type": "list",
+          "variable": "scripts",
+          "namespace": "my_namespace",
+          "bundle": "my_bundle",
+          "label": "Scripts",
+          "subtype": [
+            {
+              "key": "path",
+              "type": "file",
+              "label": "Script",
+              "question": "Which script should be run?",
+              "filetype": ".sh"
+            },
+            {
+              "key": "condition",
+              "type": "string",
+              "label": "Condition",
+              "question": "Condition for when to run it",
+              "default": "any"
+            }
+          ],
+          "while": "Do you want to add another script?"
+        }
+      ]
+```
+
+Each response is then an object, and only the `"file"` keys inside it are treated as paths:
+
+```json
+    "response": [
+      { "path": "./run-scripts/deploy.sh", "condition": "linux" },
+      { "path": "./run-scripts/rollback.sh", "condition": "any" }
+    ]
+```
+
 A file already inside the project is referred to as-is; a file from outside (as above) is copied into the module's own directory, next to `input.json`.
 
 #### Where these files end up in the built policy set
@@ -953,7 +1034,7 @@ During `cfbs build`, every `"file"` response is copied into the policy set (so i
 - Otherwise, the file is copied into `services/cfbs/`, preserving its path *relative to the project root* rather than just its filename, so that files with the same name coming from different sources (e.g. two different modules' file inputs both named `deploy.sh`) don't overwrite each other.
   - If the file lives inside the referencing module's own directory - as is normal for files placed there by `cfbs input`, like the two scripts above, or for a `"file"` input whose `"default"` points at a file the module ships itself - it's additionally namespaced under `services/cfbs/modules/<module-directory>/...`, keeping different modules' same-named files apart from each other.
 
-Continuing the example above, both scripts live inside `./run-scripts/`, next to `input.json`, so `cfbs build` copies them under `modules/run-scripts/`:
+Continuing the list-of-files example above, both scripts live inside `./run-scripts/`, next to `input.json`, so `cfbs build` copies them under `modules/run-scripts/`:
 
 ```
 $ cfbs build

cfbs/build.py

@@ -17,6 +17,7 @@
 import subprocess
 from cfbs.augments import generate_augment
 from cfbs.cfbs_config import CFBSConfig
+from cfbs.module_input import map_file_responses
 from cfbs.utils import (
     CFBSUserError,
     cli_tool_present,
@@ -287,9 +288,6 @@ def _localize_file_inputs(name, input_data, destination, build_modules):
     build step (e.g. the project author set one up manually), that step's
     destination is used instead of making a redundant copy.
     """
-    if not isinstance(input_data, list):
-        return
-
     module_dir_name = name[2:] if name.startswith("./") else name
     module_dir_name = os.path.basename(module_dir_name.rstrip("/"))
 
@@ -314,14 +312,7 @@ def _localize(rel_path):
         cp(rel_path, dest)
         return "$(sys.inputdir)/" + os.path.relpath(dest, destination)
 
-    for element in input_data:
-        if not isinstance(element, dict) or element.get("type") != "file":
-            continue
-        response = element.get("response")
-        if isinstance(response, list):
-            element["response"] = [_localize(path) for path in response]
-        else:
-            element["response"] = _localize(response)
+    map_file_responses(input_data, _localize)
 
 
 def _perform_input_step(args, name, destination, prefix, build_modules):

cfbs/cfbs_config.py

@@ -580,49 +580,43 @@ def _input_file(input_data):
             if filetypes is not None and not isinstance(filetypes, list):
                 filetypes = [filetypes]
 
-            def _one_file():
-                while True:
-                    response = prompt_user(
-                        self.non_interactive,
-                        input_data["question"],
-                        default=input_data.get("default"),
-                    )
-                    if self.non_interactive:
-                        return response
-                    if filetypes and not any(
-                        response.endswith(filetype) for filetype in filetypes
-                    ):
-                        print(
-                            "'%s' does not have one of the accepted file extensions (%s), please try again"
-                            % (response, ", ".join(filetypes))
-                        )
-                        continue
-                    if not os.path.isfile(response):
-                        print("File '%s' not found, please try again" % response)
-                        continue
+            while True:
+                response = prompt_user(
+                    self.non_interactive,
+                    input_data["question"],
+                    default=input_data.get("default"),
+                )
+                if self.non_interactive:
                     return response
-
-            if "while" not in input_data:
-                return _one_file()
-
-            result = {_one_file()}
-
-            while prompt_user_yesno(
-                self.non_interactive, input_data["while"], default="no"
-            ):
-                result.add(_one_file())
-            return list(result)
+                if filetypes and not any(
+                    response.endswith(filetype) for filetype in filetypes
+                ):
+                    print(
+                        "'%s' does not have one of the accepted file extensions (%s), please try again"
+                        % (response, ", ".join(filetypes))
+                    )
+                    continue
+                if not os.path.isfile(response):
+                    print("File '%s' not found, please try again" % response)
+                    continue
+                return response
+
+        def _input_subtype(subtype):
+            if subtype["type"] == "string":
+                return _input_string(subtype)
+            if subtype["type"] == "string-multiline":
+                return _input_multiline_string(subtype)
+            if subtype["type"] == "file":
+                return _input_file(subtype)
+            raise CFBSExitError(
+                "Subtype of type '%s' not supported for type list" % subtype["type"]
+            )
 
         def _input_elements(subtype):
             result = OrderedDict()
             for element in subtype:
                 _check_keys(["type", "label", "question", "key"], element)
-                if element["type"] != "string":
-                    raise CFBSExitError(
-                        "Subtype of type '%s' not supported for type list"
-                        % element["type"]
-                    )
-                result[element["key"]] = _input_string(element)
+                result[element["key"]] = _input_subtype(element)
             return result
 
         def _input_list(input_data):
@@ -641,16 +635,11 @@ def _input_list(input_data):
 
             elif isinstance(subtype, dict):
                 _check_keys(["type", "label", "question"], subtype)
-                if subtype["type"] != "string":
-                    raise CFBSExitError(
-                        "Subtype of type '%s' not supported for type list"
-                        % subtype["type"]
-                    )
-                result = [_input_string(subtype)]
+                result = [_input_subtype(subtype)]
                 while prompt_user_yesno(
                     self.non_interactive, input_data["while"], default="no"
                 ):
-                    result.append(_input_string(subtype))
+                    result.append(_input_subtype(subtype))
                 return result
             raise CFBSExitError(
                 "Expected the value of attribute 'subtype' to be a JSON list or object, not: %s"

cfbs/commands.py

@@ -130,6 +130,7 @@ def search_command(terms: List[str]):
 from cfbs.git_magic import commit_after_command, git_commit_maybe_prompt
 from cfbs.prompts import prompt_user, prompt_user_yesno
 from cfbs.module import Module, is_module_absolute, is_module_added_manually
+from cfbs.module_input import map_file_responses
 
 _MODULES_URL = "https://archive.build.cfengine.com/modules"
 
@@ -1821,9 +1822,7 @@ def _place_file_input(module_name, input_data):
     A file already inside the project is left where it is and simply
     referred to. A file from outside the project is copied into the
     module's directory, next to its input.json, and the response is
-    updated to point at that copy. A "file" input using "while" to collect
-    multiple files has a list of paths as its response, each handled the
-    same way.
+    updated to point at that copy.
 
     Returns the list of paths that were copied into the project, so they
     can be committed alongside input.json.
@@ -1850,14 +1849,7 @@ def _place(path):
         copied_files.append(dest)
         return dest
 
-    for definition in input_data:
-        if definition.get("type") != "file":
-            continue
-        response = definition.get("response")
-        if isinstance(response, list):
-            definition["response"] = [_place(path) for path in response]
-        else:
-            definition["response"] = _place(response)
+    map_file_responses(input_data, _place)
 
     return copied_files
 

cfbs/module_input.py

@@ -0,0 +1,55 @@
+"""
+Helpers for working with module input definitions and their responses.
+"""
+
+
+def _is_file(definition):
+    return isinstance(definition, dict) and definition.get("type") == "file"
+
+
+def _file_keys(subtype):
+    """The keys of a keyed list "subtype" which hold file paths"""
+    return [part["key"] for part in subtype if _is_file(part) and "key" in part]
+
+
+def _transform_keys(entry, keys, transform):
+    """Transform the given keys of a single response object"""
+    if not isinstance(entry, dict):
+        return
+    for key in keys:
+        if key in entry:
+            entry[key] = transform(entry[key])
+
+
+def map_file_responses(input_data, transform):
+    """Apply transform to every file path in a module's input responses.
+
+    Covers a top level "file" input and a "list" input with a "file" subtype.
+    Responses are rewritten in place, and anything which isn't a file path is
+    left alone.
+    """
+    if not isinstance(input_data, list):
+        return
+
+    for element in input_data:
+        if not isinstance(element, dict) or "response" not in element:
+            # Without a response there is nothing to map
+            continue
+        response = element["response"]
+
+        if _is_file(element):
+            # Single file
+            element["response"] = transform(response)
+
+        elif element.get("type") == "list" and isinstance(response, list):
+            subtype = element.get("subtype")
+
+            if _is_file(subtype):
+                # List of files
+                element["response"] = [transform(path) for path in response]
+
+            elif isinstance(subtype, list):
+                # List of objects where one or more keys are files
+                keys = _file_keys(subtype)
+                for entry in response:
+                    _transform_keys(entry, keys, transform)