wip: support for user subsystems

.github/workflows/test.yaml

@@ -37,3 +37,7 @@ jobs:
           flux start jobspec run ./examples/group-with-group.yaml
           flux start jobspec run ./examples/task-with-group.yaml
           flux start python3 ./examples/flux/receive-job.py
+
+          # We don't need flux for satisfies
+          jobspec satisfy ./examples/subsystems/jobspec-spack-subystem-unsatisfied.yaml --subsystem-dir ./examples/subsystems || echo "Correct"
+          jobspec satisfy ./examples/subsystems/jobspec-spack-subystem-satisfied.yaml --subsystem-dir ./examples/subsystems

docs/docs/commands.md

@@ -6,7 +6,40 @@ A JobSpec consists of one or more tasks that have dependencies. This level of de
 The JobSpec library here reads in the JobSpec and can map that into specific cluster submit commands.
 Here is an example that assumes receiving a Jobspec on a flux cluster.
 
-### 1. Start Flux
+### Satisfy
+
+We are prototyping user-space subsystems, for which we do a satisfy request of a contender jobspec against a directory of user space subsystem files, which each should
+be JGF (json graph format) graphs. While this can be paired with run (to determine if the run should proceed) we provide a separate "satisfy" command to test and prototype the tool.
+We also provide a set of example user subsystems in `examples/subsystems` for each of environment modules and spack. This means we can do satsify requests of jobspecs against the subsystem directory as follows. Here is an example that is satisfied:
+
+```bash
+$ jobspec satisfy ./examples/subsystems/jobspec-spack-subystem-satisfied.yaml --subsystem-dir ./examples/subsystems
+```
+```console
+SELECT name from subsystems WHERE name = 'spack';
+SELECT label from nodes WHERE subsystem = 'spack' AND type = 'package';
+hairy-peanut-butter-6064 OK
+```
+
+And here is one not satisfied:
+
+```bash
+$ jobspec satisfy ./examples/subsystems/jobspec-spack-subystem-unsatisfied.yaml --subsystem-dir ./examples/subsystems
+```
+```console
+SELECT name from subsystems WHERE name = 'spack';
+SELECT label from nodes WHERE subsystem = 'spack' AND type = 'package';
+SELECT * from attributes WHERE name = 'name' AND value = 'lammps' and node IN ('package1','package2023','package2028','package2031','package2043','pac...
+loopy-car-9711 NOT OK
+```
+
+If run from python, the function "satisfied" would return False and the broker could respond appropriately. If you don't provide the `--subsystem-dir` it will default to `~/.compspec/subsystems`, which likely doesn't exist (and you'll get an error). Also note that the subsystem metadata
+is expected to be in JSON and our jobspec files are in yaml, so we can throw them into the same examples directory without issue.
+
+
+### Run
+
+#### 1. Start Flux
 
 Start up the development environment to find yourself in a container with flux. Start a test instance:
 
@@ -32,7 +65,7 @@ Ensure you have jobspec installed! Yes, we are vscode, installing to the contain
 sudo pip install -e .
 ```
 
-### 2. Command Line Examples
+#### 2. Command Line Examples
 
 We are going to run the [examples/hello-world-jobspec.yaml](examples/hello-world-jobspec.yaml). This setup is way overly
 complex for this because we don't actually need to do any staging or special work, but it's an example, so intended to be so.
@@ -70,7 +103,7 @@ jobspec run -t flux ./examples/hello-world-jobspec.yaml
 jobspec run --transformer flux ./examples/hello-world-jobspec.yaml
 ```
 
-### 3. Nested Examples
+#### 3. Nested Examples
 
 Try running some advanced examples. Here is a group within a task.
 
@@ -141,7 +174,7 @@ flux submit --job-name group-2-task-0 --flags=waitable bash -c echo Starting tas
 flux job wait --all
 ```
 
-### 4. Python Examples
+#### 4. Python Examples
 
 It could also be the case that you want something running inside a lead broker instance to receive Jobspecs incrementally and then
 run them. This Python example can help with that by showing how to accomplish the same, but from within Python.

docs/docs/spec.md

@@ -81,7 +81,84 @@ tasks:
   command: ["/bin/bash", "job.sh"]
 ```
 
-This above assumes a shared filesystem.
+This above assumes a shared filesystem. In addition to steps and the obvious command or depends on, tasks can have references to [resources](#resources), [requires](#requires), and [groups](#groups), discussed below.
+
+
+### Steps
+
+**Not thought through yet**
+
+Different workload managers have functionality for staging files, or similar tasks. We will try to define these as abstractions called "steps" where common needs (e.g., paths) are defined as variables, and each workload manager has a transformer that takes the variables and writes to the correct logic. A good example with flux is `flux archive`, which previously was called `flux filemap`. We might generalize this to the idea of staging files. We currently support the following steps:
+
+| Name   | Description |
+|--------|-------------|
+| stage  | stage files or directories |
+
+We hope to add these minimally, and only choosing ones that might be supported across environments or workload managers.
+
+#### Stage
+
+By default, the JobSpec doesn't know anything about having a shared filesystem or not. In the case of not, you might need to stage files, or ensure they are present across nodes before running any task. Here is what a staging task for a directory might look like:
+
+```yaml
+tasks:
+- name: setup
+  steps:
+    - name: stage
+      path: /tmp/path-for-workflow
+```
+
+or a file:
+
+```yaml
+- name: setup
+  steps:
+    - name: stage
+      path: /home/dinosaur/kubeconfig
+```
+
+
+If we were to write this out with a task (with flux commands) it might look like this:
+
+```yaml
+- name: root
+  command:
+    - /bin/bash
+    - -c
+    - flux archive create -n kubeconfig -C ./home/dinosaur kubeconfig
+    - flux -r all -x 0 flux exec flux archive get -n kubeconfig -C ./home/dinosaur
+```
+
+Instead of doing the above, we use the abstraction, and the underlying transformer does the translation. This means that different cluster transformers would parse the jobspec, and convert that into whatever their filemap/archive command is. We would likely have similar, abstract workload manager steps.
+
+## Requires
+
+The "requires" section (on the top level) includes user-level subsystem requirements for the JobSpec, which might include (but are not limited to) software, network, or I/O. This metadata will be discovered typically in the user home as separate JGF files (one per user subsystem) under `~/.compspec/subsystems`. The requires section, akin to resources, includes named groups of subsystem requirements that can be referenced in tasks. For example:
+
+```yaml
+requires:
+  spack-software:
+    - name: spack      # subsystem name
+      field: type      # field in metadata
+      match: package   # this is a spack package node
+      attribute: name  # These would be new - for the "attributes" section of the JGF
+      value: lammps
+
+  module-software:
+    - name: environment-modules
+      field: type
+      match: module
+      attribute: software
+      value: lammps
+
+tasks:
+- name: build
+  command: ["spack", "install", "ior"]
+  resources: spack-build
+  requires: spack-software|module-software
+```
+
+The above is referencing a [spack user subsystem](https://gist.github.com/vsoch/af1c57b558a476d1bb67fd78b284677e#file-spack-subsystem-json-L35-L43) and an [environment module](https://gist.github.com/vsoch/adba1cd620fb8280006e1533a3ab9928) user subsystem that are only accessible to the user that submits or owns the JobSpec. The library here, JobSpec Next Generation, will do a small satisfy check against the JobSpec and the user subsystems to inform the final selection.
 
 ## Groups
 
@@ -277,7 +354,7 @@ tasks:
 
 ### Requires
 
-The "requires" section includes compatibility metadata or key value pairs that are provided to a scheduler or image selection process to inform resource needs. Since we need to know the level of the graph to look (for example, a node attribute is different from a GPU one) we place them on the level of the resource definition. Any specification of "requires" is OPTIONAL. Requires are also implemented as a list of key value pairs (all strings) to make it easy for serialization into an interface, and parsing by custom match algorithm interfaces. Here is an example of adding requires to the spack job above.
+The "requires" section (on the level of a resource) includes compatibility metadata or key value pairs that are provided to a scheduler or image selection process to inform resource needs. Since we need to know the level of the graph to look (for example, a node attribute is different from a GPU one) we place them on the level of the resource definition. Any specification of "requires" is OPTIONAL. Requires are also implemented as a list of key value pairs (all strings) to make it easy for serialization into an interface, and parsing by custom match algorithm interfaces. Here is an example of adding requires to the spack job above.
 
 
 ```yaml
@@ -390,53 +467,6 @@ tasks:
   resources: spack
 ```
 
-### Steps
-
-**Not thought through yet**
-
-Different workload managers have functionality for staging files, or similar tasks. We will try to define these as abstractions called "steps" where common needs (e.g., paths) are defined as variables, and each workload manager has a transformer that takes the variables and writes to the correct logic. A good example with flux is `flux archive`, which previously was called `flux filemap`. We might generalize this to the idea of staging files. We currently support the following steps:
-
-| Name   | Description |
-|--------|-------------|
-| stage  | stage files or directories |
-
-We hope to add these minimally, and only choosing ones that might be supported across environments or workload managers.
-
-#### Stage
-
-By default, the JobSpec doesn't know anything about having a shared filesystem or not. In the case of not, you might need to stage files, or ensure they are present across nodes before running any task. Here is what a staging task for a directory might look like:
-
-```yaml
-tasks:
-- name: setup
-  steps:
-    - name: stage
-      path: /tmp/path-for-workflow
-```
-
-or a file:
-
-```yaml
-- name: setup
-  steps:
-    - name: stage
-      path: /home/dinosaur/kubeconfig
-```
-
-
-If we were to write this out with a task (with flux commands) it might look like this:
-
-```yaml
-- name: root
-  command:
-    - /bin/bash
-    - -c
-    - flux archive create -n kubeconfig -C ./home/dinosaur kubeconfig
-    - flux -r all -x 0 flux exec flux archive get -n kubeconfig -C ./home/dinosaur
-```
-
-Instead of doing the above, we use the abstraction, and the underlying transformer does the translation. This means that different cluster transformers would parse the jobspec, and convert that into whatever their filemap/archive command is. We would likely have similar, abstract workload manager steps.
-
 ## Support for Services
 
 A service can be:

examples/flux/jobspec.yaml

@@ -18,13 +18,9 @@ tasks:
 
 - name: task-2
   depends_on: ["task-1"]
-
-  # This is an example of a custom "task-level' requires
-  requires:
-    hardware.gpu.available: "yes"
   command:
       - bash
       - -c
       - "echo Starting task 2; sleep 3; echo Finishing task 2"
 
-  resources: single-node
+  resources: single-node

examples/subsystems/jobspec-spack-subystem-satisfied.yaml

@@ -0,0 +1,30 @@
+version: 1
+requires:
+  spack-software: 
+  - name: spack      # subsystem name
+    field: type      # field in metadata
+    match: package   # this is a spack package node
+    attribute: name  # These would be new - for the "attributes" section of the JGF
+    value: openssl
+
+resources:
+  sleep-resources:  
+    type: node
+    count: 1
+    requires:
+     - name: system
+       field: cpu.target
+       value: amd64
+
+    with:
+    - type: core
+      count: 4
+
+tasks:
+- name: task-1
+  command:
+    - bash
+    - -c
+    - "echo Starting task 1; sleep 3; echo Finishing task 1"
+  resources: sleep-resources
+  requires: spack-software

examples/subsystems/jobspec-spack-subystem-unsatisfied.yaml

@@ -0,0 +1,30 @@
+version: 1
+requires:
+  spack-software: 
+  - name: spack      # subsystem name
+    field: type      # field in metadata
+    match: package   # this is a spack package node
+    attribute: name  # These would be new - for the "attributes" section of the JGF
+    value: lammps
+
+resources:
+  sleep-resources:  
+    type: node
+    count: 1
+    requires:
+     - name: system
+       field: cpu.target
+       value: amd64
+
+    with:
+    - type: core
+      count: 4
+
+tasks:
+- name: task-1
+  command:
+    - bash
+    - -c
+    - "echo Starting task 1; sleep 3; echo Finishing task 1"
+  resources: sleep-resources
+  requires: spack-software