Release: 1.0 (worker, engine, cli, runtime) (#592)

* engine: spike mapping console logs to an adaptor logger * runtime: messy tweak to module loading * engine,runtime: revert linker change and fix tests * engine: track test file * logger: dont stringify json output AND serialize errors This cause problems with the worker because errors get flattened to {}, and also we have to double parse. Now the logger will just emit whatever it logged to whatever the log emmiter is, so JSON stays as JSON. Which is good, but it no longer guarantees it'll be serializable * logger: tidy * engine: don't parse json logs coming out of the logger * engine, worker: better handling of objects coming from the logger The logger always sends raw json, but the log message is stringified by the engine, and rebuilt by the worker before sending to lightning this last bit needs work but its better * engine: fix tests * logger: tests and types * cli: update test * engine: types * worker: update tests * logger: set a special json emitter so that json logs get nicely printed in the CLI * logger: fix types * logger: log all json to .log * tests: fixes * logger: fix tests * logger: serialise print() properly * logger: types * engine: fix logs to gcp They were neglecting to parse the strings sent out by the new json logger * test: update log handling * engine: fix passing test It was secretly failing under the hood * runtime: add tests on job logger and errors * logger: improve detection of error objects * engine: tests on error logging * engine: restore adaptor logger * changesets * Tidy ups * engine: refactor log messages (and be a bit more lenient about structure) * worker: simplify logging * tiny tidyups * remove old docs * lexicon: start building a central lexicon of definitions * runtime: huge refactor of runtime core API * runtime: more refactoring * runtime: take initial state out of the execution plan * fix tests * runtime: changeset * runtime: extra type tweakings * runtime: readme * runtime: jobs -> steps (mostly) there are cases where job is more accurate and useful * cli: start refactoring towrads new runtime API Done a big chunk of execute but still a way to go * cli: basically get the CLI working again * cli: types * cli: fix a bunch of tests, update workflow parsing * cli: fix execute and compile tests * cli: more test fixes * fix more cli tests * cli: fix integration tests * cli: tidy * runtime: remove strict mode * remove strict mode * cli: default workflow name to the file name * runtime: tweak log output * cli: remove log * cli: types * docs * deploy: adjust logging * engine: update types * engine: update names and types This is 90% of the basic rename done. Tests may even pass * runtime: male statePropsToRemove a system options, rather than workflow specific If a workflow wants to remove props, it'll add an fn bock * engine: restore statePropsToRemove tests * mock: update to lexicon * worker: start mapping to lexicon. Handled run-> plan conversion * worker: typings * worker: fix all tests * engine: types * worker: fix cheeky test somehow missed it last time * tests: fix cli tests * worker: update test * package lock * tests: update test * changesets and housekeeping * more housekeeping * engine: tweak test * runtime: tweak error messages * worker: stricter type checkign on tests * fix test * typing in worker tests * worker: update channel mock * lexicon: docs * Run -> LightningPlan * version bumps for logger and mock * mock: return error if dataclip not found * worker: better handling of dataclip errors * lightning-mock: fix test * worker: changeset * worker: fix test Don't return the loaded dataclip after the refactor * worker: fix test again * Backend renaming (1.0 version bumps plus the lexicon) (#585) * lexicon: start building a central lexicon of definitions * runtime: huge refactor of runtime core API * runtime: more refactoring * runtime: take initial state out of the execution plan * fix tests * runtime: changeset * runtime: extra type tweakings * runtime: readme * runtime: jobs -> steps (mostly) there are cases where job is more accurate and useful * cli: start refactoring towrads new runtime API Done a big chunk of execute but still a way to go * cli: basically get the CLI working again * cli: types * cli: fix a bunch of tests, update workflow parsing * cli: fix execute and compile tests * cli: more test fixes * fix more cli tests * cli: fix integration tests * cli: tidy * runtime: remove strict mode * remove strict mode * cli: default workflow name to the file name * runtime: tweak log output * cli: remove log * cli: types * docs * deploy: adjust logging * engine: update types * engine: update names and types This is 90% of the basic rename done. Tests may even pass * runtime: male statePropsToRemove a system options, rather than workflow specific If a workflow wants to remove props, it'll add an fn bock * engine: restore statePropsToRemove tests * mock: update to lexicon * worker: start mapping to lexicon. Handled run-> plan conversion * worker: typings * worker: fix all tests * engine: types * worker: fix cheeky test somehow missed it last time * tests: fix cli tests * worker: update test * package lock * tests: update test * changesets and housekeeping * more housekeeping * engine: tweak test * runtime: tweak error messages * worker: stricter type checkign on tests * fix test * typing in worker tests * worker: update channel mock * lexicon: docs * Run -> LightningPlan * version bumps for logger and mock * Send worker versions (#593) * worker: send worker and API versions to Lightning * lexicon: fix API_VERSION export * cli: dont print compiler,runtime versions, also show monorepo for adaptor * cli tweak output to optionally show components * worker: simplify version output * mock: resolve conflict * Autoinstall by default (#594) * lexicon: start building a central lexicon of definitions * runtime: huge refactor of runtime core API * runtime: more refactoring * runtime: take initial state out of the execution plan * fix tests * runtime: changeset * runtime: extra type tweakings * runtime: readme * runtime: jobs -> steps (mostly) there are cases where job is more accurate and useful * cli: start refactoring towrads new runtime API Done a big chunk of execute but still a way to go * cli: basically get the CLI working again * cli: types * cli: fix a bunch of tests, update workflow parsing * cli: fix execute and compile tests * cli: more test fixes * fix more cli tests * cli: fix integration tests * cli: tidy * runtime: remove strict mode * remove strict mode * cli: default workflow name to the file name * runtime: tweak log output * cli: remove log * cli: types * docs * deploy: adjust logging * engine: update types * engine: update names and types This is 90% of the basic rename done. Tests may even pass * runtime: male statePropsToRemove a system options, rather than workflow specific If a workflow wants to remove props, it'll add an fn bock * engine: restore statePropsToRemove tests * mock: update to lexicon * worker: start mapping to lexicon. Handled run-> plan conversion * worker: typings * worker: fix all tests * engine: types * worker: fix cheeky test somehow missed it last time * tests: fix cli tests * worker: update test * package lock * tests: update test * changesets and housekeeping * more housekeeping * engine: tweak test * runtime: tweak error messages * worker: stricter type checkign on tests * fix test * typing in worker tests * worker: update channel mock * lexicon: docs * Run -> LightningPlan * version bumps for logger and mock * cli: autoinstall by default * cli: docs * changeset * cli: fix tests Need to disable autoinstall now or some tests will blow up! * openfnx: update console output * runtime: fix tests * worker: support output_dataclips on run options * worker: additioonal test of output_dataclips * types * mock: error if a credential does not exist * engine: throw nice exception if credentials fail to load * tests: add tset for bad credential * worker: bad credential test * mock: update dev endpoint to allow invalid credentials * changeset * worker: move tesdt into reasons * worker: tweak logs * Verify run token (#598) * worker: start trying to verify the attempt token * worker: roughly verify the run token * mock: generate a real jwt for runs * mock: tweak key handling * worker: verify the run token * changesets * todo * worker: support public key from env * worker: better cli handling * error handling * worker: destroy server if run token is invalid * test: add integration test for errors * tests: add keys to more tests * test: fix privateKey * tidyups * more tidyups * version lock pheonix to 1.7.10 1.7.11 introduces a compatability issue * logger: add proxy function to the mock * engine: don't send adaptor logs to stdout * tests: add test for adaptor logs * changeset * tests: remove logging * types * logger: rethink mock proxy. It's still not working. * logger: fix mock proxy function * engine: fix tests * tests: update tests * worker: fixed a tricky issue with server shutdown If a server is destroyed before the lightning connection returned, the workloop will still fire even if the server is technically destroyed * package lock * package lock * tests: tweak output * tests: run serially * tests: reorganise * version: [email protected] [email protected] --------- Co-authored-by: Taylor Downs <[email protected]>
OpenFn · Feb 15, 2024 · 7554a34 · 7554a34
1 parent 0f00be0
commit 7554a34
Show file tree

Hide file tree

Showing 220 changed files with 6,620 additions and 5,167 deletions.
diff --git a/build/install-global.js b/build/install-global.js
@@ -14,6 +14,8 @@ const outputPath = process.argv[2] || './dist';
 
 // Package everything up like a local build
 exec('git branch --show-current', {}, async (err, branchName) => {
+  console.log('Installing openfnx for branch:', branchName);
+  console.log();
   const files = await findPackages();
   const pkgs = mapPackages(files);
   await ensureOutputPath(outputPath);
@@ -37,9 +39,13 @@ exec('git branch --show-current', {}, async (err, branchName) => {
   ).then(async () => {
     const cliPath = getLocalTarballName(pkgs['@openfn/cli']);
     const command = `npm install -g ${path.resolve(outputPath, cliPath)}`;
-    console.log(command);
+    //console.log(command);
 
     await exec(command);
     // install the local CLI globally
+
+    console.log();
+    console.log('openfnx installed successfully! To test:');
+    console.log('  openfnx --version');
   });
 });
diff --git a/integration-tests/cli/test/cli.test.ts b/integration-tests/cli/test/cli.test.ts
@@ -14,7 +14,7 @@ test.serial('openfn version', async (t) => {
 test.serial('openfn test', async (t) => {
   const { stdout } = await run(t.title);
   t.regex(stdout, /Versions:/);
-  t.regex(stdout, /Running test job.../);
+  t.regex(stdout, /Running test workflow/);
   t.regex(stdout, /Result: 42/);
 });
 

diff --git a/integration-tests/cli/test/errors.test.ts b/integration-tests/cli/test/errors.test.ts
@@ -2,19 +2,20 @@ import test from 'ava';
 import path from 'node:path';
 import run from '../src/run';
 import { extractLogs, assertLog } from '../src/util';
+import { stderr } from 'node:process';
 
 const jobsPath = path.resolve('test/fixtures');
 
 // These are all errors that will stop the CLI from even running
 
-test.serial('job not found', async (t) => {
+test.serial('expression not found', async (t) => {
   const { stdout, err } = await run('openfn blah.js --log-json');
   t.is(err.code, 1);
 
   const stdlogs = extractLogs(stdout);
 
-  assertLog(t, stdlogs, /job not found/i);
-  assertLog(t, stdlogs, /failed to load the job from blah.js/i);
+  assertLog(t, stdlogs, /expression not found/i);
+  assertLog(t, stdlogs, /failed to load the expression from blah.js/i);
   assertLog(t, stdlogs, /critical error: aborting command/i);
 });
 

diff --git a/integration-tests/cli/test/execute-workflow.test.ts b/integration-tests/cli/test/execute-workflow.test.ts
@@ -83,6 +83,15 @@ test.serial(
   }
 );
 
+// Run a new-style execution plan with custom start
+test.serial(`openfn ${jobsPath}/plan.json -i`, async (t) => {
+  const { err } = await run(t.title);
+  t.falsy(err);
+
+  const out = getJSON();
+  t.deepEqual(out.data.userId, 1);
+});
+
 test.serial(`openfn ${jobsPath}/wf-conditional.json`, async (t) => {
   const { err } = await run(t.title);
   t.falsy(err);
@@ -124,36 +133,6 @@ test.serial(
   }
 );
 
-test.serial(`openfn ${jobsPath}/wf-strict.json --strict`, async (t) => {
-  const { err } = await run(t.title);
-  t.falsy(err);
-
-  const out = getJSON();
-  t.deepEqual(out, {
-    data: {
-      name: 'jane',
-    },
-  });
-});
-
-test.serial(`openfn ${jobsPath}/wf-strict.json --no-strict`, async (t) => {
-  const { err } = await run(t.title);
-  t.falsy(err);
-
-  const out = getJSON();
-  t.deepEqual(out, {
-    x: 22,
-    data: {
-      name: 'jane',
-    },
-    references: [
-      {
-        name: 'bob',
-      },
-    ],
-  });
-});
-
 test.serial(
   `openfn ${jobsPath}/wf-errors.json -S "{ \\"data\\": { \\"number\\": 2 } }"`,
   async (t) => {
@@ -169,8 +148,8 @@ test.serial(
   }
 );
 
-test.serial(
-  `openfn ${jobsPath}/wf-errors.json -S "{ \\"data\\": { \\"number\\": 32 } }"`,
+test.serial.only(
+  `openfn ${jobsPath}/wf-errors.json -iS "{ \\"data\\": { \\"number\\": 32 } }"`,
   async (t) => {
     const { err } = await run(t.title);
     t.falsy(err);
@@ -189,7 +168,7 @@ test.serial(
             severity: 'fail',
             source: 'runtime',
           },
-          jobId: 'start',
+          stepId: 'start',
           message: 'abort',
           type: 'JobError',
         },

diff --git a/integration-tests/cli/test/fixtures/circular.json b/integration-tests/cli/test/fixtures/circular.json
@@ -1,14 +1,16 @@
 {
-  "jobs": [
-    {
-      "id": "a",
-      "expression": "x",
-      "next": { "b": true }
-    },
-    {
-      "id": "b",
-      "expression": "x",
-      "next": { "a": true }
-    }
-  ]
+  "workflow": {
+    "steps": [
+      {
+        "id": "a",
+        "expression": "x",
+        "next": { "b": true }
+      },
+      {
+        "id": "b",
+        "expression": "x",
+        "next": { "a": true }
+      }
+    ]
+  }
 }
diff --git a/integration-tests/cli/test/fixtures/invalid-config-path.json b/integration-tests/cli/test/fixtures/invalid-config-path.json
@@ -1,7 +1,10 @@
 {
-  "jobs": [
-    {
-      "configuration": "does-not-exist.json"
-    }
-  ]
+  "workflow": {
+    "steps": [
+      {
+        "configuration": "does-not-exist.json",
+        "expression": "."
+      }
+    ]
+  }
 }
diff --git a/integration-tests/cli/test/fixtures/invalid-exp-path.json b/integration-tests/cli/test/fixtures/invalid-exp-path.json
@@ -1,7 +1,9 @@
 {
-  "jobs": [
-    {
-      "expression": "does-not-exist.js"
-    }
-  ]
+  "workflow": {
+    "steps": [
+      {
+        "expression": "does-not-exist.js"
+      }
+    ]
+  }
 }
diff --git a/integration-tests/cli/test/fixtures/invalid-start.json b/integration-tests/cli/test/fixtures/invalid-start.json
@@ -1,9 +1,13 @@
 {
-  "start": "nope",
-  "jobs": [
-    {
-      "id": "x",
-      "expression": "fn((state) => state)"
-    }
-  ]
+  "options": {
+    "start": "nope"
+  },
+  "workflow": {
+    "steps": [
+      {
+        "id": "x",
+        "expression": "fn((state) => state)"
+      }
+    ]
+  }
 }
diff --git a/integration-tests/cli/test/fixtures/invalid-syntax.json b/integration-tests/cli/test/fixtures/invalid-syntax.json
@@ -1,7 +1,9 @@
 {
-  "jobs": [
-    {
-      "expression": "invalid.js"
-    }
-  ]
+  "workflow": {
+    "steps": [
+      {
+        "expression": "invalid.js"
+      }
+    ]
+  }
 }
diff --git a/integration-tests/cli/test/fixtures/multiple-inputs.json b/integration-tests/cli/test/fixtures/multiple-inputs.json
@@ -1,18 +1,20 @@
 {
-  "jobs": [
-    {
-      "id": "a",
-      "expression": "x",
-      "next": { "b": true, "c": true }
-    },
-    {
-      "id": "b",
-      "expression": "x",
-      "next": { "c": true }
-    },
-    {
-      "id": "c",
-      "expression": "x"
-    }
-  ]
+  "workflow": {
+    "steps": [
+      {
+        "id": "a",
+        "expression": "x",
+        "next": { "b": true, "c": true }
+      },
+      {
+        "id": "b",
+        "expression": "x",
+        "next": { "c": true }
+      },
+      {
+        "id": "c",
+        "expression": "x"
+      }
+    ]
+  }
 }
diff --git a/integration-tests/cli/test/fixtures/plan.json b/integration-tests/cli/test/fixtures/plan.json
@@ -0,0 +1,19 @@
+{
+  "options": {
+    "start": "b"
+  },
+  "workflow": {
+    "steps": [
+      {
+        "id": "a",
+        "adaptor": "common",
+        "expression": "fn((state) => { return state; });"
+      },
+      {
+        "id": "b",
+        "adaptor": "http",
+        "expression": "get('https://jsonplaceholder.typicode.com/todos/1')"
+      }
+    ]
+  }
+}
diff --git a/integration-tests/cli/test/fixtures/wf-array.json b/integration-tests/cli/test/fixtures/wf-array.json
@@ -1,21 +1,23 @@
 {
-  "jobs": [
-    {
-      "id": "a",
-      "adaptor": "common",
-      "expression": "fn((state) => { if (!state.data.items) { state.data.items = []; } return state; });",
-      "next": { "b": true }
-    },
-    {
-      "id": "b",
-      "adaptor": "common",
-      "expression": "fn((state) => { state.data.items.push('b'); return state; });",
-      "next": { "c": true }
-    },
-    {
-      "id": "c",
-      "adaptor": "common",
-      "expression": "fn((state) => { state.data.items.push('c'); return state; });"
-    }
-  ]
+  "workflow": {
+    "steps": [
+      {
+        "id": "a",
+        "adaptor": "common",
+        "expression": "fn((state) => { if (!state.data.items) { state.data.items = []; } return state; });",
+        "next": { "b": true }
+      },
+      {
+        "id": "b",
+        "adaptor": "common",
+        "expression": "fn((state) => { state.data.items.push('b'); return state; });",
+        "next": { "c": true }
+      },
+      {
+        "id": "c",
+        "adaptor": "common",
+        "expression": "fn((state) => { state.data.items.push('c'); return state; });"
+      }
+    ]
+  }
 }
diff --git a/integration-tests/cli/test/fixtures/wf-conditional.json b/integration-tests/cli/test/fixtures/wf-conditional.json
@@ -1,29 +1,31 @@
 {
-  "start": "start",
-  "jobs": [
-    {
-      "id": "start",
-      "state": {
-        "data": {
-          "number": 1
+  "options": { "start": "start" },
+  "workflow": {
+    "steps": [
+      {
+        "id": "start",
+        "state": {
+          "data": {
+            "number": 1
+          }
+        },
+        "adaptor": "common",
+        "expression": "fn((state) => state);",
+        "next": {
+          "small": { "condition": "state.data.number < 10" },
+          "large": { "condition": "state.data.number >= 10" }
         }
       },
-      "adaptor": "common",
-      "expression": "fn((state) => state);",
-      "next": {
-        "small": { "condition": "state.data.number < 10" },
-        "large": { "condition": "state.data.number >= 10" }
+      {
+        "id": "small",
+        "adaptor": "common",
+        "expression": "fn((state) => { state.data.result = \"small\"; return state; });"
+      },
+      {
+        "id": "large",
+        "adaptor": "common",
+        "expression": "fn((state) => { state.data.result = \"large\"; return state; });"
       }
-    },
-    {
-      "id": "small",
-      "adaptor": "common",
-      "expression": "fn((state) => { state.data.result = \"small\"; return state; });"
-    },
-    {
-      "id": "large",
-      "adaptor": "common",
-      "expression": "fn((state) => { state.data.result = \"large\"; return state; });"
-    }
-  ]
+    ]
+  }
 }