From 898fdfa588a74c36d75f61d94126b50374e01f65 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Wed, 17 Jul 2024 17:22:59 +0100 Subject: [PATCH 1/3] document new promise-like APIs --- docs/jobs/job-writing-guide.md | 129 ++++++++++++++++++++++++++++++++- 1 file changed, 126 insertions(+), 3 deletions(-) diff --git a/docs/jobs/job-writing-guide.md b/docs/jobs/job-writing-guide.md index 7bb6b40e6a72..d4df63dbf725 100644 --- a/docs/jobs/job-writing-guide.md +++ b/docs/jobs/job-writing-guide.md @@ -90,8 +90,130 @@ Your job code should only contain Operations at the top level/scope - you should NOT include any other JavaScript statements. We'll talk about this more in a minute. +## Promise-like Operations + +:::tip + +Promise support is new to the openfn since July 2024, introduced in +`compiler@version` and `runtime@version` + +::: + +Operations behave like Javascript Promises in that they have `.then()` and +`.catch()` functions. This is useful for creating your own callbacks and error +handling + +### Callback with then() + +`then()` is available on every operation. It accepts the state as its argument +and MUST return a state object to be passed to the _next_ operation. + +```js +fn().then(state => { + console.log(state); + return state; // always remember to return state! +}); +``` + +:::info Note for developers + +Support for .then() is added by the compiler. Operations technically don't +return a Promise, they return a function, but the compiler will modify the job +code and wrap the operation in a deferred promise call. + +::: + +Most operations, when they run, will take your state object as an input, modify +it, and return it. The changed state will either be passed to the next +operation, or returned as the output of the job. + +Sometimes it's useful to compose operations together, like this: + +```js +each($.items, post(`patient/${$.data.id}`, $.data)); +``` + +The `each` function will take an array and, for each item, invoke a callback +with a scoped state. This means it takes your state object and sets the item +under iteration to `state.data`. In order words,in the callback `state.data` is +scoped to each item in the array. + +```js +each($.items, state => { + console.log(state.data); // each item in the items array + console.log(state.index); // the current index of iteration + return state; +}); +``` + +So in the example above, every item in state.items will be passed to a HTTP +`post()` function, where the id will be embedded in a URL and the item itself +will be uploaded to the server. + +But what if you want to do something with the scoped state AFTER the request? +Maybe you want to check the status code and log an error, or maybe you want to +mutate the data before writing it back to state. + +You can use `operation().then()` for this: + +```js +each( + $.items, + post(`patient/${$.data.id}`, $.data).then(state => { + state.completed.push(state.data); + return state; + }) +); +``` + +Now this expression will: + +- Iterate over each item in `state.items` +- Call the post operation with scoped state (ie, the item in `state.data`) +- Once the post is complete, pass the result as scoped state into the `.then()` + callback + +### Error handling with catch() + +Most adaptors will throw an error when something goes wrong, which may result in +the job (and maybe even workflow) ending early. + +Because every operation has a `catch()`, you have the opportunity in your job +code to intercept and even suppress the error. + +```js +get('patients').catch((error, state) => { + state.error = error; + return state; +}); +``` + +The error callback is passed two arguments: the error thrown by the adaptor, and +the state object. + +If you want to continue execution, you should return the state object from the +catch. This state will then be passed into the next operation. + +If you _do_ want to terminate execution, perhaps with some logging for debugging +or with a different error, you should throw from inside the catch handler. + +```js +get('patients').catch((error, state) => { + console.log('Error ocurred faithing patients', error); + throw error; +}); +``` + ## Callbacks and fn() +:::caution + +As of July 2024, callbacks are going to be phased out of the adaptor APIs. See +[Promise-like Operations](#promise-like-operations) for tips on how to use +callbacks with adaptors APIs that don't explicitly support them. + +::: + Many Operations give you access to a callback function. Callbacks will be invoked with state, will run whatever code you like, and must @@ -214,8 +336,9 @@ this: ```js get('/patients'); -each('$.data.patients[*]', (item, index) => { +each('$.data.patients[*]', state => { item.id = `item-${index}`; + return state; }); post('/patients', dataValue('patients')); ``` @@ -1048,8 +1171,8 @@ throw an exception to recognise that the job has failed. ## Compilation The code you write isn't technically executable JavaScript. You can't just run -it through node.js. It needs to be transformed or compiled into portable -vanilla JS code. +it through node.js. It needs to be transformed or compiled into portable vanilla +JS code. :::warning From 3de480695d85887878ccca82196b80f8ef8b619f Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 25 Jul 2024 14:19:49 +0100 Subject: [PATCH 2/3] re-write promise docs --- docs/jobs/job-writing-guide.md | 254 ++++++++++++++++++--------------- 1 file changed, 140 insertions(+), 114 deletions(-) diff --git a/docs/jobs/job-writing-guide.md b/docs/jobs/job-writing-guide.md index d4df63dbf725..043a8661b73e 100644 --- a/docs/jobs/job-writing-guide.md +++ b/docs/jobs/job-writing-guide.md @@ -90,120 +90,6 @@ Your job code should only contain Operations at the top level/scope - you should NOT include any other JavaScript statements. We'll talk about this more in a minute. -## Promise-like Operations - -:::tip - -Promise support is new to the openfn since July 2024, introduced in -`compiler@version` and `runtime@version` - -::: - -Operations behave like Javascript Promises in that they have `.then()` and -`.catch()` functions. This is useful for creating your own callbacks and error -handling - -### Callback with then() - -`then()` is available on every operation. It accepts the state as its argument -and MUST return a state object to be passed to the _next_ operation. - -```js -fn().then(state => { - console.log(state); - return state; // always remember to return state! -}); -``` - -:::info Note for developers - -Support for .then() is added by the compiler. Operations technically don't -return a Promise, they return a function, but the compiler will modify the job -code and wrap the operation in a deferred promise call. - -::: - -Most operations, when they run, will take your state object as an input, modify -it, and return it. The changed state will either be passed to the next -operation, or returned as the output of the job. - -Sometimes it's useful to compose operations together, like this: - -```js -each($.items, post(`patient/${$.data.id}`, $.data)); -``` - -The `each` function will take an array and, for each item, invoke a callback -with a scoped state. This means it takes your state object and sets the item -under iteration to `state.data`. In order words,in the callback `state.data` is -scoped to each item in the array. - -```js -each($.items, state => { - console.log(state.data); // each item in the items array - console.log(state.index); // the current index of iteration - return state; -}); -``` - -So in the example above, every item in state.items will be passed to a HTTP -`post()` function, where the id will be embedded in a URL and the item itself -will be uploaded to the server. - -But what if you want to do something with the scoped state AFTER the request? -Maybe you want to check the status code and log an error, or maybe you want to -mutate the data before writing it back to state. - -You can use `operation().then()` for this: - -```js -each( - $.items, - post(`patient/${$.data.id}`, $.data).then(state => { - state.completed.push(state.data); - return state; - }) -); -``` - -Now this expression will: - -- Iterate over each item in `state.items` -- Call the post operation with scoped state (ie, the item in `state.data`) -- Once the post is complete, pass the result as scoped state into the `.then()` - callback - -### Error handling with catch() - -Most adaptors will throw an error when something goes wrong, which may result in -the job (and maybe even workflow) ending early. - -Because every operation has a `catch()`, you have the opportunity in your job -code to intercept and even suppress the error. - -```js -get('patients').catch((error, state) => { - state.error = error; - return state; -}); -``` - -The error callback is passed two arguments: the error thrown by the adaptor, and -the state object. - -If you want to continue execution, you should return the state object from the -catch. This state will then be passed into the next operation. - -If you _do_ want to terminate execution, perhaps with some logging for debugging -or with a different error, you should throw from inside the catch handler. - -```js -get('patients').catch((error, state) => { - console.log('Error ocurred faithing patients', error); - throw error; -}); -``` - ## Callbacks and fn() :::caution @@ -691,6 +577,146 @@ you how the compiler is treating your State operators. +## Operations and Promises + +:::tip + +Promise support was added in July 2024 to `@openfn/compiler@0.2.0`. It is +available in the CLI from version 1.7.0 and the Lightning Worker from versison +1.4.0. + +::: + +Operations behave like Javascript Promises in that they have `.then()` and +`.catch()` functions. This is useful for creating your own callbacks and error +handling. + +:::info Note for developers + +Support for .then() is added by the compiler. Operations technically don't +return a Promise, they return a function, but the compiler will modify the job +code and wrap the operation in a deferred promise call. + +::: + +### Callback with then() + +`then()` is available on every operation, and contains a callback to be executed +once the operation has completed. + +The callback will receive the state returned by the operation, and must return +the state object to be passed to the _next_ operation. + +For example: + +```js +get($.data.url).then(state => { + console.log(state); + return state; // always remember to return state! +}); +``` + +If you're familiar with the callback pattern in our adaptors, `.then()` performs +exactly the same job as a callback. It gives you the opportunity to transform +the state returned by some operation. + +Usually, you don't need a callback or a `.then()` - you can just execute +operations serially. The following code is functionally the same as the prior +example: + +```js +get($.data.url); +fn(state => { + console.log(state); + return state; // always remember to return state! +}); +``` + +Where `.then()` is particularly useful is when composing operations with _scoped +state_, like with `each()`: + +```js +each($.items, post(`patient/${$.data.id}`, $.data)); +``` + +:::tip + +You can read more about the `each()` operation in +[Iteration with Each](#iteration-with-each). + +::: + +The `each` function will take an array and, for each item, invoke a callback +with a scoped state. This means it takes your state object and sets the item +under iteration to `state.data`. In order words, `state.data` inside the +callback is _scoped_ to each item in the array. + +```js +each($.items, state => { + console.log(state.data); // each item in the items array + console.log(state.index); // the current index of iteration + return state; +}); +``` + +So in the example above, every item in `state.items` will be passed to a HTTP +`post()` function, where the id will be embedded in a URL and the item itself +will be uploaded to the server. + +But what if you want to do something with the scoped state AFTER the request? +Maybe you want to check the status code and log an error, or maybe you want to +mutate the data before writing it back to state. + +You can use `operation().then()` for this: + +```js +each( + $.items, + post(`patient/${$.data.id}`, $.data).then(state => { + state.completed.push(state.data); + return state; + }) +); +``` + +Now this expression will: + +- Iterate over each item in `state.items` +- Call the post operation with scoped state (ie, the item in `state.data`) +- Once the post is complete, pass the result as scoped state into the `.then()` + callback + +### Error handling with catch() + +Most adaptors will throw an error when something goes wrong, which may result in +the job (and maybe even workflow) ending early. + +Because every operation has a `catch()`, you have the opportunity in your job +code to intercept and even suppress the error. + +```js +get('patients').catch((error, state) => { + state.error = error; + return state; +}); +``` + +The error callback is passed two arguments: the error thrown by the adaptor, and +the state object. + +If you want to continue execution, you should return the state object from the +catch. This state will then be passed into the next operation. + +If you _do_ want to terminate execution, perhaps with some logging for debugging +or with a different error, you should throw from inside the catch handler. + +```js +get('patients').catch((error, state) => { + console.log('Error ocurred faithing patients', error); + throw error; +}); +``` + ## Mapping Objects A common use-case in OpenFn fn is to map/convert/transform an object from system From 6ee1364004133204491c8cb38100cb76e99b33be Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Mon, 29 Jul 2024 12:17:28 +0100 Subject: [PATCH 3/3] fix typo in promise docs --- docs/jobs/job-writing-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/jobs/job-writing-guide.md b/docs/jobs/job-writing-guide.md index 043a8661b73e..5bec26f0eb1e 100644 --- a/docs/jobs/job-writing-guide.md +++ b/docs/jobs/job-writing-guide.md @@ -648,7 +648,7 @@ You can read more about the `each()` operation in The `each` function will take an array and, for each item, invoke a callback with a scoped state. This means it takes your state object and sets the item -under iteration to `state.data`. In order words, `state.data` inside the +under iteration to `state.data`. In other words, `state.data` inside the callback is _scoped_ to each item in the array. ```js