Release 24.41.0

CHANGELOG.md

@@ -1,5 +1,11 @@
 # Change Log
 
+## 24.0.0
+* The Analytics Builder Block SDK is now moving to a continuous delivery (CD) model.
+
+## 10.18.1.0
+* .git and .github subdirectories are now skipped when building extensions. See [Building a block into an extension](doc/030-BuildingExtensions.md) for more details.
+
 ## 10.18.0.0
 * Cumulocity IoT input blocks can now receive from assets (and from their child devices). Existing input blocks may need to be updated to enable receiving from assets. See [Update Cumulocity IoT input blocks to receive from assets](doc/151-MigrateInputBlocksForAssetInput.md) for more details.
 

README.md

@@ -14,7 +14,13 @@ This excludes the Analytics Framework 'CDP' file, which is provided in binary fo
 
 ## Analytics Builder version
 
-This version of the SDK supports Analytics Builder 10.18.0.  To make use of this, you will require an installation of Apama 10.15.3. If you do not already have access, then you can download the 'community' edition from [Apama Downloads](https://apamacommunity.com/downloads) (shortly after the official Software AG release date).
+Use the 'main' branch for the current release or switch to the appropriate branch for Long-term support (LTS) / Maintenance releases.
+
+## System requirements
+
+This script requires an installation of Python 3.7+ and runs on either Windows or Linux. To build extensions or configure Software AG Designer, a full installation of Apama 10.15.4 is required. If you do not already have access, then you can download the Apama Community Edition from [Apama Downloads](https://apamacommunity.com/downloads) (shortly after the official Software AG release date). If you choose to install Apama, you can skip the manual installation of Python, as it is shipped with Apama.
+
+The [apama-builder](https://hub.docker.com/r/softwareag/apama-builder) Docker image from Docker Hub can also be used for building and managing extensions.
 
 Note that Analytics Builder requires your Cumulocity IoT tenant to be subscribed to an 'apama-ctrl' microservice.  The 'apama-ctrl-starter' microservice offers only restricted functionality and does not support custom blocks.
 

analytics_builder

@@ -29,11 +29,6 @@ class SubCommand(object):
 		else:
 			self.description = description
 
-def check_apama_home():
-	apama_home = os.getenv('APAMA_HOME', None)
-	if not apama_home: raise Exception('Please run this script from an apama_env shell or Apama Command Prompt.')
-	if not os.path.exists(apama_home): raise Exception('APAMA_HOME path does not exist: %s' % apama_home)
-
 def main():
 	commands = [
 		Command('build', 'build artifacts', [
@@ -108,7 +103,6 @@ def main():
 
 if __name__ == "__main__":
 	try:
-		check_apama_home()
 		main()
 	except Exception as e:
 		print(f'Command failed: {e}', file=sys.stderr)

analytics_builder.cmd

@@ -5,16 +5,19 @@ rem Use, reproduction, transfer, publication or disclosure is prohibited except
 
 setlocal
 
-if not defined APAMA_HOME (goto UNDEFINED)
-
 set THIS_SCRIPT=%~$PATH:0
 call :getpath %THIS_SCRIPT
 set "PATH=%APAMA_HOME%\third_party\python;%PATH%"
+
+set PYTHON_VERSION=0
+for /f %%i in ('python -c "import sys; print(sys.version_info[0])"') do set PYTHON_VERSION=%%i
+if %PYTHON_VERSION% LSS 3 (goto PY_UNDEFINED)
+
 python.exe "%THIS_DIR%/analytics_builder" %* || EXIT /B 1
 goto END
 
-:UNDEFINED
-echo Please run this script from an apama_env shell or Apama Command Prompt.
+:PY_UNDEFINED
+echo Add an appropriate Python 3 version to your path. Refer to the system requirements in the README of the Analytics Builder Block SDK documentation.
 goto END
 
 :END

bundles/AnalyticsBuilder.bnd

@@ -1,7 +1,7 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <bundle name="Analytics Builder" type="monitorscript" singleton="true">
 	<version>
-		<id>10.18.0.4</id>
+		<id>24.0.0</id>
 		<date>2019-06-07</date>
 		<author>Software AG</author>
 		<comments/>
@@ -17,6 +17,7 @@
 		</fileset>
 	</monitors>
 	<dependencies>
+		<dependency bundle-filename="CumulocityUtilities.bnd" catalog="${APAMA_HOME}/catalogs/bundles"/>
 		<dependency bundle-filename="JSONPlugin.bnd" catalog="${APAMA_HOME}/catalogs/bundles/"/>
 		<dependency bundle-filename="MemoryStore.bnd" catalog="${APAMA_HOME}/catalogs/bundles/"/>
 		<dependency bundle-filename="ScenarioService.bnd" catalog="${APAMA_HOME}/catalogs/bundles/"/>

bundles/CumulocityHelper.bnd

@@ -1,7 +1,7 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <bundle name="Cumulocity Block Helpers" type="monitorscript" singleton="true">
 	<version>
-		<id>10.18.0.4</id>
+		<id>24.0.0</id>
 		<date>2020-06-29</date>
 		<author>Software AG</author>
 		<comments/>

doc/000-contents.md

@@ -16,6 +16,7 @@
 15. [Asynchronous validations](110-AsynchronousValidations.md)
 16. [Migrating input and output blocks to the version 2 API](150-MigrateInputOutputBlocks.md)
 17. [Update Cumulocity IoT input blocks to receive from assets](151-MigrateInputBlocksForAssetInput.md)
-18. [Sharing data across partitions and workers](160-SharingDataAcrossPartition.md)
+18. [Update Cumulocity IoT input blocks to receive from all input sources](152-MigrateInputBlocksForAllInputs.md)
+19. [Sharing data across partitions and workers](160-SharingDataAcrossPartition.md)
 
 [Next: Introduction >](005-Intro.md) 

doc/005-Intro.md

@@ -6,7 +6,7 @@ Analytic models run in the Apama correlator, and are managed by the Analytics Bu
 
 Blocks can be tested using the PySys test framework that is included in an Apama installation. In order to develop, test and package blocks, you will need a full installation of Apama.
 
-Blocks are implemented in Apama's Event Processing Language (EPL). This guide assumes a working knowledge of EPL. Refer to the [Apama documentation on developing EPL applications](https://documentation.softwareag.com/pam/10.15.3/en/webhelp/pam-webhelp/index.html#page/pam-webhelp%2Fco-DevApaAppInEpl_how_this_book_is_organized.html) and the [API Reference for EPL (ApamaDoc)](https://documentation.softwareag.com/pam/10.15.3/en/webhelp/ApamaDoc/index.html). This guide also assumes a working knowledge of the Analytics Builder data model. Refer to the [Streaming Analytics guide](https://cumulocity.com/guides/10.17.0/streaming-analytics/analytics-builder/#getting-started).
+Blocks are implemented in Apama's Event Processing Language (EPL). This guide assumes a working knowledge of EPL. Refer to the [Apama documentation on developing EPL applications](https://documentation.softwareag.com/pam/10.15.4/en/webhelp/pam-webhelp/index.html#page/pam-webhelp%2Fco-DevApaAppInEpl_how_this_book_is_organized.html) and the [API Reference for EPL (ApamaDoc)](https://documentation.softwareag.com/pam/10.15.4/en/webhelp/related/ApamaDoc/index.html). This guide also assumes a working knowledge of the Analytics Builder data model. Refer to the [Streaming Analytics guide](https://cumulocity.com/docs/streaming-analytics/analytics-builder/#getting-started).
 
 The Analytics Builder block SDK has branches in a GitHub repository. You need to download the appropriate release of the block SDK based on the version of the Cumulocity IoT tenant you are using. Refer to the [Releases](https://github.com/SoftwareAG/apama-analytics-builder-block-sdk/releases) page of the block SDK. The version of the block SDK should be compatible with the version of the Cumulocity IoT tenant.
 

doc/030-BuildingExtensions.md

@@ -2,7 +2,7 @@
 
 Once a block is written in EPL, it can be packaged into an "extension". Extensions are **.zip** files that can be used to add blocks to the Analytics Builder runtime. Analytics Builder is deployed within Cumulocity IoT, and extensions are stored in the inventory. The extensions are only read when the Apama-ctrl microservice inside Cumulocity IoT is started, so the service must be restarted to make use of a new block. Restarting the microservice will lose state in any running models or custom EPL applications running in that tenant. This SDK provides a command line utility called `analytics_builder` which is available in the root directory of the SDK. This can be used to build an extension or list an extension or manage an extension in a Cumulocity IoT installation. You must navigate out of the model editor to the model manager and back to the editor to expose the new blocks after uploading an extension.
 
-Most of the `analytics_builder` commands use a `--input` argument which specifies the path to a directory. All files except **.log**, **.classpath**, **.dependencies**, **.project**, **.deploy**, **.launch**, **.out**, and **.o** found under that directory will be included, and message files matching **\*-messages.json** or named **messages.json** will be used for the runtime messages.
+Most of the `analytics_builder` commands use an `--input` argument that specifies the path to a directory. All files in that directory are included in the **.zip** file, except for the files with the following extensions: **.log**, **.classpath**, **.dependencies**, **.project**, **.deploy**, **.launch**, **.out**, and **.o**. Files in the **.git** and **.github** subdirectories are also not included. The message files named **messages.json** or matching **\*-messages.json** are used for the runtime messages.
 
 The `analytics_builder` script is run from an Apama command prompt (on Windows, run **Apama Command Prompt** from the Start Menu group of your Apama installation; on Linux, source the `apama_env` script). **Note:** You must place the script in a directory that does not have any spaces in its full path. 
 
@@ -18,6 +18,8 @@ The `analytics_builder` script takes a two-word command, followed by any argumen
 
 * `build extension --cumulocity_url <url> --username <user> --password <password> --name sample-blocks`
 
+*  A full version of Apama is required for building an extension.
+
   Build and upload an extension to a Cumulocity IoT instance. For example:
 
   ```bash
@@ -80,4 +82,35 @@ For `build extension` and `upload extension`:
 
 This avoids any confusion as to the functionality and ownership of the blocks. The samples are not productized blocks, and are subject to removal or changes in future releases without notice.
 
+## Invalid or corrupt extensions
+
+If the uploaded extension is invalid, corrupt, or duplicate, it is not injected into the correlator and warning or error messages are logged in the microservice logs. Here is a list of the warning or error messages that are generated and how to resolve the problem:
+
+* `Refusing to overwrite "<path>" as it is a core component`
+
+  If the extension tries to overwrite a file that belongs to the core component, a warning message is generated and the file is ignored. You should either delete or rename the file.
+
+* `Not copying extensions in safe mode. Extensions present but not loaded: <list of extensions>`
+
+  If there is an error while applying extensions, the Apama-ctrl microservice restarts in safe mode and extensions are not loaded. In such a scenario either the invalid or corrupt extension should be replaced or deleted and then the microservice should be restarted.
+
+* `Only files with extension .txt are supported for arguments, skipping: <path>`
+
+  Arguments should be provided in a file with a `.txt` extension. Otherwise, the file is ignored.
+
+* `Cannot handle file of unknown type: <file name>`
+
+  If the extension file is not a zip file, is corrupted, or the path to the extension directory is not valid, the above error is logged. You should either replace or delete the file.
+
+* `Duplicate extensions identified, starting the microservice without all extensions: <list of extensions>. Multiple extensions with the same name have been found: <list of duplicate extensions>`
+
+  If multiple extensions with the same name are specified, an alarm is raised, an error is logged, and none of the extensions are loaded. Delete or rename the listed duplicate extensions.
+
+* `Caught exception while creating extensions directory <extension dir> and reverted back to using the default '/config/extensions' folder`
+
+  If an error occurs while creating the specified extension directory, the default location is used. Specify a valid path or investigate why the folder could not be created. 
+
+* `Multiple managed objects found with pas_extension=<extension name>. Delete them and upload a new extension with the same name.`
+
+  When uploading an extension to the Cumulocity IoT inventory, if an extension with the same name already exists, its contents are replaced. If more than one extension with the same name is found in the inventory, the above error is raised. Delete the extensions and then upload a new extension with the same name.
 [< Prev: Naming and documenting blocks](020-NamingAndDoc.md) | [Contents](000-contents.md) | [Next: Testing blocks >](035-Testing.md) 

doc/035-Testing.md

@@ -1,6 +1,6 @@
 # Testing blocks
 
-Blocks can be tested using the PySys testing framework. This is included in the Apama installation, along with extensions for using Apama with PySys. Built on top of the Apama extensions is a framework to test blocks. Refer to the [Apama Python API documentation](https://documentation.softwareag.com/pam/10.15.3/en/webhelp/pydoc/index.html).
+Blocks can be tested using the PySys testing framework. This is included in the Apama installation, along with extensions for using Apama with PySys. Built on top of the Apama extensions is a framework to test blocks. Refer to the [Apama Python API documentation](https://documentation.softwareag.com/pam/10.15.4/en/webhelp/related/pydoc/index.html).
 
 The samples include tests. The `pysystestproject.xml` configuration relies on the environment variable `ANALYTICS_BUILDER_SDK` being set to the location of the block SDK using an absolute path. PySys tests should contain a `run.py` with a class that extends `apama.analyticsbuilder.basetest:AnalyticsBuilderBaseTest`. In the `execute` method of the test, start a correlator with the `self.startAnalyticsBuilderCorrelator()` method. This starts a correlator, injects the Analytics Builder framework into it, and returns a `CorrelatorHelper` object. Provide a `blockSourceDir` parameter with the path to the source of the blocks, typically within the project tree (use `self.project.SOURCE` from the supplied `pysysproject.xml` file). Then, create a model to test the block with the `self.createTestModel('<blockUnderTest>')`  This results in a model being activated in the correlator with an input and output connected to every input and output of the block, and an identifier of the model is returned. The block can be exercised by sending events created by the `self.inputEvent` method, for a given block input identifier.
 
@@ -14,10 +14,11 @@ Both `createTestModel` and `inputEvent` take an optional argument: `id` - an ide
 * `id` (optional) - identifier of the model
 * `corr` (optional) - correlator to use; defaults to using the last correlator started by `startAnalyticsBuilderCorrelator`.
 * `inputs` (optional) - a dictionary that maps the input names to their types.  For example `{ "input1" : "string", "input2" : "integer" }`. If an input identifier is not specified, the input defaults to the `float` type. If the value for the identifier is set to `None` or an empty string, that input is not connected.
+* `outputs` (optional) - a dictionary that maps the output names to their types.  For example `{ "output1" : None}`. If the value for the identifier is set to `None` or an empty string, that output is not connected to a TestOutput block. As a consequence this output will not be logged and it cannot be used in assertions.
 * `wiring` (optional unless testing multiple blocks) - list of strings containing source block index, output port identifier, target block index, input port identifier separated by colons - e.g. `['0:timeWindow:1:window']`
 
 
-The following methods can be used to check the output of the block is as expected:
+The following methods can be used to check the output of the block is as expected (unless the output has been mapped to `None` or an empty string):
 
 * `assertBlockOutput` checks that the series of outputs generated from a given outputId are as supplied in a list of values.  (optional parameters for partitionId and modelId)
 * `outputFromBlock` returns a list of the values sent to the named outputId (optional parameter for partitionId and modelId)

doc/060-Timers.md

@@ -5,10 +5,10 @@ Timers can be used by processing blocks (for example, the **Delay** block) to ex
 * Models executing in simulation mode, which run at some offset from real time.
 * Multiple timers firing at the same time from different blocks executing a single model "activation".
 
-The runtime thus knows what the current time of evaluation is. This is available from the `BlockBase.getModelTime()` action. Blocks should not use the `currentTime` variable.
-
 To create a timer, use the `BlockBase` methods `createTimer` and `createTimerWith`. The `createTimer` takes a relative offset time and a `payload`. The `createTimerWith` allows passing a `TimerParams` object to give more control.
 
+Blocks should not use the `currentTime` variable to get the current model time. The current model time depends on the duration of the reorder buffer used by the input blocks. If required, the current model time can be obtained from the `timestamp` field of the `Activation` object which is passed as a parameter to the `$process` and `$timerTriggered` actions.
+
 When the timer fires, it will call a method named `$timerTriggered` on the block. This can take parameters including:
 
 * `Activation $activation`
@@ -20,18 +20,20 @@ When the timer fires, it will call a method named `$timerTriggered` on the block
 
 From this `$timerTriggered`, it is thus possible to call the `$setOutput` actions with the provided `$activation` value.
 
-The `$payload` is the value provided to `createTimer` or added to the `TimerParams`.
+The `$payload` is the value provided to `createTimer` or added to the `TimerParams`. If the `$payload` is not provided, the value of the `$payload` will be the empty.
+
+The `$input_value` is one of the inputs of the block. When the timer triggers and the `$timerTriggered` action is called, `$input_value` will have the latest value that the block received and not the value when the timer was created.
 
 The following is a simple example of timers, to delay float values by 1 second:
 
 ```Java
 event Delay1Sec {
     BlockBase $base;
     action $process(float $input_value) {
-        any discard := $base.createTimerWith(TimerParams.relative(1.0));
+        any discard := $base.createTimerWith(TimerParams.relative(1.0).withPayload($input_value));
     }
-    action $timerTriggered(Activation $activation, float $input_value) {
-        $setOutput_delayed($activation, $input_value);
+    action $timerTriggered(Activation $activation, any $payload) {
+        $setOutput_delayed($activation,<float>$payload);
     }
     action<Activation, float> $setOutput_delayed;
 }