Merge pull request #33 from VatsalJagani/splunklib-install-path-param…

…eter Splunklib install path parameter
VatsalJagani · Mar 30, 2024 · 326054d · 326054d
2 parents 691314d + 823656d
commit 326054d
Show file tree

Hide file tree

Showing 9 changed files with 278 additions and 78 deletions.
diff --git a/.github/workflows/py_unit_tests.yml b/.github/workflows/py_unit_tests.yml
@@ -4,10 +4,6 @@ name: Running Python Unit Tests for Splunk-App-Action
 # Controls when the action will run. Triggers the workflow on push or pull request
 on:
   push:
-    branches:
-      - 'develop'
-
-  pull_request:
     branches:
       - '*'
       - '*/*'

diff --git a/.vscode/settings.json b/.vscode/settings.json
@@ -0,0 +1,5 @@
+{
+    "cSpell.words": [
+        "splunklib"
+    ]
+}
diff --git a/DEV_README.md b/DEV_README.md
@@ -1,17 +1,11 @@
-# README for Developers
+# README for Developers of this action
 
 * Fork the project and submit a Pull Request if you would like to contribute to the project.
 
 * Automated tests for the GitHub action present in the separate repo here [splunk-app-action-test-app](https://github.com/VatsalJagani/splunk-app-action-test-app)
 
 
-## How to Run Locally
-```
-python3 src/main.py local_test
-```
-
-
-# Notes for author
+## How to generate tag
 
 * Create a new tag and push the tag from git:
   ```
@@ -33,13 +27,12 @@ python3 src/main.py local_test
   ```
 
 
-* Run below command to install all node JS dependencies.
-  * `npm install`
-
-* Run below command if you make any changes to JS file before committing the code to repo.
-  * `ncc build index.js --license LICENSE`
-  * This do not require if there is change in python file or action.yml file.
-
+## How to run python test-cases locally
+```
+python -m pip install --upgrade pip
+python -m pip install -r tests/requirements.txt
+pytest tests --junitxml=junit/test-results.xml --cov=src --cov-config=tests/.coveragerc --cov-report=xml
+```
 
 
 ## Note from actions/upload-artifact - Zipped Artifact Downloads

diff --git a/README.md b/README.md
@@ -47,7 +47,7 @@ graph TD
 * The action automatically generates build artifact from GitHub repo.
 
 ```
-- uses: VatsalJagani/splunk-app-action@v3
+- uses: VatsalJagani/splunk-app-action@v4
   with:
     app_dir: "my_app"
 ```
@@ -57,11 +57,11 @@ graph TD
 
 * Supports multiple Apps/Add-ons in single repository.
     ```
-    - uses: VatsalJagani/splunk-app-action@v3
+    - uses: VatsalJagani/splunk-app-action@v4
       with:
         app_dir: "my_splunk_app"
 
-    - uses: VatsalJagani/splunk-app-action@v3
+    - uses: VatsalJagani/splunk-app-action@v4
       with:
         app_dir: "my_splunk_add-on"
     ```
@@ -87,7 +87,7 @@ graph TD
         * Alternatively you can use `Running User Defined Commands Before Generating the final App Build` section here to see how you can assign right permission for your App/Add-on.
     * You can add `to_make_permission_changes: true` parameter to fix the issues with file and folder permissions to avoid App-inspect check automatically.
         ```
-        - uses: VatsalJagani/splunk-app-action@v3
+        - uses: VatsalJagani/splunk-app-action@v4
           with:
             app_dir: "my_app"
             to_make_permission_changes: true
@@ -109,7 +109,7 @@ graph TD
 
     * If you wish to run the user defined linux commands before generating the App build, set the environment variables `SPLUNK_APP_ACTION_<n>`.
         ```
-        - uses: VatsalJagani/splunk-app-action@v3
+        - uses: VatsalJagani/splunk-app-action@v4
           env:
             SPLUNK_APP_ACTION_1: "find . -type f -exec chmod 644 '{}' \\;"
             SPLUNK_APP_ACTION_2: "find . -type f -name '*.sh' -exec chmod +x '{}' \\;"
@@ -125,7 +125,7 @@ graph TD
     * It allows you to run command before building the App build.
         * This could be useful if you wish to remove some files that you don't want in the build, change permission of some files before running the rest of the app build or app-inspect check.
         ```
-        - uses: VatsalJagani/splunk-app-action@v3
+        - uses: VatsalJagani/splunk-app-action@v4
           env:
             SPLUNK_APP_ACTION_1: "rm -rf extra_test_folder"
           with:
@@ -148,7 +148,7 @@ graph TD
 * It requires to set inputs: splunkbase_username and splunkbase_password.
 
 ```
-- uses: VatsalJagani/splunk-app-action@v3
+- uses: VatsalJagani/splunk-app-action@v4
   with:
     app_dir: "my_app"
     splunkbase_username: ${{ secrets.SPLUNKBASE_USERNAME }}
@@ -165,7 +165,7 @@ graph TD
 #### `whats_in_the_app` - Utility that adds information about the App inside the README.md file
 * The splunk-app-action has utility which automatically adds information about the App, like how many alerts does it have, how many dashboards does it have, etc inside the App's README.md file.
 ```
-- uses: VatsalJagani/splunk-app-action@v3
+- uses: VatsalJagani/splunk-app-action@v4
   with:
     app_dir: "my_app"
     app_utilities: "whats_in_the_app"
@@ -176,7 +176,7 @@ graph TD
 * Auto adds python logger manager, including python file necessary, props.conf to assign right sourcetype for it under the internal logs.
 
 ```
-- uses: VatsalJagani/splunk-app-action@v3
+- uses: VatsalJagani/splunk-app-action@v4
   with:
     app_dir: "my_app"
     app_utilities: "logger"
@@ -188,19 +188,31 @@ graph TD
 #### `splunk_python_sdk` - Add Splunklib or Splunk SDK for Python and Auto Upgrades It
 * This utility adds the splunklib or Splunk SDK for Python to the App and auto upgrades it whenever new version is available.
 
+Example-1
 ```
-- uses: VatsalJagani/splunk-app-action@v3
+- uses: VatsalJagani/splunk-app-action@v4
   with:
     app_dir: "my_app"
     app_utilities: "splunk_python_sdk"
     my_github_token: ${{ secrets.MY_GITHUB_TOKEN }}
 ```
 
+Example-2
+```
+- uses: VatsalJagani/splunk-app-action@v4
+  with:
+    app_dir: "my_app"
+    app_utilities: "splunk_python_sdk"
+    my_github_token: ${{ secrets.MY_GITHUB_TOKEN }}
+    splunk_python_sdk_install_path: "bin/lib"
+```
+
+
 #### `common_js_utilities` - Add Common JavaScript Utilities File
 * This utility adds a JavaScript file that contains commonly used functionality for a JavaScript code for a Splunk App.
 
 ```
-- uses: VatsalJagani/splunk-app-action@v3
+- uses: VatsalJagani/splunk-app-action@v4
   with:
     app_dir: "my_app"
     app_utilities: "common_js_utilities"
@@ -211,7 +223,7 @@ graph TD
 * This utility adds additional_packaging.py file that contains code to better generate input handler python file to easily re-generate code on change, rather than making manual changes.
 
 ```
-- uses: VatsalJagani/splunk-app-action@v3
+- uses: VatsalJagani/splunk-app-action@v4
   with:
     app_dir: "."
     use_ucc_gen: true
@@ -287,6 +299,18 @@ def stream_events(input_script: smi.Script, inputs: smi.InputDefinition, event_w
 * description: "Sourcetype for the internal app logs. Required only for logger utility."
 * required: false
 
+#### splunk_python_sdk_install_path
+* description: "Path where you would like to install splunk-python-sdk (splunklib). Path is relative to App's root folder."
+* required: false
+* default: "bin"
+
+#### is_remove_pyc_from_splunklib_dir
+* description: "Remove `.pyc` files and `__pycache__` directory from splunk-python-sdk (splunklib) installation path before generating Pull Request. Do not turn this off unless you are facing any issues explicitly."
+* required: false
+* default: true
+
+
+
 
 
 ## Troubleshooting
@@ -309,6 +333,11 @@ def stream_events(input_script: smi.Script, inputs: smi.InputDefinition, event_w
 
 ## Release Notes
 
+### v4.1
+* Added `splunk_python_sdk_install_path` parameter for `splunk_python_sdk` utility. Default value is still `bin` folder, but now user can change based on their needs.
+* Avoid `.pyc` files and `__pycache__` files getting into `splunk_python_sdk` installation folder and getting into Pull Request by default. You can turn this off with `is_remove_pyc_from_splunklib_dir` parameter.
+
+
 ### v4
 * Run the user-defined commands in the context of your App's root directory instead of Repo's root directory. Refer to `Running User Defined Commands Before Generating the final App Build` for more details.
 
@@ -322,7 +351,7 @@ def stream_events(input_script: smi.Script, inputs: smi.InputDefinition, event_w
 * From `v4` of the `splunk-app-action`, your user-defined custom command (Refer to `Running User Defined Commands Before Generating the final App Build` section) would run in a context of your App's folder instead of root folder.
     * So you need to change the code from this:
         ```
-        - uses: VatsalJagani/splunk-app-action@v3
+        - uses: VatsalJagani/splunk-app-action@v4
           env:
             SPLUNK_APP_ACTION_1: "rm -rf my_app/extra_test_folder"
             SPLUNK_APP_ACTION_2: "cat 'abc,123' >> my_app/lookups/my_custom_lookup.csv"
@@ -331,7 +360,7 @@ def stream_events(input_script: smi.Script, inputs: smi.InputDefinition, event_w
         ```
         * to
         ```
-        - uses: VatsalJagani/splunk-app-action@v3
+        - uses: VatsalJagani/splunk-app-action@v4
           env:
             SPLUNK_APP_ACTION_1: "rm -rf extra_test_folder"
             SPLUNK_APP_ACTION_2: "cat 'abc,123' >> lookups/my_custom_lookup.csv"

diff --git a/action.yml b/action.yml
@@ -54,6 +54,17 @@ inputs:
     required: false
     default: "NONE"
 
+  splunk_python_sdk_install_path:
+    description: "Path where you would like to install splunk-python-sdk (splunklib). Path is relative to App's root folder."
+    required: false
+    default: "bin"
+
+  is_remove_pyc_from_splunklib_dir:
+    description: "Remove `.pyc` files and `__pycache__` directory from splunk-python-sdk (splunklib) installation path before generating Pull Request. Do not turn this off unless you are facing any issues explicitly."
+    required: false
+    default: true
+
+
 outputs:
   stdout:
     description: "Program stdout"
@@ -92,6 +103,8 @@ runs:
         echo "current_branch_name -> ${{steps.extract_branch.outputs.branch}}"
         echo "logger_log_files_prefix -> ${{inputs.logger_log_files_prefix}}"
         echo "logger_sourcetype -> ${{inputs.logger_sourcetype}}"
+        echo "splunk_python_sdk_install_path -> ${{inputs.splunk_python_sdk_install_path}}"
+        echo "is_remove_pyc_from_splunklib_dir -> ${{inputs.is_remove_pyc_from_splunklib_dir}}"
 
     - name: "Install Python"
       uses: actions/setup-python@v5
@@ -118,6 +131,8 @@ runs:
         SPLUNK_current_branch_name: ${{steps.extract_branch.outputs.branch}}
         SPLUNK_logger_log_files_prefix: ${{inputs.logger_log_files_prefix}}
         SPLUNK_logger_sourcetype: ${{inputs.logger_sourcetype}}
+        SPLUNK_splunk_python_sdk_install_path: ${{inputs.splunk_python_sdk_install_path}}
+        SPLUNK_is_remove_pyc_from_splunklib_dir: ${{inputs.is_remove_pyc_from_splunklib_dir}}
       run: |
         python -u ${{ github.action_path }}/src/main.py
 

diff --git a/src/utilities/splunk_sdk_python/__init__.py b/src/utilities/splunk_sdk_python/__init__.py
@@ -1,6 +1,7 @@
 
 import os
 import re
+import shutil
 
 import helpers.github_action_utils as utils
 from utilities.base_utility import BaseUtility
@@ -19,11 +20,30 @@ def _get_splunklib_version(self, file_path):
             utils.info("Error with getting the splunklib version.")
 
 
+    def remove_pycache(self, directory):
+        for root, dirs, files in os.walk(directory):
+            for file in files:
+                if file.endswith(".pyc"):
+                    os.remove(os.path.join(root, file))
+            for dir in dirs:
+                if dir == "__pycache__":
+                    shutil.rmtree(os.path.join(root, dir))
+
+
     def implement_utility(self):
         utils.info("Adding SplunkPythonSDKUtility")
 
+        splunk_python_sdk_install_path = utils.get_input('splunk_python_sdk_install_path')
+        utils.info(f"splunk_python_sdk_install_path: {splunk_python_sdk_install_path}")
+        if not splunk_python_sdk_install_path or splunk_python_sdk_install_path == "NONE":
+            splunk_python_sdk_install_path = "bin"
+
+        is_remove_pyc_from_splunklib_dir = utils.str_to_boolean_default_true(
+            utils.get_input('is_remove_pyc_from_splunklib_dir'))
+        utils.info(f"is_remove_pyc_from_splunklib_dir: {is_remove_pyc_from_splunklib_dir}")
+
         folder_to_install_splunklib = os.path.join(
-            self.app_write_dir, 'bin')
+            self.app_write_dir, splunk_python_sdk_install_path)
 
         if not os.path.exists(folder_to_install_splunklib):
             os.mkdir(folder_to_install_splunklib)
@@ -55,6 +75,10 @@ def implement_utility(self):
             utils.execute_system_command(
                 f'pip install splunk-sdk --target "{folder_to_install_splunklib}"')
 
+        # Removing .pyc and __pycache__
+        if is_remove_pyc_from_splunklib_dir:
+            self.remove_pycache(folder_to_install_splunklib)
+
         new_version = self._get_splunklib_version(init_file)
         utils.info(f"New splunklib version = {new_version}")
 

diff --git a/tests/helper.py b/tests/helper.py
@@ -17,7 +17,9 @@ def setup_action_yml(test_app_repo,
                             my_github_token="NONE",
                             current_branch="NONE",
                             logger_log_files_prefix="NONE",
-                            logger_sourcetype="NONE"):
+                            logger_sourcetype="NONE",
+                            splunk_python_sdk_install_path="bin",
+                            is_remove_pyc_from_splunklib_dir="true"):
 
     app_dir_path = os.path.join(os.path.dirname(__file__), "test_app_repos", test_app_repo)
     print(f"TestIntegration.setup_action_yml_work -> app_dir_path={app_dir_path}")
@@ -37,6 +39,8 @@ def setup_action_yml(test_app_repo,
     os.environ["SPLUNK_current_branch_name"] = current_branch
     os.environ["SPLUNK_logger_log_files_prefix"] = logger_log_files_prefix
     os.environ["SPLUNK_logger_sourcetype"] = logger_sourcetype
+    os.environ["SPLUNK_splunk_python_sdk_install_path"] = splunk_python_sdk_install_path
+    os.environ["SPLUNK_is_remove_pyc_from_splunklib_dir"] = is_remove_pyc_from_splunklib_dir
 
     try:
         yield