Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adds pyproject.toml using hatch and refactors dockerfile #5

Open
wants to merge 29 commits into
base: master
Choose a base branch
from
Open

Adds pyproject.toml using hatch and refactors dockerfile #5

wants to merge 29 commits into from

Conversation

fabricebrito
Copy link
Collaborator

No description provided.

fabricebrito and others added 27 commits September 25, 2024 10:58
@fabricebrito
refactoring for hatch/pyproject.toml
89e4ce5
@fabricebrito
removes Jenkinsfile
8f87263
first improvements
f7c4f6a
toml fixes
0e89939
explicit declaration of tests deps (pyyaml not found)
81b66a0
@xmichele
Merge branch 'EOEPCA:hatch' into hatch
768a95c
@xmichele
canonical package.yaml with pypi api token reference
f9dec8c
@xmichele
Update package.yaml with trusted publishers vs api_token
a81e6f4
@xmichele
Update package.yaml for __token__ handling
1083b06
@xmichele
new token
11e07b9
@xmichele
skip_existing: true
bef1969
@xmichele
disable trusted publishers based permissions
9306653
@xmichele
going for straight upload
948c10d
@xmichele
enable coverage computation in pyproject.toml
043a2ab
@fabricebrito
Merge pull request #2 from xmichele/hatch
35b8e9b 
hatch toml + publishing
@xmichele
Dockerfile update
f3f390f 
Test : 

[2/2] STEP 1/12: FROM rockylinux:9.3-minimal
[2/2] STEP 2/12: RUN microdnf install -y --nodocs nodejs &&     microdnf clean all
--> Using cache 667899aba51f2306da09f822e16fd050fdbdb75a07af3d34e454a153be8de365
--> 667899aba51
[2/2] STEP 3/12: ENV HOME=/home/wrapper
--> Using cache 0a9aa50606757894af53f20712a907e259cf9ead35d87d142aa240823543b839
--> 0a9aa506067
[2/2] STEP 4/12: RUN useradd -u 1001 -r -g 0 -m -d ${HOME} -s /sbin/nologin         -c "Default CWL Wrapper User" wrapper &&     mkdir -p /app &&     mkdir -p /prod &&     chown -R 1001:0 /app &&     chmod g+rwx ${HOME} /app
--> Using cache 5800dea4f36d3f88b74059c268ab9198fee18f491019a00f6d9b889aa422af1b
--> 5800dea4f36
[2/2] STEP 5/12: COPY --from=build /tmp/hatch /usr/bin/hatch
--> Using cache c05669d27eb59cbf264e8ad7aeb94320efc4b77444f724d6687ce2023f5d22a0
--> c05669d27eb
[2/2] STEP 6/12: USER wrapper
--> fae4b9ba289
[2/2] STEP 7/12: COPY --chown=1001:0 . /app
--> 6f4a7e88ea7
[2/2] STEP 8/12: WORKDIR /app
--> d4c2ac31639
[2/2] STEP 9/12: ENV VIRTUAL_ENV=/app/envs/wrapper
--> f08b369922e
[2/2] STEP 10/12: ENV PATH="$VIRTUAL_ENV/bin:$PATH"
--> afe7786f3d0
[2/2] STEP 11/12: RUN hatch env prune &&     hatch env create prod &&     rm -fr /tmp/* /app/.git /app/.pytest_cache
Creating environment: prod
Installing Python distribution: 3.12
Installing project in development mode
Checking dependencies
Syncing dependencies
--> fb060933ca1
[2/2] STEP 12/12: CMD ["bash", "-c", "source /app/envs/wrapper/bin/activate && cwl-wrapper"]
[2/2] COMMIT cwl-wrapper
--> dced1594b00
Successfully tagged localhost/cwl-wrapper:latest
dced1594b004868c722bed221891bbfbec0a98a5a7de10e014082c497f9cf5cb
(base) jovyan@jupyter-mlongobardodev:~/projects/cwl-wrapper-hatch$ 
(base) jovyan@jupyter-mlongobardodev:~/projects/cwl-wrapper-hatch$ 
(base) jovyan@jupyter-mlongobardodev:~/projects/cwl-wrapper-hatch$ docker run -it --rm -v $PWD/myfolder/:/myfolder --entrypoint=bash localhost/cwl-wrapper:latest 
[wrapper@80ed5a42da87 app]$ cwl-wrapper --help
Usage: cwl-wrapper [OPTIONS] CWL

  The cwl-parser

Options:
  --output TEXT       default main cel template assets/maincwl.yaml
  --stagein TEXT      default stagein template assets/stagein.yaml
  --stageout TEXT     default "stageout" template assets/stageout.yaml
  --maincwl TEXT      default maincwl assets/maincwl.yaml
  --rulez TEXT        rules default file assets/rules.yaml
  --assets TEXT       use <value> as maincwl from assets/<values>
  --workflow-id TEXT  workflow id
  -c, --conf FILE     Read options from FILE instead of command line; default
                      file: /home/jovyan/.cwlwrapper/default.conf
  --help              Show this message and exit.
(base) jovyan@jupyter-mlongobardodev:~/projects/cwl-wrapper-hatch$ docker run -it --rm -v $PWD/myfolder/:/app/myfolder --entrypoint=bash localhost/cwl-wrapper:latest 
[wrapper@1929d7c0cc59 app]$ cwl-wrapper --help 
Usage: cwl-wrapper [OPTIONS] CWL

  The cwl-parser

Options:
  --output TEXT       default main cel template assets/maincwl.yaml
  --stagein TEXT      default stagein template assets/stagein.yaml
  --stageout TEXT     default "stageout" template assets/stageout.yaml
  --maincwl TEXT      default maincwl assets/maincwl.yaml
  --rulez TEXT        rules default file assets/rules.yaml
  --assets TEXT       use <value> as maincwl from assets/<values>
  --workflow-id TEXT  workflow id
  -c, --conf FILE     Read options from FILE instead of command line; default
                      file: /home/jovyan/.cwlwrapper/default.conf
  --help              Show this message and exit.
[wrapper@1929d7c0cc59 app]$ cwl-wrapper myfolder/map-composition.cwl 
$graph:
- $namespaces:
    cwltool: http://commonwl.org/cwltool#
  class: Workflow
  doc: Main stage manager
  hints:
    cwltool:Secrets:
      secrets:
      - ADES_STAGEIN_AWS_SERVICEURL
      - ADES_STAGEIN_AWS_ACCESS_KEY_ID
      - ADES_STAGEIN_AWS_SECRET_ACCESS_KEY
      - ADES_STAGEOUT_AWS_SERVICEURL
      - ADES_STAGEOUT_AWS_ACCESS_KEY_ID
      - ADES_STAGEOUT_AWS_SECRET_ACCESS_KEY
  id: main
  inputs:
    ADES_STAGEIN_AWS_ACCESS_KEY_ID:
      type: string?
    ADES_STAGEIN_AWS_REGION:
      type: string?
    ADES_STAGEIN_AWS_SECRET_ACCESS_KEY:
      type: string?
    ADES_STAGEIN_AWS_SERVICEURL:
      type: string?
    ADES_STAGEOUT_AWS_ACCESS_KEY_ID:
      type: string?
    ADES_STAGEOUT_AWS_REGION:
      type: string?
    ADES_STAGEOUT_AWS_SECRET_ACCESS_KEY:
      type: string?
    ADES_STAGEOUT_AWS_SERVICEURL:
      type: string?
    ADES_STAGEOUT_OUTPUT:
      type: string?
    base_map:
      id: base_map
      type: string
    bounding_box:
      id: bounding_box
      type: string?
    catalog_url:
      id: catalog_url
      type: string
    copyrights:
      id: copyrights
      type: string
    date:
      id: date
      type: string?
    input_raster:
      id: input_raster
      type: string[]
    legend:
      id: legend
      type: string[]?
    logo:
      id: logo
      type: string[]?
    map_desc:
      id: map_desc
      type: string?
    process:
      type: string
    subtitle:
      id: subtitle
      type: string
    title:
      id: title
      type: string
  label: macro-cwl
  outputs:
    s3_catalog_output:
      id: s3_catalog_output
      outputSource:
      - node_stage_out/s3_catalog_output
      type: string
    wf_outputs:
      id: wf_outputs
      outputSource:
      - node_stage_out/wf_outputs_out
      type: Directory
  requirements:
    InlineJavascriptRequirement: {}
    ScatterFeatureRequirement: {}
    SubworkflowFeatureRequirement: {}
  steps:
    node_stage_out:
      in:
        ADES_STAGEOUT_AWS_ACCESS_KEY_ID: ADES_STAGEOUT_AWS_ACCESS_KEY_ID
        ADES_STAGEOUT_AWS_REGION: ADES_STAGEOUT_AWS_REGION
        ADES_STAGEOUT_AWS_SECRET_ACCESS_KEY: ADES_STAGEOUT_AWS_SECRET_ACCESS_KEY
        ADES_STAGEOUT_AWS_SERVICEURL: ADES_STAGEOUT_AWS_SERVICEURL
        ADES_STAGEOUT_OUTPUT: ADES_STAGEOUT_OUTPUT
        process: process
        wf_outputs: on_stage/wf_outputs
      out:
      - s3_catalog_output
      - wf_outputs_out
      run:
        arguments:
        - copy
        - -v
        - -r
        - '4'
        - -o
        - $( inputs.ADES_STAGEOUT_OUTPUT + "/" + inputs.process )
        - valueFrom: "${\n    if( !Array.isArray(inputs.wf_outputs) )\n    {\n   \
            \     return inputs.wf_outputs.path + \"/catalog.json\";\n    }\n    var\
            \ args=[];\n    for (var i = 0; i < inputs.wf_outputs.length; i++)\n \
            \   {\n        args.push(inputs.wf_outputs[i].path + \"/catalog.json\"\
            );\n    }\n    return args;\n}\n"
        baseCommand: Stars
        class: CommandLineTool
        cwlVersion: v1.0
        doc: Run Stars for staging results
        hints:
          DockerRequirement:
            dockerPull: terradue/stars:2.3.0
          cwltool:Secrets:
            secrets:
            - ADES_STAGEOUT_AWS_SERVICEURL
            - ADES_STAGEOUT_AWS_ACCESS_KEY_ID
            - ADES_STAGEOUT_AWS_SECRET_ACCESS_KEY
        id: stars
        inputs:
          ADES_STAGEOUT_AWS_ACCESS_KEY_ID:
            type: string?
          ADES_STAGEOUT_AWS_REGION:
            type: string?
          ADES_STAGEOUT_AWS_SECRET_ACCESS_KEY:
            type: string?
          ADES_STAGEOUT_AWS_SERVICEURL:
            type: string?
          ADES_STAGEOUT_OUTPUT:
            type: string?
          process:
            type: string
          wf_outputs:
            type: Directory
        outputs:
          s3_catalog_output:
            outputBinding:
              outputEval: ${ return inputs.ADES_STAGEOUT_OUTPUT + "/" + inputs.process
                + "/catalog.json"; }
            type: string
          wf_outputs_out:
            outputBinding:
              glob: .
            type: Directory
        requirements:
          EnvVarRequirement:
            envDef:
              AWS_ACCESS_KEY_ID: $(inputs.ADES_STAGEOUT_AWS_ACCESS_KEY_ID)
              AWS_SECRET_ACCESS_KEY: $(inputs.ADES_STAGEOUT_AWS_SECRET_ACCESS_KEY)
              AWS__AuthenticationRegion: $(inputs.ADES_STAGEOUT_AWS_REGION)
              AWS__Region: $(inputs.ADES_STAGEOUT_AWS_REGION)
              AWS__ServiceURL: $(inputs.ADES_STAGEOUT_AWS_SERVICEURL)
              PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
          InlineJavascriptRequirement: {}
          ResourceRequirement: {}
    on_stage:
      in:
        base_map: base_map
        bounding_box: bounding_box
        catalog_url: catalog_url
        copyrights: copyrights
        date: date
        input_raster: input_raster
        legend: legend
        logo: logo
        map_desc: map_desc
        subtitle: subtitle
        title: title
      out:
      - wf_outputs
      run: '#cpe-map-composition'
- arguments:
  - valueFrom: "${\n  var input_reference=[];\n  for (var i = 0; i < inputs.input_raster.length;\
      \ i++) {\n    input_reference.push(\"--input-raster\");\n    input_reference.push(inputs.input_raster[i]);\n\
      \  }\n  if (inputs.logo && inputs.logo.length > 0) {\n      for (var i = 0;\
      \ i < inputs.logo.length; i++) {\n      input_reference.push(\"--logo\");\n\
      \      input_reference.push(inputs.logo[i]);\n      }\n  }\n  if (inputs.legend\
      \ && inputs.legend.length > 0) {\n      for (var i = 0; i < inputs.legend.length;\
      \ i++) {\n      input_reference.push(\"--legend\");\n      input_reference.push(inputs.legend[i]);\n\
      \      }\n  }              \n  return input_reference;\n}\n"
  baseCommand: map-composition
  class: CommandLineTool
  hints:
    DockerRequirement:
      dockerPull: cr.terradue.com/cpe/map-composition:0.1.5
    NetworkAccess:
      networkAccess: true
  id: map-composition
  inputs:
    base_map:
      inputBinding:
        prefix: --basemap
      type: string
    bounding_box:
      inputBinding:
        prefix: --bbox
      type: string?
    catalog_url:
      inputBinding:
        prefix: --catalog-url
      type: string
    copyrights:
      inputBinding:
        prefix: --copyrights
      type: string
    date:
      inputBinding:
        prefix: --date
      type: string?
    input_raster:
      type: string[]
    legend:
      inputBinding:
        prefix: --legend
      type: string[]?
    logo:
      type: string[]?
    map_desc:
      inputBinding:
        prefix: --map-description
      type: string?
    subtitle:
      inputBinding:
        prefix: --subtitle
      type: string
    title:
      inputBinding:
        prefix: --title
      type: string
  outputs:
    results:
      outputBinding:
        glob: .
      type: Directory
  requirements:
    DockerRequirement:
      dockerPull: cr.terradue.com/cpe/map-composition:0.1.5
    EnvVarRequirement:
      envDef:
        GDAL_CACHEMAX: '4096'
        GDAL_NUM_THREADS: ALL_CPUS
        LD_LIBRARY_PATH: /srv/conda/envs/env_map_composition/lib:/usr/lib64:$LD_LIBRARY_PATH
        PATH: /srv/conda/envs/env_map_composition/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/srv/conda/envs/env_map_composition/bin
        PREFIX: /srv/conda/envs/env_map_composition
        QT_QPA_PLATFORM: offscreen
        XDG_RUNTIME_DIR: /tmp/runtime-root
    InlineJavascriptRequirement: {}
    ResourceRequirement:
      coresMax: 4
      ramMax: 8192
- class: Workflow
  doc: Map Composition Service
  id: cpe-map-composition
  inputs:
    base_map:
      type: string
    bounding_box:
      type: string?
    catalog_url:
      type: string
    copyrights:
      type: string
    date:
      type: string?
    input_raster:
      type: string[]
    legend:
      type: string[]?
    logo:
      type: string[]?
    map_desc:
      type: string?
    subtitle:
      type: string
    title:
      type: string
  label: map-composition
  outputs:
    wf_outputs:
      outputSource: mapComp_step/results
      type: Directory
  steps:
    mapComp_step:
      in:
        base_map: base_map
        bounding_box: bounding_box
        catalog_url: catalog_url
        copyrights: copyrights
        date: date
        input_raster: input_raster
        legend: legend
        logo: logo
        map_desc: map_desc
        subtitle: subtitle
        title: title
      out:
      - results
      run: '#map-composition'
$namespaces:
  s: https://schema.org/
cwlVersion: v1.2
s:softwareVersion: 0.1.5
@fabricebrito
Merge pull request #3 from xmichele/hatch
75c4dfc 
Dockerfile update + test
@fabricebrito
tidy pyproject.toml
3eb6d66
@fabricebrito
test ci package
111a2f7
@fabricebrito
CI add write
312df60
@fabricebrito
CI pypi if block
03e45f2
@fabricebrito
ci kebab-case
e1761ed
@xmichele
Update pyproject.toml with project.dependencies
29c60e8
@xmichele
Update __init__.py for bump to 0.12.3
8c06ba2
@fabricebrito
Merge pull request #4 from xmichele/hatch
3a4b6d2 
add project.dependencies to make a self contained package pip install ready
@fabricebrito
adds yq
307ec5d
@fabricebrito
fixes type using outputbinding type
9a02c39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@fabricebrito @xmichele