Merge branch 'master' into examples/mistral7b_vllm

SECURITY.md

@@ -20,22 +20,22 @@ TorchServe as much as possible relies on automated tools to do security scanning
     1. HTTP - `8080`, `8081`, `8082`
     2. gRPC - `7070`, `7071`
 
-    These ports are accessible to `localhost` by default.  The address can be configured by following the [guide](https://github.com/pytorch/serve/blob/master/docs/configuration.md#configure-torchserve-listening-address-and-port)
+    These ports are accessible to `localhost` by default.  The address can be configured by following the [guide](https://pytorch.org/serve/configuration.html#configure-torchserve-listening-address-and-port)
     TorchServe does not prevent users from configuring the address to be `0.0.0.0`. Please be aware of the security risks if you use `0.0.0.0`
-2. TorchServe's Docker image is configured to expose the ports `8080`, `8081`, `8082`, `7070`, `7071` to the host by default(https://github.com/pytorch/serve/blob/master/docker/Dockerfile). When these ports are mapped to the host, make sure to use `localhost` or a specific ip address.
+2. TorchServe's Docker image is configured to expose the ports `8080`, `8081`, `8082`, `7070`, `7071` to the host by [default](https://github.com/pytorch/serve/blob/master/docker/Dockerfile). When these ports are mapped to the host, make sure to use `localhost` or a specific ip address.
 
 3. Be sure to validate the authenticity of the `.mar` file being used with TorchServe.
-    1. A `.mar` file being downloaded from the internet from an untrusted source may have malicious code, compromising the integrity of your application
+    1. A `.mar` file being downloaded from the internet from an untrustworthy source may have malicious code, compromising the integrity of your application
     2. TorchServe executes arbitrary python code packaged in the `mar` file. Make sure that you've either audited that the code you're using is safe and/or is from a source that you trust
-4. By default TorchServe allows you to register models from all URLs. Make sure to set `allowed_urls` parameter in config.properties to restrict this. You can find more details in the [configuration guide](https://github.com/pytorch/serve/blob/master/docs/configuration.md#other-properties)
+4. By default TorchServe allows you to register models from all URLs. Make sure to set `allowed_urls` parameter in config.properties to restrict this. You can find more details in the [configuration guide](https://pytorch.org/serve/configuration.html#other-properties)
     - `use_env_allowed_urls=true` is required in config.properties to read `allowed_urls` from environment variable
 5. Enable SSL:
 
     TorchServe supports two ways to configure SSL:
     1. Using a keystore
     2. Using private-key/certificate files
 
-    You can find more details in the [configuration guide](https://github.com/pytorch/serve/blob/master/docs/configuration.md#enable-ssl)
+    You can find more details in the [configuration guide](https://pytorch.org/serve/configuration.html#enable-ssl)
 
 
 

docker/Dockerfile

@@ -64,7 +64,7 @@ RUN export USE_CUDA=1
 
 ARG CUDA_VERSION=""
 
-RUN git clone --depth 1 https://github.com/pytorch/serve.git
+RUN git clone --depth 1 --recursive https://github.com/pytorch/serve.git
 
 WORKDIR "serve"
 
@@ -181,3 +181,61 @@ ENV TS_RUN_IN_DOCKER True
 
 WORKDIR /home/serve
 CMD ["python", "test/regression_tests.py"]
+
+#Final image for developer Docker image
+FROM ${BASE_IMAGE} as dev-image
+# Re-state ARG PYTHON_VERSION to make it active in this build-stage (uses default define at the top)
+ARG PYTHON_VERSION
+ARG BRANCH_NAME
+ARG BUILD_WITH_IPEX
+ARG IPEX_VERSION=1.11.0
+ARG IPEX_URL=https://software.intel.com/ipex-whl-stable
+ENV PYTHONUNBUFFERED TRUE
+RUN --mount=type=cache,target=/var/cache/apt \
+    apt-get update && \
+    apt-get upgrade -y && \
+    apt-get install software-properties-common -y && \
+    add-apt-repository -y ppa:deadsnakes/ppa && \
+    DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \
+    fakeroot \
+    ca-certificates \
+    dpkg-dev \
+    sudo \
+    g++ \
+    git \
+    python$PYTHON_VERSION \
+    python$PYTHON_VERSION-dev \
+    python3-distutils \
+    python$PYTHON_VERSION-venv \
+    # using openjdk-17-jdk due to circular dependency(ca-certificates) bug in openjdk-17-jre-headless debian package
+    # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1009905
+    openjdk-17-jdk \
+    build-essential \
+    curl \
+    vim \
+    numactl \
+    && if [ "$BUILD_WITH_IPEX" = "true" ]; then apt-get update && apt-get install -y libjemalloc-dev libgoogle-perftools-dev libomp-dev && ln -s /usr/lib/x86_64-linux-gnu/libjemalloc.so /usr/lib/libjemalloc.so && ln -s /usr/lib/x86_64-linux-gnu/libtcmalloc.so /usr/lib/libtcmalloc.so && ln -s /usr/lib/x86_64-linux-gnu/libiomp5.so /usr/lib/libiomp5.so; fi \
+    && rm -rf /var/lib/apt/lists/*
+RUN git clone --recursive https://github.com/pytorch/serve.git \
+    && cd serve \
+    && git checkout ${BRANCH_NAME}
+COPY --from=compile-image /home/venv /home/venv
+ENV PATH="/home/venv/bin:$PATH"
+WORKDIR "serve"
+RUN python -m pip install -U pip setuptools \
+    && python -m pip install --no-cache-dir -r requirements/developer.txt \
+    && python ts_scripts/install_from_src.py \
+    && useradd -m model-server \
+    && mkdir -p /home/model-server/tmp \
+    && cp docker/dockerd-entrypoint.sh /usr/local/bin/dockerd-entrypoint.sh \
+    && chmod +x /usr/local/bin/dockerd-entrypoint.sh \
+    && chown -R model-server /home/model-server \
+    && cp docker/config.properties /home/model-server/config.properties \
+    && mkdir /home/model-server/model-store && chown -R model-server /home/model-server/model-store \
+    && chown -R model-server /home/venv
+EXPOSE 8080 8081 8082 7070 7071
+USER model-server
+WORKDIR /home/model-server
+ENV TEMP=/home/model-server/tmp
+ENTRYPOINT ["/usr/local/bin/dockerd-entrypoint.sh"]
+CMD ["serve"]

docker/README.md

@@ -1,6 +1,8 @@
 ### Deprecation notice:
 [Dockerfile.neuron.dev](https://github.com/pytorch/serve/blob/master/docker/Dockerfile.neuron.dev) has been deprecated. Please refer to [deep learning containers](https://github.com/aws/deep-learning-containers/blob/master/available_images.md) repository for neuron torchserve containers.
 
+[Dockerfile.dev](https://github.com/sachanub/serve/blob/master/docker/Dockerfile.dev) has been deprecated. Please refer to [Dockerfile](https://github.com/sachanub/serve/blob/master/docker/Dockerfile) for dev torchserve containers.
+
 ## Contents of this Document
 
 * [Prerequisites](#prerequisites)
@@ -28,19 +30,18 @@ cd serve/docker
 
 # Create TorchServe docker image
 
-Use `build_image.sh` script to build the docker images. The script builds the `production`, `dev` , `ci` and `codebuild` docker images.
+Use `build_image.sh` script to build the docker images. The script builds the `production`, `dev` and `ci` docker images.
 | Parameter | Description |
 |------|------|
 |-h, --help|Show script help|
 |-b, --branch_name|Specify a branch name to use. Default: master |
 |-g, --gpu|Build image with GPU based ubuntu base image|
 |-bi, --baseimage specify base docker image. Example: nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu20.04|
-|-bt, --buildtype|Which type of docker image to build. Can be one of : production, dev, ci, codebuild|
+|-bt, --buildtype|Which type of docker image to build. Can be one of : production, dev, ci|
 |-t, --tag|Tag name for image. If not specified, script uses torchserve default tag names.|
 |-cv, --cudaversion| Specify to cuda version to use. Supported values `cu92`, `cu101`, `cu102`, `cu111`, `cu113`, `cu116`, `cu117`, `cu118`. `cu121`, Default `cu121`|
 |-ipex, --build-with-ipex| Specify to build with intel_extension_for_pytorch. If not specified, script builds without intel_extension_for_pytorch.|
 |-n, --nightly| Specify to build with TorchServe nightly.|
-|--codebuild| Set if you need [AWS CodeBuild](https://aws.amazon.com/codebuild/)|
 |-py, --pythonversion| Specify the python version to use. Supported values `3.8`, `3.9`, `3.10`, `3.11`. Default `3.9`|
 
 
@@ -146,44 +147,6 @@ Creates a docker image with `torchserve` and `torch-model-archiver` installed fr
 ./build_image.sh -bt dev -ipex -t torchserve-ipex:1.0
 ```
 
-**CODEBUILD ENVIRONMENT IMAGES**
-
-Creates a docker image for codebuild environment
-
-- For creating CPU based image :
-
-```bash
-./build_image.sh -bt codebuild
-```
-
-- For creating GPU based image with cuda version 11.1:
-
-```bash
-./build_image.sh -bt codebuild -g -cv cu113
-```
-
-```bash
-./build_image.sh -bt codebuild -g -cv cu111
-```
-
- - For creating GPU based image with cuda version 10.1:
-
-```bash
-./build_image.sh -bt codebuild -g -cv cu101
-```
-
- - For creating GPU based image with cuda version 9.2:
-
-```bash
-./build_image.sh -bt codebuild -g -cv cu92
-```
-
- - For creating image with a custom tag:
-
-```bash
-./build_image.sh -bt codebuild -t torchserve-codebuild:1.0
-```
-
 
 ## Start a container with a TorchServe image
 

docker/build_image.sh

@@ -24,7 +24,7 @@ do
           echo "-b, --branch_name=BRANCH_NAME specify a branch_name to use"
           echo "-g, --gpu specify to use gpu"
           echo "-bi, --baseimage specify base docker image. Example: nvidia/cuda:11.7.0-cudnn8-runtime-ubuntu20.04 "
-          echo "-bt, --buildtype specify to created image for codebuild. Possible values: production, dev, codebuild."
+          echo "-bt, --buildtype specify for type of created image. Possible values: production, dev, ci."
           echo "-cv, --cudaversion specify to cuda version to use"
           echo "-t, --tag specify tag name for docker image"
           echo "-lf, --use-local-serve-folder specify this option for the benchmark image if the current 'serve' folder should be used during automated benchmarks"
@@ -142,11 +142,6 @@ then
   DOCKER_TAG="pytorch/torchserve:dev-$MACHINE"
 fi
 
-if [ "${BUILD_TYPE}" == "codebuild" ] && ! $USE_CUSTOM_TAG ;
-then
-  DOCKER_TAG="pytorch/torchserve:codebuild-$MACHINE"
-fi
-
 if [ "$USE_CUSTOM_TAG" = true ]
 then
   DOCKER_TAG=${CUSTOM_TAG}
@@ -164,9 +159,6 @@ then
 elif [ "${BUILD_TYPE}" == "ci" ]
 then
   DOCKER_BUILDKIT=1 docker build --file Dockerfile --build-arg BASE_IMAGE="${BASE_IMAGE}" --build-arg CUDA_VERSION="${CUDA_VERSION}"  --build-arg PYTHON_VERSION="${PYTHON_VERSION}" --build-arg BUILD_NIGHTLY="${BUILD_NIGHTLY}" --build-arg BRANCH_NAME="${BRANCH_NAME}"  -t "${DOCKER_TAG}" --target ci-image  .
-elif [ "${BUILD_TYPE}" == "benchmark" ]
-then
-  DOCKER_BUILDKIT=1 docker build --pull --no-cache --file Dockerfile.benchmark --build-arg USE_LOCAL_SERVE_FOLDER=$USE_LOCAL_SERVE_FOLDER --build-arg BASE_IMAGE="${BASE_IMAGE}" --build-arg BRANCH_NAME="${BRANCH_NAME}" --build-arg CUDA_VERSION="${CUDA_VERSION}" --build-arg MACHINE_TYPE="${MACHINE}" --build-arg PYTHON_VERSION="${PYTHON_VERSION}" -t "${DOCKER_TAG}" .
 else
-  DOCKER_BUILDKIT=1 docker build --pull --no-cache --file Dockerfile.dev -t "${DOCKER_TAG}" --build-arg BUILD_TYPE="${BUILD_TYPE}" --build-arg BASE_IMAGE=$BASE_IMAGE --build-arg BRANCH_NAME="${BRANCH_NAME}" --build-arg CUDA_VERSION="${CUDA_VERSION}" --build-arg MACHINE_TYPE="${MACHINE}" --build-arg BUILD_WITH_IPEX="${BUILD_WITH_IPEX}" --build-arg PYTHON_VERSION="${PYTHON_VERSION}" .
-fi
+  DOCKER_BUILDKIT=1 docker build --file Dockerfile --build-arg BASE_IMAGE="${BASE_IMAGE}" --build-arg CUDA_VERSION="${CUDA_VERSION}"  --build-arg PYTHON_VERSION="${PYTHON_VERSION}" --build-arg BUILD_NIGHTLY="${BUILD_NIGHTLY}" --build-arg BRANCH_NAME="${BRANCH_NAME}" --build-arg BUILD_WITH_IPEX="${BUILD_WITH_IPEX}"  -t "${DOCKER_TAG}" --target dev-image  .
+fi

docs/Security.md

@@ -0,0 +1,45 @@
+# Security Policy
+
+## Supported Versions
+
+| Version | Supported          |
+|---------| ------------------ |
+| 0.9.0   | :white_check_mark: |
+
+
+## How we do security
+
+TorchServe as much as possible relies on automated tools to do security scanning, in particular we support
+1. Dependency Analysis: Using Dependabot
+2. Docker Scanning: Using Snyk
+3. Code Analysis: Using CodeQL
+
+## Important Security Guidelines
+
+1. TorchServe listens on the following ports
+    1. HTTP - `8080`, `8081`, `8082`
+    2. gRPC - `7070`, `7071`
+
+    These ports are accessible to `localhost` by default.  The address can be configured by following the [guide](https://pytorch.org/serve/configuration.html#configure-torchserve-listening-address-and-port)
+    TorchServe does not prevent users from configuring the address to be `0.0.0.0`. Please be aware of the security risks if you use `0.0.0.0`
+2. TorchServe's Docker image is configured to expose the ports `8080`, `8081`, `8082`, `7070`, `7071` to the host by [default](https://github.com/pytorch/serve/blob/master/docker/Dockerfile). When these ports are mapped to the host, make sure to use `localhost` or a specific ip address.
+
+3. Be sure to validate the authenticity of the `.mar` file being used with TorchServe.
+    1. A `.mar` file being downloaded from the internet from an untrustworthy source may have malicious code, compromising the integrity of your application
+    2. TorchServe executes arbitrary python code packaged in the `mar` file. Make sure that you've either audited that the code you're using is safe and/or is from a source that you trust
+4. By default TorchServe allows you to register models from all URLs. Make sure to set `allowed_urls` parameter in config.properties to restrict this. You can find more details in the [configuration guide](https://pytorch.org/serve/configuration.html#other-properties)
+    - `use_env_allowed_urls=true` is required in config.properties to read `allowed_urls` from environment variable
+5. Enable SSL:
+
+    TorchServe supports two ways to configure SSL:
+    1. Using a keystore
+    2. Using private-key/certificate files
+
+    You can find more details in the [configuration guide](https://pytorch.org/serve/configuration.html#enable-ssl)
+
+
+
+
+## Reporting a Vulnerability
+
+If you find a serious vulnerability please report it to [email protected] and [email protected]

docs/contents.rst

@@ -23,6 +23,7 @@
   use_cases
   workflows
   large_model_inference
+  Security
   FAQs
 
 .. toctree::

docs/index.rst

@@ -84,6 +84,13 @@ What's going on in TorchServe?
    :link: Troubleshooting.html
    :tags: Troubleshooting,Performance
 
+.. customcarditem::
+   :header: TorchServe Security Policy
+   :card_description: Security Policy
+   :image: https://user-images.githubusercontent.com/880376/83180095-c44cc600-a0d7-11ea-97c1-23abb4cdbe4d.jpg
+   :link: security.html
+   :tags: Security 
+
 .. customcarditem::
    :header: FAQs
    :card_description: Various frequently asked questions.

docs/sphinx/Makefile

@@ -25,4 +25,5 @@ docset: html
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %: Makefile
-	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+	cp ../../SECURITY.md ../Security.md
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

examples/Huggingface_Transformers/README.md

@@ -106,6 +106,9 @@ torchserve --start --model-store model_store --models my_tc=BERTSeqClassificatio
 ### Run an inference
 
 To run an inference: `curl -X POST http://127.0.0.1:8080/predictions/my_tc -T Seq_classification_artifacts/sample_text_captum_input.txt`
+
+The response should be a "Not Accepted" classification.
+
 To get an explanation: `curl -X POST http://127.0.0.1:8080/explanations/my_tc -T Seq_classification_artifacts/sample_text_captum_input.txt`
 
 ## Token Classification

requirements/developer.txt

@@ -14,7 +14,7 @@ pre-commit==3.3.2
 twine==4.0.2
 mypy==1.3.0
 torchpippy==0.1.1
-intel_extension_for_pytorch==2.0.100; sys_platform != 'win32' and sys_platform != 'darwin'
+intel_extension_for_pytorch==2.1.0; sys_platform != 'win32' and sys_platform != 'darwin'
 onnxruntime==1.15.0
 googleapis-common-protos
 onnx==1.14.1