Why is an IAM role required for fully local use?

[johann-petrak]

I tried to test code for deploying a PyTorchModel on SageMaker fully locally doing something like:

I have also created the file ~/.sagemaker/config.yaml as described in https://sagemaker.readthedocs.io/en/stable/overview.html#local-mode

sess = LocalSession()
sess.config = {'local': {'local_code': True}}

model = PyTorchModel(
    model_data="model1.tar.gz",
    role=SOMEROLE, 
    framework_version="1.8.1",  
    py_version="py3",
    entry_point="inference.py",
    )
model.sagemaker_session = sess

predictor = model.deploy(
    instance_type="local", 
    initial_instance_count=1,
    deserializer=JSONDeserializer(),
    serializer=JSONSerializer(),
)

This fails, unless SOMEROLE is a proper IAM Role, but why is this needed? I do not see why the code would have to connect to AWS and even use my credentials at all if I just run EVERYTHING locally.

When I inspect the docker-compose config file that is created I notice that the SAGEMAKER_SUBMIT_DIRECTORY variable is set to a S3 bucket - why? I just use my local file system!

When running fully locally like this, which actions are actually using AWS access and how can this be prevented??

[icywang86rui]

AWS credentials are required so you can download the inference image used in LocalMode. - https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/local/image.py#L150

As for SAGEMAKER_SUBMIT_DIRECTORY, the model shouldn't upload the code to S3 in localmode - https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/model.py#L1114. I will do more digging.

[johann-petrak]

Thanks for that explanation!

I pulled the image already beforehand into my local Docker, but I can see that this would be needed in case the image is not there.

If the SAGEMAKER_SUBMIT_DIRECTORY variable is just ignored in that case that is fine too, I just was not sure to understand everything that is happening under the hood.

So if in effect everything is running locally and no AWS components/API calls are used, and the role is only needed in case the image needs to get pulled, that is just fine then.

Thanks!

[johann-petrak]

@icywang86rui I just tried the same code again, but with a much larger mordel, completely local again, but I got the exception shown below which indicates the there was an attempt to upload the model to S3 although that would be completely useless and unnecessary (probably the deepest stack trace I ever got).

I double checked my S3 console and found that there is a bucked I do not remember creating with the name schema
sagemaker-us-west-2-999999999999 which contains a directory pytorch-inference-2021-06-14-14-51-56-049/ for every
local deployment that succeeded.

The other issue of course is, if I would instead not want to deploy the image locally, but instead use an AWS instance by specifying the instance type with the instance_type parameter, why would I then get that error?

Finally, even when running on SageMaker, why would every deploy invocation save the model into a separate new directory, even when the model does not change? A simple approach of fingerprinting the tar.gz file could prevent this.

---------------------------------------------------------------------------
ClientError                               Traceback (most recent call last)
~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/boto3/s3/transfer.py in upload_file(self, filename, bucket, key, callback, extra_args)
    278         try:
--> 279             future.result()
    280         # If a client error was raised, add the backwards compatibility layer

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/futures.py in result(self)
    105             # out of this and propogate the exception.
--> 106             return self._coordinator.result()
    107         except KeyboardInterrupt as e:

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/futures.py in result(self)
    264         if self._exception:
--> 265             raise self._exception
    266         return self._result

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/httpsession.py in send(self, request)
    331                 decode_content=False,
--> 332                 chunked=self._chunked(request.headers),
    333             )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    705                 headers=headers,
--> 706                 chunked=chunked,
    707             )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    393             else:
--> 394                 conn.request(method, url, **httplib_request_kw)
    395 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connection.py in request(self, method, url, body, headers)
    233             headers["User-Agent"] = _get_default_user_agent()
--> 234         super(HTTPConnection, self).request(method, url, body=body, headers=headers)
    235 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked)
   1286         """Send a complete request to the server."""
-> 1287         self._send_request(method, url, body, headers, encode_chunked)
   1288 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_request(self, method, url, body, headers, *args, **kwargs)
     92         rval = super(AWSConnection, self)._send_request(
---> 93             method, url, body, headers, *args, **kwargs)
     94         self._expect_header_set = False

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1332             body = _encode(body, 'body')
-> 1333         self.endheaders(body, encode_chunked=encode_chunked)
   1334 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked)
   1281             raise CannotSendHeader()
-> 1282         self._send_output(message_body, encode_chunked=encode_chunked)
   1283 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_output(self, message_body, *args, **kwargs)
    126             if urllib3.util.wait_for_read(self.sock, 1):
--> 127                 self._handle_expect_response(message_body)
    128                 return

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _handle_expect_response(self, message_body)
    169                              "now sending request body.")
--> 170                 self._send_message_body(message_body)
    171             elif len(parts) == 3 and parts[0].startswith(b'HTTP/'):

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_message_body(self, message_body)
    196         if message_body is not None:
--> 197             self.send(message_body)
    198 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in send(self, str)
    203             return
--> 204         return super(AWSConnection, self).send(str)
    205 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in send(self, data)
    993             while 1:
--> 994                 datablock = data.read(blocksize)
    995                 if not datablock:

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/utils.py in read(self, amount)
    482             amount_to_read = min(amount_left, amount)
--> 483         data = self._fileobj.read(amount_to_read)
    484         self._amount_read += len(data)

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/upload.py in read(self, amount)
     84         if self._transfer_coordinator.exception:
---> 85             raise self._transfer_coordinator.exception
     86         return self._fileobj.read(amount)

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/httpsession.py in send(self, request)
    331                 decode_content=False,
--> 332                 chunked=self._chunked(request.headers),
    333             )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    705                 headers=headers,
--> 706                 chunked=chunked,
    707             )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    393             else:
--> 394                 conn.request(method, url, **httplib_request_kw)
    395 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connection.py in request(self, method, url, body, headers)
    233             headers["User-Agent"] = _get_default_user_agent()
--> 234         super(HTTPConnection, self).request(method, url, body=body, headers=headers)
    235 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked)
   1286         """Send a complete request to the server."""
-> 1287         self._send_request(method, url, body, headers, encode_chunked)
   1288 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_request(self, method, url, body, headers, *args, **kwargs)
     92         rval = super(AWSConnection, self)._send_request(
---> 93             method, url, body, headers, *args, **kwargs)
     94         self._expect_header_set = False

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1332             body = _encode(body, 'body')
-> 1333         self.endheaders(body, encode_chunked=encode_chunked)
   1334 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked)
   1281             raise CannotSendHeader()
-> 1282         self._send_output(message_body, encode_chunked=encode_chunked)
   1283 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_output(self, message_body, *args, **kwargs)
    126             if urllib3.util.wait_for_read(self.sock, 1):
--> 127                 self._handle_expect_response(message_body)
    128                 return

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _handle_expect_response(self, message_body)
    169                              "now sending request body.")
--> 170                 self._send_message_body(message_body)
    171             elif len(parts) == 3 and parts[0].startswith(b'HTTP/'):

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_message_body(self, message_body)
    196         if message_body is not None:
--> 197             self.send(message_body)
    198 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in send(self, str)
    203             return
--> 204         return super(AWSConnection, self).send(str)
    205 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in send(self, data)
    993             while 1:
--> 994                 datablock = data.read(blocksize)
    995                 if not datablock:

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/utils.py in read(self, amount)
    482             amount_to_read = min(amount_left, amount)
--> 483         data = self._fileobj.read(amount_to_read)
    484         self._amount_read += len(data)

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/upload.py in read(self, amount)
     84         if self._transfer_coordinator.exception:
---> 85             raise self._transfer_coordinator.exception
     86         return self._fileobj.read(amount)

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/httpsession.py in send(self, request)
    331                 decode_content=False,
--> 332                 chunked=self._chunked(request.headers),
    333             )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    705                 headers=headers,
--> 706                 chunked=chunked,
    707             )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    393             else:
--> 394                 conn.request(method, url, **httplib_request_kw)
    395 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connection.py in request(self, method, url, body, headers)
    233             headers["User-Agent"] = _get_default_user_agent()
--> 234         super(HTTPConnection, self).request(method, url, body=body, headers=headers)
    235 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked)
   1286         """Send a complete request to the server."""
-> 1287         self._send_request(method, url, body, headers, encode_chunked)
   1288 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_request(self, method, url, body, headers, *args, **kwargs)
     92         rval = super(AWSConnection, self)._send_request(
---> 93             method, url, body, headers, *args, **kwargs)
     94         self._expect_header_set = False

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1332             body = _encode(body, 'body')
-> 1333         self.endheaders(body, encode_chunked=encode_chunked)
   1334 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked)
   1281             raise CannotSendHeader()
-> 1282         self._send_output(message_body, encode_chunked=encode_chunked)
   1283 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_output(self, message_body, *args, **kwargs)
    126             if urllib3.util.wait_for_read(self.sock, 1):
--> 127                 self._handle_expect_response(message_body)
    128                 return

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _handle_expect_response(self, message_body)
    169                              "now sending request body.")
--> 170                 self._send_message_body(message_body)
    171             elif len(parts) == 3 and parts[0].startswith(b'HTTP/'):

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_message_body(self, message_body)
    196         if message_body is not None:
--> 197             self.send(message_body)
    198 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in send(self, str)
    203             return
--> 204         return super(AWSConnection, self).send(str)
    205 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in send(self, data)
    993             while 1:
--> 994                 datablock = data.read(blocksize)
    995                 if not datablock:

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/utils.py in read(self, amount)
    482             amount_to_read = min(amount_left, amount)
--> 483         data = self._fileobj.read(amount_to_read)
    484         self._amount_read += len(data)

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/upload.py in read(self, amount)
     84         if self._transfer_coordinator.exception:
---> 85             raise self._transfer_coordinator.exception
     86         return self._fileobj.read(amount)

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/httpsession.py in send(self, request)
    331                 decode_content=False,
--> 332                 chunked=self._chunked(request.headers),
    333             )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    705                 headers=headers,
--> 706                 chunked=chunked,
    707             )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    393             else:
--> 394                 conn.request(method, url, **httplib_request_kw)
    395 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connection.py in request(self, method, url, body, headers)
    233             headers["User-Agent"] = _get_default_user_agent()
--> 234         super(HTTPConnection, self).request(method, url, body=body, headers=headers)
    235 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked)
   1286         """Send a complete request to the server."""
-> 1287         self._send_request(method, url, body, headers, encode_chunked)
   1288 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_request(self, method, url, body, headers, *args, **kwargs)
     92         rval = super(AWSConnection, self)._send_request(
---> 93             method, url, body, headers, *args, **kwargs)
     94         self._expect_header_set = False

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1332             body = _encode(body, 'body')
-> 1333         self.endheaders(body, encode_chunked=encode_chunked)
   1334 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked)
   1281             raise CannotSendHeader()
-> 1282         self._send_output(message_body, encode_chunked=encode_chunked)
   1283 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_output(self, message_body, *args, **kwargs)
    126             if urllib3.util.wait_for_read(self.sock, 1):
--> 127                 self._handle_expect_response(message_body)
    128                 return

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _handle_expect_response(self, message_body)
    169                              "now sending request body.")
--> 170                 self._send_message_body(message_body)
    171             elif len(parts) == 3 and parts[0].startswith(b'HTTP/'):

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_message_body(self, message_body)
    196         if message_body is not None:
--> 197             self.send(message_body)
    198 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in send(self, str)
    203             return
--> 204         return super(AWSConnection, self).send(str)
    205 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in send(self, data)
    993             while 1:
--> 994                 datablock = data.read(blocksize)
    995                 if not datablock:

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/utils.py in read(self, amount)
    482             amount_to_read = min(amount_left, amount)
--> 483         data = self._fileobj.read(amount_to_read)
    484         self._amount_read += len(data)

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/upload.py in read(self, amount)
     84         if self._transfer_coordinator.exception:
---> 85             raise self._transfer_coordinator.exception
     86         return self._fileobj.read(amount)

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/httpsession.py in send(self, request)
    331                 decode_content=False,
--> 332                 chunked=self._chunked(request.headers),
    333             )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    705                 headers=headers,
--> 706                 chunked=chunked,
    707             )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    393             else:
--> 394                 conn.request(method, url, **httplib_request_kw)
    395 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connection.py in request(self, method, url, body, headers)
    233             headers["User-Agent"] = _get_default_user_agent()
--> 234         super(HTTPConnection, self).request(method, url, body=body, headers=headers)
    235 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked)
   1286         """Send a complete request to the server."""
-> 1287         self._send_request(method, url, body, headers, encode_chunked)
   1288 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_request(self, method, url, body, headers, *args, **kwargs)
     92         rval = super(AWSConnection, self)._send_request(
---> 93             method, url, body, headers, *args, **kwargs)
     94         self._expect_header_set = False

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1332             body = _encode(body, 'body')
-> 1333         self.endheaders(body, encode_chunked=encode_chunked)
   1334 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked)
   1281             raise CannotSendHeader()
-> 1282         self._send_output(message_body, encode_chunked=encode_chunked)
   1283 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_output(self, message_body, *args, **kwargs)
    126             if urllib3.util.wait_for_read(self.sock, 1):
--> 127                 self._handle_expect_response(message_body)
    128                 return

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _handle_expect_response(self, message_body)
    169                              "now sending request body.")
--> 170                 self._send_message_body(message_body)
    171             elif len(parts) == 3 and parts[0].startswith(b'HTTP/'):

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_message_body(self, message_body)
    196         if message_body is not None:
--> 197             self.send(message_body)
    198 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in send(self, str)
    203             return
--> 204         return super(AWSConnection, self).send(str)
    205 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in send(self, data)
    993             while 1:
--> 994                 datablock = data.read(blocksize)
    995                 if not datablock:

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/utils.py in read(self, amount)
    482             amount_to_read = min(amount_left, amount)
--> 483         data = self._fileobj.read(amount_to_read)
    484         self._amount_read += len(data)

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/upload.py in read(self, amount)
     84         if self._transfer_coordinator.exception:
---> 85             raise self._transfer_coordinator.exception
     86         return self._fileobj.read(amount)

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/httpsession.py in send(self, request)
    331                 decode_content=False,
--> 332                 chunked=self._chunked(request.headers),
    333             )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    705                 headers=headers,
--> 706                 chunked=chunked,
    707             )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    393             else:
--> 394                 conn.request(method, url, **httplib_request_kw)
    395 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connection.py in request(self, method, url, body, headers)
    233             headers["User-Agent"] = _get_default_user_agent()
--> 234         super(HTTPConnection, self).request(method, url, body=body, headers=headers)
    235 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked)
   1286         """Send a complete request to the server."""
-> 1287         self._send_request(method, url, body, headers, encode_chunked)
   1288 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_request(self, method, url, body, headers, *args, **kwargs)
     92         rval = super(AWSConnection, self)._send_request(
---> 93             method, url, body, headers, *args, **kwargs)
     94         self._expect_header_set = False

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1332             body = _encode(body, 'body')
-> 1333         self.endheaders(body, encode_chunked=encode_chunked)
   1334 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked)
   1281             raise CannotSendHeader()
-> 1282         self._send_output(message_body, encode_chunked=encode_chunked)
   1283 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_output(self, message_body, *args, **kwargs)
    126             if urllib3.util.wait_for_read(self.sock, 1):
--> 127                 self._handle_expect_response(message_body)
    128                 return

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _handle_expect_response(self, message_body)
    169                              "now sending request body.")
--> 170                 self._send_message_body(message_body)
    171             elif len(parts) == 3 and parts[0].startswith(b'HTTP/'):

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_message_body(self, message_body)
    196         if message_body is not None:
--> 197             self.send(message_body)
    198 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in send(self, str)
    203             return
--> 204         return super(AWSConnection, self).send(str)
    205 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in send(self, data)
    993             while 1:
--> 994                 datablock = data.read(blocksize)
    995                 if not datablock:

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/utils.py in read(self, amount)
    482             amount_to_read = min(amount_left, amount)
--> 483         data = self._fileobj.read(amount_to_read)
    484         self._amount_read += len(data)

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/upload.py in read(self, amount)
     84         if self._transfer_coordinator.exception:
---> 85             raise self._transfer_coordinator.exception
     86         return self._fileobj.read(amount)

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/httpsession.py in send(self, request)
    331                 decode_content=False,
--> 332                 chunked=self._chunked(request.headers),
    333             )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    705                 headers=headers,
--> 706                 chunked=chunked,
    707             )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    393             else:
--> 394                 conn.request(method, url, **httplib_request_kw)
    395 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connection.py in request(self, method, url, body, headers)
    233             headers["User-Agent"] = _get_default_user_agent()
--> 234         super(HTTPConnection, self).request(method, url, body=body, headers=headers)
    235 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked)
   1286         """Send a complete request to the server."""
-> 1287         self._send_request(method, url, body, headers, encode_chunked)
   1288 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_request(self, method, url, body, headers, *args, **kwargs)
     92         rval = super(AWSConnection, self)._send_request(
---> 93             method, url, body, headers, *args, **kwargs)
     94         self._expect_header_set = False

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1332             body = _encode(body, 'body')
-> 1333         self.endheaders(body, encode_chunked=encode_chunked)
   1334 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked)
   1281             raise CannotSendHeader()
-> 1282         self._send_output(message_body, encode_chunked=encode_chunked)
   1283 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_output(self, message_body, *args, **kwargs)
    126             if urllib3.util.wait_for_read(self.sock, 1):
--> 127                 self._handle_expect_response(message_body)
    128                 return

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _handle_expect_response(self, message_body)
    169                              "now sending request body.")
--> 170                 self._send_message_body(message_body)
    171             elif len(parts) == 3 and parts[0].startswith(b'HTTP/'):

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_message_body(self, message_body)
    196         if message_body is not None:
--> 197             self.send(message_body)
    198 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in send(self, str)
    203             return
--> 204         return super(AWSConnection, self).send(str)
    205 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in send(self, data)
    993             while 1:
--> 994                 datablock = data.read(blocksize)
    995                 if not datablock:

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/utils.py in read(self, amount)
    482             amount_to_read = min(amount_left, amount)
--> 483         data = self._fileobj.read(amount_to_read)
    484         self._amount_read += len(data)

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/upload.py in read(self, amount)
     84         if self._transfer_coordinator.exception:
---> 85             raise self._transfer_coordinator.exception
     86         return self._fileobj.read(amount)

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/httpsession.py in send(self, request)
    331                 decode_content=False,
--> 332                 chunked=self._chunked(request.headers),
    333             )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    705                 headers=headers,
--> 706                 chunked=chunked,
    707             )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    393             else:
--> 394                 conn.request(method, url, **httplib_request_kw)
    395 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connection.py in request(self, method, url, body, headers)
    233             headers["User-Agent"] = _get_default_user_agent()
--> 234         super(HTTPConnection, self).request(method, url, body=body, headers=headers)
    235 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked)
   1286         """Send a complete request to the server."""
-> 1287         self._send_request(method, url, body, headers, encode_chunked)
   1288 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_request(self, method, url, body, headers, *args, **kwargs)
     92         rval = super(AWSConnection, self)._send_request(
---> 93             method, url, body, headers, *args, **kwargs)
     94         self._expect_header_set = False

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1332             body = _encode(body, 'body')
-> 1333         self.endheaders(body, encode_chunked=encode_chunked)
   1334 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked)
   1281             raise CannotSendHeader()
-> 1282         self._send_output(message_body, encode_chunked=encode_chunked)
   1283 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_output(self, message_body, *args, **kwargs)
    126             if urllib3.util.wait_for_read(self.sock, 1):
--> 127                 self._handle_expect_response(message_body)
    128                 return

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _handle_expect_response(self, message_body)
    169                              "now sending request body.")
--> 170                 self._send_message_body(message_body)
    171             elif len(parts) == 3 and parts[0].startswith(b'HTTP/'):

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_message_body(self, message_body)
    196         if message_body is not None:
--> 197             self.send(message_body)
    198 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in send(self, str)
    203             return
--> 204         return super(AWSConnection, self).send(str)
    205 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in send(self, data)
    993             while 1:
--> 994                 datablock = data.read(blocksize)
    995                 if not datablock:

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/utils.py in read(self, amount)
    482             amount_to_read = min(amount_left, amount)
--> 483         data = self._fileobj.read(amount_to_read)
    484         self._amount_read += len(data)

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/upload.py in read(self, amount)
     84         if self._transfer_coordinator.exception:
---> 85             raise self._transfer_coordinator.exception
     86         return self._fileobj.read(amount)

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/httpsession.py in send(self, request)
    331                 decode_content=False,
--> 332                 chunked=self._chunked(request.headers),
    333             )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    705                 headers=headers,
--> 706                 chunked=chunked,
    707             )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    393             else:
--> 394                 conn.request(method, url, **httplib_request_kw)
    395 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/urllib3/connection.py in request(self, method, url, body, headers)
    233             headers["User-Agent"] = _get_default_user_agent()
--> 234         super(HTTPConnection, self).request(method, url, body=body, headers=headers)
    235 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in request(self, method, url, body, headers, encode_chunked)
   1286         """Send a complete request to the server."""
-> 1287         self._send_request(method, url, body, headers, encode_chunked)
   1288 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_request(self, method, url, body, headers, *args, **kwargs)
     92         rval = super(AWSConnection, self)._send_request(
---> 93             method, url, body, headers, *args, **kwargs)
     94         self._expect_header_set = False

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in _send_request(self, method, url, body, headers, encode_chunked)
   1332             body = _encode(body, 'body')
-> 1333         self.endheaders(body, encode_chunked=encode_chunked)
   1334 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in endheaders(self, message_body, encode_chunked)
   1281             raise CannotSendHeader()
-> 1282         self._send_output(message_body, encode_chunked=encode_chunked)
   1283 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_output(self, message_body, *args, **kwargs)
    126             if urllib3.util.wait_for_read(self.sock, 1):
--> 127                 self._handle_expect_response(message_body)
    128                 return

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _handle_expect_response(self, message_body)
    169                              "now sending request body.")
--> 170                 self._send_message_body(message_body)
    171             elif len(parts) == 3 and parts[0].startswith(b'HTTP/'):

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in _send_message_body(self, message_body)
    196         if message_body is not None:
--> 197             self.send(message_body)
    198 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/awsrequest.py in send(self, str)
    203             return
--> 204         return super(AWSConnection, self).send(str)
    205 

~/software/anaconda/envs/aws-try01/lib/python3.6/http/client.py in send(self, data)
    993             while 1:
--> 994                 datablock = data.read(blocksize)
    995                 if not datablock:

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/utils.py in read(self, amount)
    482             amount_to_read = min(amount_left, amount)
--> 483         data = self._fileobj.read(amount_to_read)
    484         self._amount_read += len(data)

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/upload.py in read(self, amount)
     84         if self._transfer_coordinator.exception:
---> 85             raise self._transfer_coordinator.exception
     86         return self._fileobj.read(amount)

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/tasks.py in __call__(self)
    125             if not self._transfer_coordinator.done():
--> 126                 return self._execute_main(kwargs)
    127         except Exception as e:

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/tasks.py in _execute_main(self, kwargs)
    149 
--> 150         return_value = self._main(**kwargs)
    151         # If the task is the final task, then set the TransferFuture's

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/s3transfer/upload.py in _main(self, client, fileobj, bucket, key, upload_id, part_number, extra_args)
    723                 UploadId=upload_id, PartNumber=part_number,
--> 724                 Body=body, **extra_args)
    725         etag = response['ETag']

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
    385             # The "self" in this scope is referring to the BaseClient.
--> 386             return self._make_api_call(operation_name, kwargs)
    387 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
    704             error_class = self.exceptions.from_code(error_code)
--> 705             raise error_class(parsed_response, operation_name)
    706         else:

ClientError: An error occurred (RequestTimeout) when calling the UploadPart operation (reached max retries: 4): Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.

During handling of the above exception, another exception occurred:

S3UploadFailedError                       Traceback (most recent call last)
<ipython-input-7-0db8cc5c0fa2> in <module>
      6     initial_instance_count=1,
      7     deserializer=JSONDeserializer(),
----> 8     serializer=JSONSerializer(),
      9 )

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/sagemaker/model.py in deploy(self, initial_instance_count, instance_type, serializer, deserializer, accelerator_type, endpoint_name, tags, kms_key, wait, data_capture_config, **kwargs)
    761                 self._base_name = "-".join((self._base_name, compiled_model_suffix))
    762 
--> 763         self._create_sagemaker_model(instance_type, accelerator_type, tags)
    764         production_variant = sagemaker.production_variant(
    765             self.name, instance_type, initial_instance_count, accelerator_type=accelerator_type

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/sagemaker/model.py in _create_sagemaker_model(self, instance_type, accelerator_type, tags)
    315                 /api/latest/reference/services/sagemaker.html#SageMaker.Client.add_tags
    316         """
--> 317         container_def = self.prepare_container_def(instance_type, accelerator_type=accelerator_type)
    318 
    319         self._ensure_base_name_if_needed(container_def["Image"])

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/sagemaker/pytorch/model.py in prepare_container_def(self, instance_type, accelerator_type)
    237 
    238         deploy_key_prefix = model_code_key_prefix(self.key_prefix, self.name, deploy_image)
--> 239         self._upload_code(deploy_key_prefix, repack=self._is_mms_version())
    240         deploy_env = dict(self.env)
    241         deploy_env.update(self._framework_env_vars())

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/sagemaker/model.py in _upload_code(self, key_prefix, repack)
   1136                 repacked_model_uri=repacked_model_data,
   1137                 sagemaker_session=self.sagemaker_session,
-> 1138                 kms_key=self.model_kms_key,
   1139             )
   1140 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/sagemaker/utils.py in repack_model(inference_script, source_directory, dependencies, model_uri, repacked_model_uri, sagemaker_session, kms_key)
    420             t.add(model_dir, arcname=os.path.sep)
    421 
--> 422         _save_model(repacked_model_uri, tmp_model_path, sagemaker_session, kms_key=kms_key)
    423 
    424 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/sagemaker/utils.py in _save_model(repacked_model_uri, tmp_model_path, sagemaker_session, kms_key)
    436         sagemaker_session.boto_session.resource(
    437             "s3", region_name=sagemaker_session.boto_region_name
--> 438         ).Object(bucket, new_key).upload_file(tmp_model_path, ExtraArgs=extra_args)
    439     else:
    440         shutil.move(tmp_model_path, repacked_model_uri.replace("file://", ""))

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/boto3/s3/inject.py in object_upload_file(self, Filename, ExtraArgs, Callback, Config)
    278     return self.meta.client.upload_file(
    279         Filename=Filename, Bucket=self.bucket_name, Key=self.key,
--> 280         ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
    281 
    282 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/boto3/s3/inject.py in upload_file(self, Filename, Bucket, Key, ExtraArgs, Callback, Config)
    129         return transfer.upload_file(
    130             filename=Filename, bucket=Bucket, key=Key,
--> 131             extra_args=ExtraArgs, callback=Callback)
    132 
    133 

~/software/anaconda/envs/aws-try01/lib/python3.6/site-packages/boto3/s3/transfer.py in upload_file(self, filename, bucket, key, callback, extra_args)
    285             raise S3UploadFailedError(
    286                 "Failed to upload %s to %s: %s" % (
--> 287                     filename, '/'.join([bucket, key]), e))
    288 
    289     def download_file(self, bucket, key, filename, extra_args=None,

S3UploadFailedError: Failed to upload /tmp/tmp2ue6dq21/temp-model.tar.gz to sagemaker-us-west-2-399003097110/pytorch-inference-2021-06-17-08-44-03-971/model.tar.gz: An error occurred (RequestTimeout) when calling the UploadPart operation (reached max retries: 4): Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed.

[johann-petrak]

I have updated the bug report, the upload clearly happens because the PyTorchModel overrides the prepare_container_def method which is invoked as part of deploy() and that code always uploads the model and does not respect the local settings in the configuration.
#2463