Personalize domain recommenders (#327)

* Personalize domain recommenders

* Always deploy personalize pre-create

* Fix template typo from upstream PR

* Fix resource typo

* Create/delete Personalize resources

* CFN cleanup

.gitignore

@@ -22,8 +22,8 @@ src/aws-lambda/*/bin/
 src/aws-lambda/*/package
 src/aws-lambda/elasticsearch-pre-index/products.yaml
 src/aws-lambda/personalize-delete-resources/models
-src/aws-lambda/personalize-pre-create-campaigns/models
-src/aws-lambda/personalize-pre-create-campaigns/data
+src/aws-lambda/personalize-pre-create-resources/models
+src/aws-lambda/personalize-pre-create-resources/data
 src/aws-lambda/pinpoint-auto-workshop/pinpoint-templates
 local/*
 demo.md

README.md

@@ -22,7 +22,7 @@ In order to use the workshops, you will need to deploy the Retail Demo Store int
 
 AWS Service | Workshops Overview | Workshop Links | Level | Duration
 --- | --- | --- | --- | ---
-![Amazon Personalize](./workshop/images/Amazon-Personalize_64.png) Amazon Personalize | The Retail Demo Store uses Amazon Personalize to provide similar item recommendations, search re-ranking based on user preferences, and product recommendations based on user item interactions.  The attached workshop is a throrough walk through of the major features of Amazon Personalize, and how it can be deployed in an ecommerce application like the Retail Demo Store. | [Personalize Setup](./workshop/1-Personalization/1.1-Personalize.ipynb) | 300 | 2-2.5 hours
+![Amazon Personalize](./workshop/images/Amazon-Personalize_64.png) Amazon Personalize | The Retail Demo Store uses Amazon Personalize to provide similar item recommendations, search re-ranking based on user preferences, and product recommendations based on user item interactions.  The attached workshop is a throrough walk through of the major features of Amazon Personalize, and how it can be deployed in an ecommerce application like the Retail Demo Store. | [Personalize Setup](./workshop/1-Personalization/Lab-1-Introduction-and-data-preparation.ipynb) | 300 | 2-2.5 hours
 ![Amazon Pinpoint](./workshop/images/Amazon-Pinpoint_64.png) Amazon Pinpoint | In this workshop we will use Amazon Pinpoint to add the ability to dynamically send welcome messages, abandoned cart messages, and messages with personalized product recommendations to the customers of the Retail Demo Store. | [Email Campaigns](./workshop/4-Messaging/4.1-Pinpoint.ipynb) | 200 | 1 hour
 ![Amazon Lex](./workshop/images/Amazon-Lex_64.png) Amazon Lex |  In this module we're going to implement a conversational chatbot using Amazon Lex and integrate it into the Retail Demo Store's web UI. We'll provide some basic functionality to our chatbot such as being able to provide a return policy to users as well as wiring up the chatbot to the Amazon Personalize ML models we created in the Personalization workshop to provide personalized product recommendations to our users. | [Lex Chatbot](./workshop/5-Conversational/5.1-LexChatbot.ipynb) | 200 | 30 minutes
 ![Amazon Elasticsearch](./workshop/images/Amazon-Elasticsearch-Service_64.png) Amazon Elasticsearch | In this workshop, you will create a new Elasticsearch Index and index the Retail Demo Store product data so that users can search for products.| [Product Search](./workshop/0-StartHere/Search.ipynb) | 200 | 20 minutes
@@ -115,7 +115,7 @@ You can find the URL for the Retail Demo Store Web UI in the Outputs of your mai
 
 Look for the "WebURL" output parameter.
 
-**IMPORTANT NOTE: You must load the WebURL over HTTP and _not_ HTTPS.** The reason for this is that the microservice load balancers do not have SSL certificates configured by default and are therefore accessed by the web application over HTTP. Although the web application is served from CloudFront and CloudFront provides a default SSL certificate for the distribution, you will receive mixed content errors in the browser if you attempt to load and run the web application over HTTPS. 
+**IMPORTANT NOTE: You must load the WebURL over HTTP and _not_ HTTPS.** The reason for this is that the microservice load balancers do not have SSL certificates configured by default and are therefore accessed by the web application over HTTP. Although the web application is served from CloudFront and CloudFront provides a default SSL certificate for the distribution, you will receive mixed content errors in the browser if you attempt to load and run the web application over HTTPS.
 
 You can read more [detailed instructions on how to demo the Retail Demo Store in the Demo section at the end of this document](#delivering-a-demo-of-the-retail-demo-store).
 

aws/cloudformation-templates/base/_template.yaml

@@ -101,7 +101,7 @@ Parameters:
   GitHubBranch:
     Type: String
     Description: The GitHub branch used for deployment.
-    
+
 Resources:
   # Authentication
   Authentication:
@@ -253,6 +253,16 @@ Resources:
       Parameters:
         CleanupBucketLambdaArn: !Ref CleanupBucketLambdaArn
 
+  # Personalize Resources
+  Personalize:
+    Type: AWS::CloudFormation::Stack
+    Properties:
+      TemplateURL: !Sub https://s3.amazonaws.com/${ResourceBucket}/${ResourceBucketRelativePath}cloudformation-templates/base/personalize.yaml
+      Parameters:
+        ResourceBucket: !Ref ResourceBucket
+        Uid: !Sub ${ParentStackName}-${AWS::Region}
+        StackBucketName: !GetAtt Buckets.Outputs.StackBucketName
+
 Outputs:
   UserPoolId:
     Description: Authentication Cognito User Pool Id.
@@ -365,3 +375,7 @@ Outputs:
   WebUICDNURL:
     Description: The URL for the web application
     Value: !GetAtt CloudFront.Outputs.WebUICDNURL
+
+  PersonalizeRoleArn:
+    Description: IAM Role ARN that provides Amazon Personalize access to the stack bucket
+    Value: !GetAtt Personalize.Outputs.PersonalizeServiceRole

aws/cloudformation-templates/base/notebook.yaml

@@ -51,9 +51,9 @@ Conditions:
 Resources:
   RetailDemoStoreGitHubRepo:
     Type: AWS::SageMaker::CodeRepository
-    Properties: 
+    Properties:
       CodeRepositoryName: !Sub '${Uid}-demo-store'
-      GitConfig: 
+      GitConfig:
         Branch: !Ref GitHubBranch
         RepositoryUrl:  !Sub
         - 'https://github.com/${User}/retail-demo-store.git'
@@ -164,25 +164,23 @@ Resources:
                   - s3:GetObject
                   - s3:GetObjectVersion
                   - s3:GetBucketVersioning
-                  - s3:PutBucketPolicy
+                  - s3:GetBucketPolicy
                 Resource:
                   - !Sub arn:aws:s3:::${StackBucketName}/*
                   - !Sub arn:aws:s3:::${StackBucketName}
               -
                 Effect: "Allow"
                 Action:
-                  - iam:CreateRole
-                  - iam:AttachRolePolicy
+                  - iam:GetRole
+                  - iam:GetRolePolicy
                   - iam:PassRole
-                  - iam:DetachRolePolicy
-                  - iam:DeleteRole
                 Resource: !Sub 'arn:aws:iam::${AWS::AccountId}:role/${Uid}-PersonalizeS3'
               -
                 Effect: "Allow"
                 Action:
                   - iam:UpdateAssumeRolePolicy
                 Resource: !Sub 'arn:aws:iam::${AWS::AccountId}:role/${AWS::Region}-mParticleKinesisCrossAccountRole'
-              - 
+              -
                 Effect: "Allow"
                 Action:
                   - iam:CreatePolicy
@@ -240,7 +238,7 @@ Resources:
                 Action:
                   - ssm:PutParameter
                   - ssm:GetParameter
-                Resource: !Sub 'arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/retaildemostore-*'
+                Resource: !Sub 'arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/retaildemostore*'
               - Effect: Allow
                 Action:
                   - dynamodb:*

aws/cloudformation-templates/base/personalize.yaml

@@ -0,0 +1,74 @@
+---
+AWSTemplateFormatVersion: 2010-09-09
+
+Description: >
+    This template creates and prepares Amazon Personalize support resources
+
+Parameters:
+  ResourceBucket:
+    Type: String
+    Description: S3Bucket Bucket where the Resources are stored (cloudformation, images, lambda code)
+
+  Uid:
+    Type: String
+
+  StackBucketName:
+    Type: String
+    Description: S3 bucket name for the stack bucket used to stage Personalize input and output files
+
+Resources:
+  StackBucketPolicy:
+    Type: AWS::S3::BucketPolicy
+    Properties:
+      Bucket: !Ref StackBucketName
+      PolicyDocument:
+        Version: 2012-10-17
+        Statement:
+          - Effect: Allow
+            Action:
+              - s3:GetObject
+              - s3:PutObject
+              - s3:ListBucket
+            Resource:
+              - !Sub "arn:aws:s3:::${StackBucketName}"
+              - !Sub "arn:aws:s3:::${StackBucketName}/*"
+            Principal:
+              Service: personalize.amazonaws.com
+
+  PersonalizeServiceRole:
+    Type: AWS::IAM::Role
+    Properties:
+      RoleName: !Sub "${Uid}-PersonalizeS3"
+      AssumeRolePolicyDocument:
+        Version: 2012-10-17
+        Statement:
+          - Effect: Allow
+            Principal:
+              Service: personalize.amazonaws.com
+            Action:
+              - sts:AssumeRole
+      Policies:
+        - PolicyName: BucketAccess
+          PolicyDocument:
+            Version: 2012-10-17
+            Statement:
+              - Effect: Allow
+                Action:
+                  - s3:GetObject
+                  - s3:ListBucket
+                Resource:
+                  - !Sub "arn:aws:s3:::${ResourceBucket}" # So imports can be run directly from resource bucket
+                  - !Sub "arn:aws:s3:::${ResourceBucket}/*"  # So imports can be run directly from resource bucket
+                  - !Sub "arn:aws:s3:::${StackBucketName}"
+                  - !Sub "arn:aws:s3:::${StackBucketName}/*"
+              - Effect: Allow
+                Action:
+                  - s3:PutObject
+                Resource:
+                  - !Sub "arn:aws:s3:::${StackBucketName}"
+                  - !Sub "arn:aws:s3:::${StackBucketName}/*"
+
+Outputs:
+  PersonalizeServiceRole:
+    Description: Personalize service role ARN
+    Value: !GetAtt PersonalizeServiceRole.Arn

aws/cloudformation-templates/base/ssm.yaml

@@ -89,79 +89,86 @@ Resources:
       Value: !Ref StackBucketName
       Description: "Retail Demo Store Stack Bucket Parameter"
 
-  ParameterRelatedProductsCampaignArn:
+  ParameterPersonalizeRelatedItemsArn:
     Type: "AWS::SSM::Parameter"
     Properties:
-      Name: "retaildemostore-related-products-campaign-arn"
+      Name: "/retaildemostore/personalize/related-items-arn"
       Type: "String"
       Value: "NONE"
-      Description: "Retail Demo Store Related Products Campaign Arn Parameter"
+      Description: "Retail Demo Store Related Items Campaign/Recommender Arn Parameter"
 
-  ParameterProductRecommendationCampaignArn:
+  ParameterPersonalizePopularItemsArn:
     Type: "AWS::SSM::Parameter"
     Properties:
-      Name: "retaildemostore-product-recommendation-campaign-arn"
+      Name: "/retaildemostore/personalize/popular-items-arn"
       Type: "String"
       Value: "NONE"
-      Description: "Retail Demo Store Product Recommendation Campaign Arn Parameter"
+      Description: "Retail Demo Store Popular Items Campaign/Recommender Arn Parameter"
 
-  ParameterPersonalizedRankingCampaignArn:
+  ParameterPersonalizeRecommendedForYouArn:
     Type: "AWS::SSM::Parameter"
     Properties:
-      Name: "retaildemostore-personalized-ranking-campaign-arn"
+      Name: "/retaildemostore/personalize/recommended-for-you-arn"
       Type: "String"
       Value: "NONE"
-      Description: "Retail Demo Store Personalized Ranking Campaign Arn Parameter"
+      Description: "Retail Demo Store Recommended For You Campaign/Recommender Arn Parameter"
 
-  ParameterPersonalizedOffersCampaignArn:
+  ParameterPersonalizePersonalizedRankingArn:
     Type: "AWS::SSM::Parameter"
     Properties:
-      Name: "retaildemostore-personalized-offers-campaign-arn"
+      Name: "/retaildemostore/personalize/personalized-ranking-arn"
       Type: "String"
       Value: "NONE"
-      Description: "Retail Demo Store Personalized Coupon Offers Campaign Arn Parameter"
+      Description: "Retail Demo Store Personalized Ranking Campaign/Recommender Arn Parameter"
 
-  ParameterPersonalizeTrainConfig:
+  ParameterPersonalizePersonalizedOffersArn:
     Type: "AWS::SSM::Parameter"
     Properties:
-      Name: "retaildemostore-training-config"
+      Name: "/retaildemostore/personalize/personalized-offers-arn"
       Type: "String"
       Value: "NONE"
-      Description: "Retail Demo Store Personalize Train Config"
-      Tier: "Advanced"
+      Description: "Retail Demo Store Personalized Coupon Offers Campaign/Recommender Arn Parameter"
 
-  ParameterPersonalizeTrainState:
+  ParameterPersonalizeFilterPurchasedArn:
     Type: "AWS::SSM::Parameter"
     Properties:
-      Name: "retaildemostore-training-state"
+      Name: "/retaildemostore/personalize/filters/filter-purchased-arn"
       Type: "String"
       Value: "NONE"
-      Description: "Retail Demo Store Personalize Train State"
+      Description: "Retail Demo Store Personalize Filter Purchased Products Arn Parameter"
 
-  ParameterPersonalizeFilterPurchasedArn:
+  ParameterPersonalizeFilterCStoreArn:
     Type: "AWS::SSM::Parameter"
     Properties:
-      Name: "retaildemostore-personalize-filter-purchased-arn"
+      Name: "/retaildemostore/personalize/filters/filter-cstore-arn"
       Type: "String"
       Value: "NONE"
-      Description: "Retail Demo Store Personalize Filter Purchased Products Arn Parameter"
+      Description: "Retail Demo Store Filter C-Store Products Arn Parameter"
 
-  ParameterExperimentStrategyTableName:
+  ParameterPersonalizeFilterPurchasedAndCStoreArn:
     Type: "AWS::SSM::Parameter"
     Properties:
-      Name: "retaildemostore-experiment-strategy-table-name"
+      Name: "/retaildemostore/personalize/filters/filter-purchased-and-cstore-arn"
       Type: "String"
-      Value: !Ref ExperimentStrategyTableName
-      Description: "Retail Demo Store Experiment Strategy DDB Table Name"
+      Value: "NONE"
+      Description: "Retail Demo Store Filter Purchased and C-Store Products Arn Parameter"
 
   ParameterPersonalizeEventTrackerId:
     Type: "AWS::SSM::Parameter"
     Properties:
-      Name: "retaildemostore-personalize-event-tracker-id"
+      Name: "/retaildemostore/personalize/event-tracker-id"
       Type: "String"
       Value: "NONE"
       Description: "Retail Demo Store Personalize Event Tracker ID Parameter"
 
+  ParameterExperimentStrategyTableName:
+    Type: "AWS::SSM::Parameter"
+    Properties:
+      Name: "retaildemostore-experiment-strategy-table-name"
+      Type: "String"
+      Value: !Ref ExperimentStrategyTableName
+      Description: "Retail Demo Store Experiment Strategy DDB Table Name"
+
   ParameterAmplitudeApiKey:
     Type: "AWS::SSM::Parameter"
     Properties:
@@ -260,25 +267,21 @@ Outputs:
     Description: Stack Bucket Name Parameter
     Value: !Ref ParameterStackBucketName
 
-  ParameterRelatedProductsCampaignArn:
-    Description: Related Products Campaign Arn Parameter
-    Value: !Ref ParameterRelatedProductsCampaignArn
+  ParameterPersonalizeRelatedItemsArn:
+    Description: Also Viewed Recommender Arn Parameter
+    Value: !Ref ParameterPersonalizeRelatedItemsArn
 
-  ParameterProductRecommendationCampaignArn:
-    Description: Product Recommendation Campaign Arn Parameter
-    Value: !Ref ParameterProductRecommendationCampaignArn
+  ParameterPersonalizePopularItemsArn:
+    Description: Most Viewed Recommender Arn Parameter
+    Value: !Ref ParameterPersonalizePopularItemsArn
 
-  ParameterPersonalizedRankingCampaignArn:
-    Description: Personalized Ranking Campaign Arn
-    Value: !Ref ParameterPersonalizedRankingCampaignArn
+  ParameterPersonalizeRecommendedForYouArn:
+    Description: Recommended For You Recommender Arn Parameter
+    Value: !Ref ParameterPersonalizeRecommendedForYouArn
 
-  ParameterPersonalizeTrainConfig:
-    Description: Train config
-    Value: !Ref ParameterPersonalizeTrainConfig
-
-  ParameterPersonalizeTrainState:
-    Description: Train state
-    Value: !Ref ParameterPersonalizeTrainState
+  ParameterPersonalizePersonalizedRankingArn:
+    Description: Personalized Ranking Campaign Arn
+    Value: !Ref ParameterPersonalizePersonalizedRankingArn
 
   ParameterPersonalizeFilterPurchasedArn:
     Description: Personalize Filter Purchased Products Arn