From 06ead77998bbd172329fc0be66d7ded3a5c3906e Mon Sep 17 00:00:00 2001 From: philschmid Date: Fri, 20 Sep 2024 18:39:34 +0200 Subject: [PATCH 1/6] init --- .../vertex-notebook.ipynb | 503 ++++++++++++++++++ 1 file changed, 503 insertions(+) create mode 100644 examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/vertex-notebook.ipynb diff --git a/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/vertex-notebook.ipynb b/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/vertex-notebook.ipynb new file mode 100644 index 00000000..74d71a46 --- /dev/null +++ b/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/vertex-notebook.ipynb @@ -0,0 +1,503 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Evaluate open LLMs with Vertex AI and Gemini\n", + "\n", + "The [Gen AI Evaluation Service in Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview) lets us evaluate LLMs or Application using existing or your own evaluation criterias. It supports academic metrics like BLEU, ROUGE, or LLM as a Judge with Pointwise and Pairwise metrics or custom metrics you can define yourself. It is not directly saided but it can be assumed that Gemini is used as default Judge. \n", + "\n", + "We can use the Gen AI Evaluation Service to evaluate the performance of open models and finetuned models using Vertex AI Endpoints and compute resources. In this example we will evaluate [meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) generated summaries from news articles using a Pointwise metric based on [G-Eval](https://arxiv.org/abs/2303.16634) Coherence metric.\n", + "\n", + "We will cover the following topics:\n", + "\n", + "1. Setup / Configuration\n", + "2. Deploy Llama 3.1 8B on Vertex AI\n", + "3. Evaluate Llama 3.1 8B using different prompts on Coherence\n", + "4. Interpret the results \n", + "5. Clean up resources\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Setup / Configuration" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "First, you need to install `gcloud` in your local machine, which is the command-line tool for Google Cloud, following the instructions at [Cloud SDK Documentation - Install the gcloud CLI](https://cloud.google.com/sdk/docs/install)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Then, you also need to install the `google-cloud-aiplatform` Python SDK, required to programmatically create the Vertex AI model, register it, acreate the endpoint, and deploy it on Vertex AI." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!pip install --upgrade --quiet \"google-cloud-aiplatform[evaluation]\" huggingface_hub transformers datasets" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "For ease of use we define the following environment variables for GCP.\n", + "\n", + "_Note: Make sure to adapt the project ID to your GCP project._\n", + "\n", + "__Note: The Gen AI Evaluation Service is not available in all regions. If you want to use it, you need to select a region that supports it. At the moment of writing this example, only `us-central1` is supported._ " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%env PROJECT_ID=gcp-partnership-412108\n", + "%env LOCATION=us-central1\n", + "%env CONTAINER_URI=us-docker.pkg.dev/deeplearning-platform-release/gcr.io/huggingface-text-generation-inference-cu121.2-2.ubuntu2204.py310 " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Then you need to login into your GCP account and set the project ID to the one you want to use to register and deploy the models on Vertex AI." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!gcloud auth login\n", + "!gcloud auth application-default login # For local development\n", + "!gcloud config set project $PROJECT_ID" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Once you are logged in, you need to enable the necessary service APIs in GCP, such as the Vertex AI API, the Compute Engine API, and Google Container Registry related APIs.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "!gcloud services enable aiplatform.googleapis.com\n", + "!gcloud services enable compute.googleapis.com\n", + "!gcloud services enable container.googleapis.com\n", + "!gcloud services enable containerregistry.googleapis.com\n", + "!gcloud services enable containerfilesystem.googleapis.com" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Deploy Llama 3.1 8B on Vertex AI\n", + "\n", + "Once everything is set up, we can deploy the Llama 3.1 8B model on Vertex AI. We will use the `google-cloud-aiplatform` Python SDK to do so. [`meta-llama/Meta-Llama-3.1-8B-Instruct`](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) is a gated model, you need to login into your Hugging Face Hub account with a read-access token either fine-grained with access to the gated model, or just overall read-access to your account. More information on how to generate a read-only access token for the Hugging Face Hub in the instructions at .\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from huggingface_hub import interpreter_login\n", + "\n", + "interpreter_login()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "After we are logged in we can \"upload\" the model i.e. register the model on Vertex AI. If you want to learn more about the arguments you can pass to the `upload` method, check out [Deploy Gemma 7B with TGI on Vertex AI](https://github.com/huggingface/Google-Cloud-Containers/blob/main/examples/vertex-ai/notebooks/deploy-gemma-on-vertex-ai/vertex-notebook.ipynb).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "from google.cloud import aiplatform\n", + "\n", + "aiplatform.init(\n", + " project=os.getenv(\"PROJECT_ID\"),\n", + " location=os.getenv(\"LOCATION\"),\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We will deploy the `meta-llama/Meta-Llama-3.1-8B-Instruct` to 1x NVIDIA L4 accelerator with 24GB memory. We set TGI parameters to allow for a maximum of 8000 input tokens, 8192 maximum total tokens, and 8192 maximum batch prefill tokens." + ] + }, + { + "cell_type": "code", + "execution_count": 67, + "metadata": {}, + "outputs": [ + { + "ename": "ServiceUnavailable", + "evalue": "503 Machine type temporarily unavailable, please deploy with a different machine type or retry. 14: Machine type temporarily unavailable, please deploy with a different machine type or retry.", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mServiceUnavailable\u001b[0m Traceback (most recent call last)", + "Cell \u001b[0;32mIn[67], line 23\u001b[0m\n\u001b[1;32m 20\u001b[0m endpoint \u001b[38;5;241m=\u001b[39m aiplatform\u001b[38;5;241m.\u001b[39mEndpoint\u001b[38;5;241m.\u001b[39mcreate(display_name\u001b[38;5;241m=\u001b[39m\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mvertex_model_name\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m-endpoint\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m 22\u001b[0m \u001b[38;5;66;03m# deploy model to 1x NVIDIA L4\u001b[39;00m\n\u001b[0;32m---> 23\u001b[0m deployed_model \u001b[38;5;241m=\u001b[39m \u001b[43mmodel\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdeploy\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 24\u001b[0m \u001b[43m \u001b[49m\u001b[43mendpoint\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mendpoint\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 25\u001b[0m \u001b[43m \u001b[49m\u001b[43mmachine_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mg2-standard-4\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m 26\u001b[0m \u001b[43m \u001b[49m\u001b[43maccelerator_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mNVIDIA_L4\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m 27\u001b[0m \u001b[43m \u001b[49m\u001b[43maccelerator_count\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m1\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m 28\u001b[0m \u001b[43m)\u001b[49m\n", + "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/gcp/lib/python3.11/site-packages/google/cloud/aiplatform/models.py:5253\u001b[0m, in \u001b[0;36mModel.deploy\u001b[0;34m(self, endpoint, deployed_model_display_name, traffic_percentage, traffic_split, machine_type, min_replica_count, max_replica_count, accelerator_type, accelerator_count, tpu_topology, service_account, explanation_metadata, explanation_parameters, metadata, encryption_spec_key_name, network, sync, deploy_request_timeout, autoscaling_target_cpu_utilization, autoscaling_target_accelerator_duty_cycle, enable_access_logging, disable_container_logging, private_service_connect_config, deployment_resource_pool, reservation_affinity_type, reservation_affinity_key, reservation_affinity_values, spot)\u001b[0m\n\u001b[1;32m 5242\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m 5243\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mTraffic splitting is not yet supported for PSA based PrivateEndpoint. \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 5244\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mTry calling deploy() without providing `traffic_split`. \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 5245\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mA maximum of one model can be deployed to each private Endpoint.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 5246\u001b[0m )\n\u001b[1;32m 5248\u001b[0m explanation_spec \u001b[38;5;241m=\u001b[39m _explanation_utils\u001b[38;5;241m.\u001b[39mcreate_and_validate_explanation_spec(\n\u001b[1;32m 5249\u001b[0m explanation_metadata\u001b[38;5;241m=\u001b[39mexplanation_metadata,\n\u001b[1;32m 5250\u001b[0m explanation_parameters\u001b[38;5;241m=\u001b[39mexplanation_parameters,\n\u001b[1;32m 5251\u001b[0m )\n\u001b[0;32m-> 5253\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_deploy\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 5254\u001b[0m \u001b[43m \u001b[49m\u001b[43mendpoint\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mendpoint\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5255\u001b[0m \u001b[43m \u001b[49m\u001b[43mdeployed_model_display_name\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdeployed_model_display_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5256\u001b[0m \u001b[43m \u001b[49m\u001b[43mtraffic_percentage\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtraffic_percentage\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5257\u001b[0m \u001b[43m \u001b[49m\u001b[43mtraffic_split\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtraffic_split\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5258\u001b[0m \u001b[43m \u001b[49m\u001b[43mmachine_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmachine_type\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5259\u001b[0m \u001b[43m \u001b[49m\u001b[43mmin_replica_count\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmin_replica_count\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5260\u001b[0m \u001b[43m \u001b[49m\u001b[43mmax_replica_count\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmax_replica_count\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5261\u001b[0m \u001b[43m \u001b[49m\u001b[43maccelerator_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43maccelerator_type\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5262\u001b[0m \u001b[43m \u001b[49m\u001b[43maccelerator_count\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43maccelerator_count\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5263\u001b[0m \u001b[43m \u001b[49m\u001b[43mtpu_topology\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtpu_topology\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5264\u001b[0m \u001b[43m \u001b[49m\u001b[43mreservation_affinity_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreservation_affinity_type\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5265\u001b[0m \u001b[43m \u001b[49m\u001b[43mreservation_affinity_key\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreservation_affinity_key\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5266\u001b[0m \u001b[43m \u001b[49m\u001b[43mreservation_affinity_values\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreservation_affinity_values\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5267\u001b[0m \u001b[43m \u001b[49m\u001b[43mservice_account\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mservice_account\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5268\u001b[0m \u001b[43m \u001b[49m\u001b[43mexplanation_spec\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mexplanation_spec\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5269\u001b[0m \u001b[43m \u001b[49m\u001b[43mmetadata\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmetadata\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5270\u001b[0m \u001b[43m \u001b[49m\u001b[43mencryption_spec_key_name\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mencryption_spec_key_name\u001b[49m\n\u001b[1;32m 5271\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;129;43;01mor\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43minitializer\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mglobal_config\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mencryption_spec_key_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5272\u001b[0m \u001b[43m \u001b[49m\u001b[43mnetwork\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mnetwork\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5273\u001b[0m \u001b[43m \u001b[49m\u001b[43msync\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msync\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5274\u001b[0m \u001b[43m \u001b[49m\u001b[43mdeploy_request_timeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdeploy_request_timeout\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5275\u001b[0m \u001b[43m \u001b[49m\u001b[43mautoscaling_target_cpu_utilization\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mautoscaling_target_cpu_utilization\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5276\u001b[0m \u001b[43m \u001b[49m\u001b[43mautoscaling_target_accelerator_duty_cycle\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mautoscaling_target_accelerator_duty_cycle\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5277\u001b[0m \u001b[43m \u001b[49m\u001b[43mspot\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mspot\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5278\u001b[0m \u001b[43m \u001b[49m\u001b[43menable_access_logging\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43menable_access_logging\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5279\u001b[0m \u001b[43m \u001b[49m\u001b[43mdisable_container_logging\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdisable_container_logging\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5280\u001b[0m \u001b[43m \u001b[49m\u001b[43mprivate_service_connect_config\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mprivate_service_connect_config\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5281\u001b[0m \u001b[43m \u001b[49m\u001b[43mdeployment_resource_pool\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdeployment_resource_pool\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5282\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n", + "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/gcp/lib/python3.11/site-packages/google/cloud/aiplatform/base.py:863\u001b[0m, in \u001b[0;36moptional_sync..optional_run_in_thread..wrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 861\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m:\n\u001b[1;32m 862\u001b[0m VertexAiResourceNounWithFutureManager\u001b[38;5;241m.\u001b[39mwait(\u001b[38;5;28mself\u001b[39m)\n\u001b[0;32m--> 863\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mmethod\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 865\u001b[0m \u001b[38;5;66;03m# callbacks to call within the Future (in same Thread)\u001b[39;00m\n\u001b[1;32m 866\u001b[0m internal_callbacks \u001b[38;5;241m=\u001b[39m []\n", + "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/gcp/lib/python3.11/site-packages/google/cloud/aiplatform/models.py:5468\u001b[0m, in \u001b[0;36mModel._deploy\u001b[0;34m(self, endpoint, deployed_model_display_name, traffic_percentage, traffic_split, machine_type, min_replica_count, max_replica_count, accelerator_type, accelerator_count, tpu_topology, reservation_affinity_type, reservation_affinity_key, reservation_affinity_values, service_account, explanation_spec, metadata, encryption_spec_key_name, network, sync, deploy_request_timeout, autoscaling_target_cpu_utilization, autoscaling_target_accelerator_duty_cycle, spot, enable_access_logging, disable_container_logging, private_service_connect_config, deployment_resource_pool)\u001b[0m\n\u001b[1;32m 5456\u001b[0m endpoint \u001b[38;5;241m=\u001b[39m PrivateEndpoint\u001b[38;5;241m.\u001b[39mcreate(\n\u001b[1;32m 5457\u001b[0m display_name\u001b[38;5;241m=\u001b[39mdisplay_name,\n\u001b[1;32m 5458\u001b[0m network\u001b[38;5;241m=\u001b[39mnetwork,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 5463\u001b[0m private_service_connect_config\u001b[38;5;241m=\u001b[39mprivate_service_connect_config,\n\u001b[1;32m 5464\u001b[0m )\n\u001b[1;32m 5466\u001b[0m _LOGGER\u001b[38;5;241m.\u001b[39mlog_action_start_against_resource(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mDeploying model to\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m\"\u001b[39m, endpoint)\n\u001b[0;32m-> 5468\u001b[0m \u001b[43mendpoint\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_deploy_call\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 5469\u001b[0m \u001b[43m \u001b[49m\u001b[43mendpoint\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mapi_client\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5470\u001b[0m \u001b[43m \u001b[49m\u001b[43mendpoint\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mresource_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5471\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5472\u001b[0m \u001b[43m \u001b[49m\u001b[43mendpoint\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_gca_resource\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtraffic_split\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5473\u001b[0m \u001b[43m \u001b[49m\u001b[43mnetwork\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mnetwork\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;129;43;01mor\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mendpoint\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mnetwork\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5474\u001b[0m \u001b[43m \u001b[49m\u001b[43mdeployed_model_display_name\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdeployed_model_display_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5475\u001b[0m \u001b[43m \u001b[49m\u001b[43mtraffic_percentage\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtraffic_percentage\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5476\u001b[0m \u001b[43m \u001b[49m\u001b[43mtraffic_split\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtraffic_split\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5477\u001b[0m \u001b[43m \u001b[49m\u001b[43mmachine_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmachine_type\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5478\u001b[0m \u001b[43m \u001b[49m\u001b[43mmin_replica_count\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmin_replica_count\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5479\u001b[0m \u001b[43m \u001b[49m\u001b[43mmax_replica_count\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmax_replica_count\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5480\u001b[0m \u001b[43m \u001b[49m\u001b[43maccelerator_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43maccelerator_type\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5481\u001b[0m \u001b[43m \u001b[49m\u001b[43maccelerator_count\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43maccelerator_count\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5482\u001b[0m \u001b[43m \u001b[49m\u001b[43mtpu_topology\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtpu_topology\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5483\u001b[0m \u001b[43m \u001b[49m\u001b[43mreservation_affinity_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreservation_affinity_type\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5484\u001b[0m \u001b[43m \u001b[49m\u001b[43mreservation_affinity_key\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreservation_affinity_key\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5485\u001b[0m \u001b[43m \u001b[49m\u001b[43mreservation_affinity_values\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreservation_affinity_values\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5486\u001b[0m \u001b[43m \u001b[49m\u001b[43mservice_account\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mservice_account\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5487\u001b[0m \u001b[43m \u001b[49m\u001b[43mexplanation_spec\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mexplanation_spec\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5488\u001b[0m \u001b[43m \u001b[49m\u001b[43mmetadata\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmetadata\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5489\u001b[0m \u001b[43m \u001b[49m\u001b[43mdeploy_request_timeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdeploy_request_timeout\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5490\u001b[0m \u001b[43m \u001b[49m\u001b[43mautoscaling_target_cpu_utilization\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mautoscaling_target_cpu_utilization\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5491\u001b[0m \u001b[43m \u001b[49m\u001b[43mautoscaling_target_accelerator_duty_cycle\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mautoscaling_target_accelerator_duty_cycle\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5492\u001b[0m \u001b[43m \u001b[49m\u001b[43mspot\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mspot\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5493\u001b[0m \u001b[43m \u001b[49m\u001b[43menable_access_logging\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43menable_access_logging\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5494\u001b[0m \u001b[43m \u001b[49m\u001b[43mdisable_container_logging\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdisable_container_logging\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5495\u001b[0m \u001b[43m \u001b[49m\u001b[43mdeployment_resource_pool\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdeployment_resource_pool\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5496\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 5498\u001b[0m _LOGGER\u001b[38;5;241m.\u001b[39mlog_action_completed_against_resource(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmodel\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdeployed\u001b[39m\u001b[38;5;124m\"\u001b[39m, endpoint)\n\u001b[1;32m 5500\u001b[0m endpoint\u001b[38;5;241m.\u001b[39m_sync_gca_resource()\n", + "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/gcp/lib/python3.11/site-packages/google/cloud/aiplatform/models.py:1957\u001b[0m, in \u001b[0;36mEndpoint._deploy_call\u001b[0;34m(cls, api_client, endpoint_resource_name, model, endpoint_resource_traffic_split, network, deployed_model_display_name, traffic_percentage, traffic_split, machine_type, min_replica_count, max_replica_count, accelerator_type, accelerator_count, tpu_topology, reservation_affinity_type, reservation_affinity_key, reservation_affinity_values, service_account, explanation_spec, metadata, deploy_request_timeout, autoscaling_target_cpu_utilization, autoscaling_target_accelerator_duty_cycle, spot, enable_access_logging, disable_container_logging, deployment_resource_pool)\u001b[0m\n\u001b[1;32m 1945\u001b[0m operation_future \u001b[38;5;241m=\u001b[39m api_client\u001b[38;5;241m.\u001b[39mdeploy_model(\n\u001b[1;32m 1946\u001b[0m endpoint\u001b[38;5;241m=\u001b[39mendpoint_resource_name,\n\u001b[1;32m 1947\u001b[0m deployed_model\u001b[38;5;241m=\u001b[39mdeployed_model,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 1950\u001b[0m timeout\u001b[38;5;241m=\u001b[39mdeploy_request_timeout,\n\u001b[1;32m 1951\u001b[0m )\n\u001b[1;32m 1953\u001b[0m _LOGGER\u001b[38;5;241m.\u001b[39mlog_action_started_against_resource_with_lro(\n\u001b[1;32m 1954\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mDeploy\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmodel\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;28mcls\u001b[39m, operation_future\n\u001b[1;32m 1955\u001b[0m )\n\u001b[0;32m-> 1957\u001b[0m \u001b[43moperation_future\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mresult\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtimeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mNone\u001b[39;49;00m\u001b[43m)\u001b[49m\n", + "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/gcp/lib/python3.11/site-packages/google/api_core/future/polling.py:261\u001b[0m, in \u001b[0;36mPollingFuture.result\u001b[0;34m(self, timeout, retry, polling)\u001b[0m\n\u001b[1;32m 256\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_blocking_poll(timeout\u001b[38;5;241m=\u001b[39mtimeout, retry\u001b[38;5;241m=\u001b[39mretry, polling\u001b[38;5;241m=\u001b[39mpolling)\n\u001b[1;32m 258\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_exception \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m 259\u001b[0m \u001b[38;5;66;03m# pylint: disable=raising-bad-type\u001b[39;00m\n\u001b[1;32m 260\u001b[0m \u001b[38;5;66;03m# Pylint doesn't recognize that this is valid in this case.\u001b[39;00m\n\u001b[0;32m--> 261\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_exception\n\u001b[1;32m 263\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_result\n", + "\u001b[0;31mServiceUnavailable\u001b[0m: 503 Machine type temporarily unavailable, please deploy with a different machine type or retry. 14: Machine type temporarily unavailable, please deploy with a different machine type or retry." + ] + } + ], + "source": [ + "from huggingface_hub import get_token\n", + "\n", + "vertex_model_name = \"llama-3-1-8b-instruct\"\n", + "\n", + "model = aiplatform.Model.upload(\n", + " display_name=vertex_model_name,\n", + " serving_container_image_uri=os.getenv(\"CONTAINER_URI\"),\n", + " serving_container_environment_variables={\n", + " \"MODEL_ID\": \"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n", + " \"MAX_INPUT_TOKENS\": \"8000\",\n", + " \"MAX_TOTAL_TOKENS\": \"8192\",\n", + " \"MAX_BATCH_PREFILL_TOKENS\": \"8192\",\n", + " \"HUGGING_FACE_HUB_TOKEN\": get_token(),\n", + " },\n", + " serving_container_ports=[8080],\n", + ")\n", + "model.wait() # wait for the model to be registered\n", + "\n", + "# create endpoint\n", + "endpoint = aiplatform.Endpoint.create(display_name=f\"{vertex_model_name}-endpoint\")\n", + "\n", + "# deploy model to 1x NVIDIA L4\n", + "deployed_model = model.deploy(\n", + " endpoint=endpoint,\n", + " machine_type=\"g2-standard-4\",\n", + " accelerator_type=\"NVIDIA_L4\",\n", + " accelerator_count=1,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**WARNING**: _The Vertex AI endpoint deployment via the `deploy` method may take from 15 to 25 minutes._\n", + "\n", + "After the model is deployed, we can test our endpoint. We generate a helper `generate` function to send requests to the deployed model. This will be later used to send requests to the deployed model and collect the outputs for evaluation." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import re\n", + "from transformers import AutoTokenizer\n", + "\n", + "# grep the model id from the container spec environment variables\n", + "model_id = next((re.search(r'value: \"(.+)\"', str(item)).group(1) for item in list(model.container_spec.env) if 'MODEL_ID' in str(item)), None)\n", + "tokenizer = AutoTokenizer.from_pretrained(model_id)\n", + "\n", + "generation_config = {\n", + " \"max_new_tokens\": 256,\n", + " \"do_sample\": True,\n", + " \"top_p\": 0.2,\n", + " \"temperature\": 0.2,\n", + "}\n", + "\n", + "def generate(prompt, generation_config=generation_config):\n", + " formatted_prompt = tokenizer.apply_chat_template(\n", + " [\n", + " {\"role\": \"user\", \"content\": prompt},\n", + " ],\n", + " tokenize=False,\n", + " add_generation_prompt=True,\n", + " )\n", + " \n", + " payload = {\n", + " \"inputs\": formatted_prompt,\n", + " \"parameters\": generation_config\n", + " }\n", + " output = deployed_model.predict(instances=[payload])\n", + " generated_text = output.predictions[0]\n", + " return generated_text\n", + "\n", + "\n", + "generate(\"How many people live in Berlin?\", generation_config)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Evaluate Llama 3.1 8B using different prompts on Coherence\n", + "\n", + "We will evaluate the Llama 3.1 8B model using different prompts on Coherence. Coherence measures how well the individual sentences within a summarized news article connect together to form a unified and easily understandable narrative.\n", + "\n", + "We are going to use the new [Generative AI Evaluation Service](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview). The Gen AI Evaluation Service can be used to: \n", + "* Model selection: Choose the best pre-trained model for your task based on benchmark results and its performance on your specific data.\n", + "* Generation settings: Tweak model parameters (like temperature) to optimize output for your needs.\n", + "* Prompt engineering: Craft effective prompts and prompt templates to guide the model towards your preferred behavior and responses.\n", + "* Improve and safeguard fine-tuning: Fine-tune a model to improve performance for your use case, while avoiding biases or undesirable behaviors.\n", + "* RAG optimization: Select the most effective Retrieval Augmented Generation (RAG) architecture to enhance performance for your application.\n", + "* Migration: Continuously assess and improve the performance of your AI solution by migrating to newer models when they provide a clear advantage for your specific use case.\n", + "\n", + "In our case, we will use it to evaluate different prompt templates to achieve the most coherent summaries using Llama 3.1 8B Instruct. \n", + "\n", + "We are going to use a reference free Pointwise metric based on [G-Eval](https://arxiv.org/abs/2303.16634) Coherence metric. \n", + "\n", + "The first step is to define our prompt template and create our `PointwiseMetric`. Vertex AI returns our response from the model in the `response` field our news article will be made available in the `text` field." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from vertexai.evaluation import EvalTask, PointwiseMetric\n", + "\n", + "g_eval_coherence = \"\"\"\n", + "You are an expert evaluator. You will be given one summary written for a news article.\n", + "Your task is to rate the summary on one metric.\n", + "Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.\n", + "\n", + "Evaluation Criteria:\n", + "\n", + "Coherence (1-5) - the collective quality of all sentences. We align this dimension with the DUC quality question of structure and coherence whereby \"the summary should be well-structured and well-organized. The summary should not just be a heap of related information, but should build from sentence to a coherent body of information about a topic.\"\n", + "\n", + "Evaluation Steps:\n", + "\n", + "1. Read the news article carefully and identify the main topic and key points.\n", + "2. Read the summary and compare it to the news article. Check if the summary covers the main topic and key points of the news article, and if it presents them in a clear and logical order.\n", + "3. Assign a score for coherence on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria.\n", + "\n", + "\n", + "Example:\n", + "\n", + "\n", + "Source Text:\n", + "\n", + "{text}\n", + "\n", + "Summary:\n", + "\n", + "{response}\n", + "\n", + "Evaluation Form (scores ONLY):\n", + "\n", + "- Coherence:\"\"\"\n", + "\n", + "metric = PointwiseMetric(\n", + " metric=\"g-eval-coherence\",\n", + " metric_prompt_template=g_eval_coherence,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We are going to use [argilla/news-summary](https://huggingface.co/datasets/argilla/news-summary) dataset consisting of news article from Reuters. We are going to use a random subset of 10 articles to keep the evaluation fast. Feel free to change the dataset and the number of articles to evaluate the model with more data and different topics.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from datasets import load_dataset\n", + "\n", + "subset_size = 3\n", + "dataset = load_dataset(\"argilla/news-summary\", split=f\"train\").shuffle(seed=42).select(range(subset_size))\n", + "\n", + "print(dataset[0][\"text\"][:150])\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Before we can run the evaluation, we need to convert our dataset into a pandas dataframe. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# remove all columns except for \"text\"\n", + "to_remove = [col for col in dataset.features.keys() if col != \"text\"]\n", + "dataset = dataset.remove_columns(to_remove)\n", + "df = dataset.to_pandas()\n", + "df.head()\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Awesome! We are almost ready. Last step is to define our different summarization prompts we want to use for evaluation. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "summarization_prompts = {\n", + " \"simple\": \"Summarize the following news article: {text}\",\n", + " \"eli5\": \"Summarize the following news article in a way a 5 year old would understand: {text}\",\n", + " \"detailed\": \"\"\"Summarize the given news article, text, including all key points and supporting details? The summary should be comprehensive and accurately reflect the main message and arguments presented in the original text, while also being concise and easy to understand. To ensure accuracy, please read the text carefully and pay attention to any nuances or complexities in the language.\n", + " \n", + "Article:\n", + "{text}\"\"\"\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now we can iterate over our prompts and create different evaluation tasks, use our coherence metric to evaluate the summaries and collect the results.\n", + "\n", + "https://github.com/googleapis/python-aiplatform/blob/main/vertexai/evaluation/_evaluation.py" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# for prompt_name, prompt in summarization_prompts.items():\n", + "prompt_name = \"simple\"\n", + "prompt = summarization_prompts[prompt_name]\n", + "\n", + "# 1. add new prompt column\n", + "df[\"prompt\"] = df[\"text\"].apply(lambda x: prompt.format(text=x))\n", + "\n", + "# 2. create eval task\n", + "eval_task = EvalTask(\n", + " dataset=df,\n", + " metrics=[metric],\n", + " experiment=f\"llama-3-1-8b-instruct-{prompt_name}\",\n", + ")\n", + "# 3. run eval task\n", + "results = eval_task.evaluate(model=generate)\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "You can find more examples on how to use the Gen AI Evaluation Service in the [Vertex AI Generative AI documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Resource clean-up" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Finally, you can already release the resources that you've created as follows, to avoid unnecessary costs:\n", + "\n", + "* `deployed_model.undeploy_all` to undeploy the model from all the endpoints.\n", + "* `deployed_model.delete` to delete the endpoint/s where the model was deployed gracefully, after the `undeploy_all` method.\n", + "* `model.delete` to delete the model from the registry." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "deployed_model.undeploy_all()\n", + "deployed_model.delete()\n", + "model.delete()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Alternatively, you can also remove those from the Google Cloud Console following the steps:\n", + "\n", + "* Go to Vertex AI in Google Cloud\n", + "* Go to Deploy and use -> Online prediction\n", + "* Click on the endpoint and then on the deployed model/s to \"Undeploy model from endpoint\"\n", + "* Then go back to the endpoint list and remove the endpoint\n", + "* Finally, go to Deploy and use -> Model Registry, and remove the model" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.12" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} From 644de94104550cb8e368f9b6e62ae20db32d1ee4 Mon Sep 17 00:00:00 2001 From: philschmid Date: Sun, 22 Sep 2024 10:42:47 +0200 Subject: [PATCH 2/6] v1 --- .../assets/experiment-results.png | Bin 0 -> 49121 bytes .../vertex-notebook.ipynb | 147 ++++++++++-------- 2 files changed, 83 insertions(+), 64 deletions(-) create mode 100644 examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/assets/experiment-results.png diff --git a/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/assets/experiment-results.png b/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/assets/experiment-results.png new file mode 100644 index 0000000000000000000000000000000000000000..6c8c559e06fa65a548b6c0c57a27e49ae2e6f023 GIT binary patch literal 49121 zcmeFZWmsIzwl&(gdjbS^lHd^BgN5KB!QI`ZaSbjBPSD^4cXxMpcX#)1vG+M=pXc5; z+`r$Co2MIAx~po{nl-Cx&M`(2C@U?7f{2F*0)bG(KM8*UfnZF5>kD{T;3t)Dwi^h9 zm}e{`Br7f?L?UZrX<%%o4+4D(jE;qqjhw;mI=agA_r-=M{Af9<3VKIcKAh>3<4Ym% zmI>NBNL^X2r6QHgJ1AA+o7K0V{4}y47&%%`dqHd%eRbMtStnvf-vzg`#RNx#!}+Cq z+ryfz`FU6Ua7#I@uAC!m}C#fP`?W0!YAhfrhh~r)m36byge0`xo?VFoT zbN(72sUgRKEZv8@qo;}W>Uv33kSu&F!n~w3S8Mi9rcR8YQjh?lsNKSAk>Ap=0QtMBL@T!Q_t`MysebvXt zVr?<_yoxbaeT*`8uFrj&+Qn~G`=P}p*|^19%PRVD7b8>NN$M8e@`tZ^s5<`wTcAPd z{fH20{5{4>ZaA4zJFnU5?VGyHxoK;7Z^)s%kI&SVC~2@yU^|oi7*Sy?2w|JpePtlN zs#nsw6&Jhnnyo8s<99>@rif%GL7B+eU64FEg)?(FWnfAq0TFB0%zbw%=b$9YS<036 zXS<&qUnBTsme-h?imEa$&-@-qyGr5PId1EooEY~0NQ){*6>-AQecKYB92gFa|MnR)RZgA_dDJ5=LY z)!}VxtfB)_q7p@PNf<)JbP+fVHWoKN<#OEnE87?Dc3rplOTlb+gDH>LgGI_m_nwh1F{fb;iCmMWKqtM z>|kr*)qR{(;f6^*AU1q&YT7c{gxP>*ru|ByQOzV=M8%dBh?*e%EF~aM#2geZ_RIB z*A8NgjuY+I?|18G+=Rjo&eTCEK~>T;LuP}i!>dDaL7($%@H}Wi)uy)MNJJcbwdIY} zSz@hR%`@*l&ou8ePdLwbh^2v8^u4OnWli@4@}9~SqY-<_ugS;5D}|yP^Bh6iR|_(> zIH=giDr)$1wFoYiB<4ndpD>ON2h;cJpGuPX)EZQLRE?1l5u-nCqEsk4VrNOd(b|99 z{WG3AZWzJSt1fCF_`L|OM-59_+bdiJ`^=qqDenac@yC|j=3^MMkd!r@T`}nYNNPg878%+Sw@E=W@7{sX+||BCl!}QNmI_^ z@)=A<#^pLHaOP^N>T`;!`RauW1-!YsIat5T#Ipsr0_o*q3lnnc#H!RC)np_XWGzEA zk`{&By!q7ml|542vmeEvgnZum(7_JClKb`g(Y6t^akX`c>XEajBc~&#yL2k))>{8L z5nP+2Dh^B%FBQ-2p_88;UyQPJ9xgY8Fz{M~Z1E0)aWkM`@l&o;PMS&OoZxA8jdUuz zF-77HRti?fPQqTIJD?q-4OihSji@>;o7I~zxtN@7|D~&L;x^=Gwq{kVY}pX-yI_rTOnQ%>P`(>M8- zHOs*#7gq}#b6ula)=I4T*8Ffy74C$O6izbDaq2C}pR_0mKH+XLa^7(iRUgjt&dVhu75%s(8=)*= zw5I5xGK9LahI}FedJeHH$bmT_c(7@2t zDa7Z`*6xagISvlxjpT9W%}p5<%~Ts$(dnAsa2ct$)?QxJ?%7VESsu^GHvFt+vR@mJ zib+6ZZ&)Izs9{wkcN97tM&sij-w@vlJiV>tuzYui@j(96by&crdS&^LJT%axp`|SQ z_1GTYaPA@{-~Iigv5Cv-LzY%3$j<-;uAa^4%4#Uu5~ zE0v$-ax;9Zd?$@wtDdVd56*XvhfgXIYBo*d>E$b)w_XKsd_?O+^|sv)vTpIB%7Qbo zZTD>t8+;p5uVG({h!u;KhV$^_IhCIo-VU)w|A~f)Uit(54Qk@|1R*ObD{IAW1^4bU zm-WJSNwscOxC@bGE_)t>Wh&dNL=bg5wZW?D@9t+hJC|mjsLB_|L{x z@YlX_+tKJs3p?z4j4jeQX|&0W9C3R5(>GBzl~|ZmBi*#<+V&_aAFHmoCRLXZx^TH5 zRYhP?G4D{nFk4ofeGoR66zs@*q*2pV>$v7V{E)mDu=v)bL<)(zl}6su-#3G=NW&*EI8WeR*e z_?GmcZN2MW=6%W^{xN<>_rTjz9h(lt`KeXQ&Lx$mI}Z<@Xg*d-z2Iq<0ubUGD;+YB zWgdv6i=?5|925|2c(?P$$Dj$q0REmj`Jl$IpNPa90@Q*7Iu`8wiSmg%xc@W&dlgln zn%mRdj6%2YDrsjdJ5&(T6WzaD`o2qj0>7ExtkVBUV(}@_i(+?a(*wf02h#7u)=)YS zy(;J{i5o~sf!+hx@E{0(V-PfO1p&PAfHw#P72yYh1Ab!yZ{bwP|Na)nG!^Q~T97=qtF2>cZ^uhU_I#uN`SVu8{b0-Vi|9ULo z0U4gZVPK+XWcZ)9fvP;upK{3>gZ0gng^kSt^8okYWB$O(^RM#%uW$Zy$NyDR=|44@ zSUA}Kch&#()&H~VH(PxhAxm@Mo_2iy8Jhp9{J+2auZlbj&yW7UO!03!|Lar0(0qtI z4F5A|e28$-HW46@07zU|@GBT%KMAIOL3q}GskTm5B}bOvvo&4$r(kZ0;JP-*;OLGG zpD#*w*4Rq(Nv&xxL3KjE59wtY4q^>^)ojKn#0rya?>z46c(*7b<9mQtzsI7a2`x&$ z9vKPb3$JgVQw^fC3fn~lpoRbbB7g|2cn|+VGi0(LXo^YfW)WHJ7n%_k&?)Hkgb(=p z>lFyN6O0u`6buGB#9v=Ol=7Hgd@2|L0UdJ)HE4aom!67Z@(=P0eF+eNK>wkn3c?Xo zexWbu&!&O;|4s9cgY|!vX>!yh;VtS8*y^qpT_j4?$_%7TrwWq_8ZKuYdf#O$WlJZ0 zzT63Z(L1rc(oXXU*)0;?9&ffLD@+yV%*qDL(nHxwF~~)yE2cC|O$%$X8Xj-=I@&dy zUjN;L(Y-JsMK+Z8Y~4mgaAt1zijF*Hr!+mx6U<5j3Ej>|*bMsPA`WiJUYfccA5-6U z%)XsJT&SBH3Z_L!epX1jQ3JW)iBAv)qnQ-WbL4^?yywT{iZ{u%EG+vpfW+C>MO~^ zOB2A02Ywvhw>Q&?K#|g4&8h{)Z~FG2YANP4pOWS7u&$`#dNr8Mc$9*=a@DxbXoT!; zGr{unMul>jLFHN)_bkg0&sycx_DGx4TB+M?x$$I{6#iDOYLmMgQ2A&+0ts60=6f=) zORa9@tO*T{L2uMMhufX(xls1S?aDp=fAqWm`mj4zK(0VJ!7d`83cdmus!%MG$W}^o zVqWL+i@m9#&M-n#O}C3^-jjALHjhWwEs?$On)6YKr0m0@rpMjM&Ll={?~9$WE{ob7 zDttcIYP2kw7jG9VAV=@RXtgX*g$&V0Q2+FJcR6F2!ihHR%B#k0zpM9jzr`;BIt2V= zGF1=?pUq6Er}8@zj>^GoMS((riUocO?QhR3^RX-`iPo1rab4(n?8EtPyOpv)LbB#F zrJ1T(o|(v}C0zEqQC#^N4bGF**18GomKr^mwu78qbQQyZy9)L59ydcRI9Jpi)|!gr zx?#NJp{@w8Hkje_+51$=?@M@E&UeNRfS&io3MzCRl#fZ%^9rlRs872bH58c7RbDRp zpc&ZLLk9kRZ4VH@^GbVV+D2_RYn(mvJ}+PA;qHn(=KZlkx+scjJ-FmanLb6m)~+O& zzP7tT4Y{RKwaUWbdc}Wk__t6WrMt)rv!Qqu^R079%mzAr-9GbGMA4i@UBUWihwb4O z_s1K<1SVb4o=UlN(Hb28XtVCqzC=#je7Czxhwb0~F$5*Rbe%(>PF!|hTcN>v$s(ob z&o_zWUkxBPOR}FVp((6R6HYu>sH-}GEKLvN>V-wY({r4+_J4L7rzLvz+7HwQp?IvN zfcwsphx6!crFb+p0L#ziv~Kte>y(=A{+vx8oo$ee*Gu~^$>&s+fC^x$mgW`K_7J zrQr?Uz{bC{x1W@+w1`K4 z1|pTrRRx_F3`K@60~F%HF@JGG`AkVtQu)Q2hUq0^22hifuq6jq2E8;*Gckd%C$~B} z?j>WkWBpLV9}-c0$#GM^AbiR%_&Wbt@+D)g0);}i(h6R(fNU-hUMjq|prm@qm?uD? zPTLWMmn;CdQ$-(l1+eVS<|~((E!I0#+U<;-bDbDU3 zZSJKZP8SU=1-vU-(e5?&5V6O-_EFQ*qsPv1{Ne3leM2GAWN+GyY2Mw`L#>+ItP1NB?B^fHHjG75 z8TKXDwNX+ARm+BoJ_(K_6t?N+#-~RY)nY9#-t%E0QTJmq(N9ap*-2{GMUA(;Nk@PO zY`Nb(X*g}XCMRsQC7e>Tm6`)?JzwH0nD4QbHZ zk2Rm#o_3pGjqWsRl@!}dJLPw{E88Jn!=}gEJnrPu;#rft8?z|ah}c+7_nD5Nt5CMt zZihuDL)}x)`oSdbsc07Hoi;T7`gIOmx04PWhxMOgwdP1T^yLW_wLM{+o4Bz=eNO0p zHeT5rL@W6tm~5A8Qts^-V$}oSg+==*b;=DGzIGP(d$6kg#J2-aDAczrufoKZ{(1dL z{e!Ul4WG@Oe7sbF&dwgB+W#~AGIREvTUJ{fH`$n7qh@rJb=gq|hp z6xRswTcT?BP$i-|LpRmdHS?PxkH;}5It}ZN*Ms<2(Tj6{?-&9(e}k&xgk|4WWY=3n1DtrCM9^%C;FcLY(YBX?F?tTN!BYHAxHdk42hPsj121)e2N+OnivRGS)*gjXL zU8-KJ3doGB{OI>?oRNqnxYl7#YQN6@#?l z#;)6+pP?6eD<~kyDrh>aIHb7OK8Qtw!oQ1wsBHc9k&;>THwEi58dtLMRwxD)OCahm z13Fyg3pEA7pmdEzz1iVD7;C$N*PO&770=uuw!z0~NQKD3z>{!N-@%Pnq-Uu{t+dKY zz*dMww5(yZHMZqthw@V zQ9=P378)yBZ#fnw9+XE`HC5Q%jl_ExDZ^JY4igll9e+BswuZa-Lt7|+!`hlw{2JMR zX!A}^jPM(obk=vVX5m*MEIbPSXvBxma!tgR;+7*~>kpUJGVPM}f>rZ25>?!;`@RTR zpJO1ksJ-b{-zWcqFP1yAE?Eajpz++SM`n6kD}^{Rh%N>oS-+CetN(gZjXGCtz4pBIQYL75zn!nK z-HI>=Td&DU%*G>I$0(CS+qK&^eT7@>2Jf%hOVk+j_*>z{uj=z}e7|R2L1#P7XZ3Cq zKL1AxrT##;t}w+gK^ll;>n+G>pFP#6!zN{=O%z$tyAAb!-+5c?p7hn79FGha``3zo z5BUN2VYP02nIB1THJ%|jUOJIu#3{d&cm@3g6Bn@4(l~Ves0r4#F^1c|1Mll5yvA7=l-Jf=PdnrOAgezv(kAu7~X7 zAaP!M7gCZ4?Zk09bYTX7O(N93Vry->*m9hMz)H$dyJ@MWWd6@do!1{K&#nDs+#?lX zmC7!J3Ih303Saw3zOr)ovI>9TF;Nb$bUa!dG>wW4B0W9%NW)_?qX_SZJJ6R~fAWz@ zn73UnMC{24L#c8zmRF!(PZ^3RJe-B-#vf*9nJ_4usSj>hdykHuzFizztF%}ThvyK)w(8GJ3Rbu%;0 zew%5uskgrso8NRVGiv)mtiUqIppPqgI}j^OG=$JeN9oVN#F%O&sniFpY=)kGd?Q%w zhoDiL5xH)>A3m1-&oSdy7ziGtHDoc)nA8v!4GT@+$&)LH2*{KYH^~2VbVQbi+9`8+ zu^Kt`KLRV9eR%&FB(<-qZyOXE&v1dQ%3^W_j+&KVu33r2ZsXmds)_13Tq`xhpR`mP z9@F1s)e8XNU?HW%L+$iisj_nv)cR{D!QxaC03?0~#a|_caT%{?h zAMKcEaB*}jC{vDT-U^gC4O84(3+J~G@^oLI)&6=ONyBC*amo7&o=4g zp3=j5gS`+rT5R!^|38m@6g!0PY|h&~UW@cTlIseyK5n%3*dl4Xqcjx^PTDRGnpqz< zc?D%uQj?m-L|SK8kq1#vB&3QAA5eU#Cr&JN*nmQFKLt-#$7}4?wNs46%^?^|4W3`h zeq@nT*4Nk7M>QG5j;>%;04#3h$7+f-{Kbbc)cGWRD~Z*y)+_skXO{k-v#W37pdl6E z>0(CwN9jdZTt1|M-a|GeBg2gQC~gQ}9-@lyv)X>vlG6~HUu+V@#r3T~Ad*~#OBgEO z^v|J?0CZ3=NCN*jgE7rq>Bk1Q1lcjpqQ)mf#i%}#SC|4DBLBP{{5xf9g+5h>6-=(s z4!ZqIfB&80egh~c^VeRAr7vZ?=u`kGA}REjknP{BSZdI7_DOd9S>fewQB3+7Z`b|v z*#4Vq`icQ$pWmQ0dS1j};(+Xv5ycJX9}|mzf9Uo8Ir|K1rcwUq#O>dhJ^V9HllA-P zB?HKcJfkZZU`m>Q9{YdS)dnE8wh#Zj3H>*9_$L79_j|}9y}yn5^YD>KraD}UIk^(E zntXK%}KMhQ?zvGrHj_H+cgsBbV~x1Al*hU{Y7g z7w`rWNC3Z6k@@(4RYL+T!4*JQo@&24-UB2+QDekiR{1ZYYe8NDp_yLMilSnnc>3nE z-MZ8KM-DtY__ZYSpWhudSpqqzd9L{_DI}Tf1vHR5qbX_vgp$4skY1RduJ?RP zdAC9Gv)M4kZF5#>c4m7if>HuAd@O90l+kum@;Rdn;o@|?wxWsJ;{uXZ7a;2ohRd)~ z`2pP-ioZ2mSj|T|XEfCrPOO<#Ki1{GU(#jP8S+LyD4Q8A`GawUI>11&*{pZlh+hJ5 zyTi$AEGX&wvmySf65XyCBp;oS_VDyYr>&%oQZ$_1uIKb(zEO=W$32ctrM!Crtyh(4 z2FRm&KO{e{N6O4Q;e2fRsNWFxIr;QGq&~Imv5xtSOv$ls96x~M0|eD&JC@8e5YI_4 zhcpXx$pP@BEC)0;gN7N9ZP)QWoR3v%wLsqOWT)&`&RMW<`Ai2hG#Z8ToPO#=mPue6 zC~UY?ae6YO_;v{b^8W-Z67&gUQ;aj*kTbw#oA@PiCg{F zA_cgMw#6oHQPVs5tmsyz)?xqA;BC{af~gOv!)CHtnsi97_dBs zPL*p2yWZv4LT5N6U=M}#YD|Wo?&Q+YAn^eVwJH~G^eJ(r=S>&_!y2>cB<a^YQj5M2RyVUaVlbV zbiPnf#LpdwZ1aI!+hLG%O9{wZ4}i?D0)WY?UDBGwY(jH=)O16*(MZtQDUx627qw(fCdA$duok1SDb_#qrJg9!&w@sg0ZUEi|g9NIKe=CgCj8zt}Rg^!lhj*L=hgwkbH+o=72b(VM>F7_Q_42q;Ki>03pV7463c9WT$_Gd%y^)Qn@K`rh zEXVL{^-NUZy|myA&|_YIY4Ug&RkZyzvqIY8jWzNCr#YraqviS>n5U9yHb7jly5fOiu3$|g(uA|CG>i%Q+Yl9!8e55Z(jO=8ZufS)qyhOH$^E$ zaSJSdHLT7vAwQv@S}-p6f#4Z6m>VRIw(CdbzqgvoIhj@g@X&l9ls#nM)3&nxqgqwS z3*ce&^S?|BcZeoB=K^LN@koi5_yCeK9{^)BM@P{x(7p5^3sfqYVF!@QXDdv%d}p80 z!$VI1<{CpNvQyd-E&_A}m-}TywJqa+@(Oo}Miv1?%Y4Cp%05XAHBDY7X(KZ1NX4{Z zW3?6E*AA6?pK(e%gC4-Xb}c^ouU#!TYT`&gZODMESAnR$qENk#Yb8ng&B2a1b<#SQ z2eIq17yi06A#N&>-LPO`+feE!bK}SPzOzrT7(AP`9oVW$NnG}7;{X#|p&Q28iBN5K zUz3G`_rVIP2ha!9)dDH3jLO~RfjXS)m=t>j@Wf*a9&3kX&-^zQ?RGcIaVmWi02tHX zf2Ju|ZzRVA81P$*=BHsHG=QidP*cC~X*uOF?5NPe)B=5;lkxcBg4F5QOQ=7N4=Ggc zYT0lx*|%V8bfjbv3|3Uue|1k}=9k-#?-CzYAMT_2 zGMP~~8#I??FF3V^bank>Nskz3?22k0=G;y!12_UJPdOP*>cd=M8F$xN2pz6lxG@@F%b60{AtA3`-L9UaZTHzOWt-OksFRxO`G^x-_emFVN*sM;J8mM| zyp_-u;fnViHiQ1=qu&p)Ca*gjbBu#IVDH$PZL2mFATnm*lo=I6CJmSL)lQ^VGcv`n zGd8W!#14|j23H;LQiNA&q?l8;)SzVz8qd`_!vit2$VZW?n+tu~6+#rl8lEl6BEv$C5`5$1~ zeU*;rl)wa|I8*F;#fnTx*Vw-Lmh6km8wx331iWY{xH}|vn12}lf+#A>G5G9 zgCCW3#ai9WuUroLeSpAD?lt0P;SmEdEr2$+Ce*?U*fwMJey^crM|GpO$-`6EE0rj{ z)`@Sq4_gR8XQg!zBQvLj=y<#qe?Az@CmMaKndt+f#8T>JP;d(_)GuH?UnFup7Q9m8 zWg=;JIGmqB(ZaQ2hBqMl-hylIm{(9~vu8qf?aVO5=XR@9E>Apwvgm?b{pkvDXM3oE zV_KUJ_3nmZo(`|&nq3r5dO)}FqIT%Z5xU`98szSiazQK3({6Hg0{9MONJhSQ zG=QikhK4(X8q4dHMBt4`RC?0K{x!5AN~+tMDJ|bXnx%RFh(k^|B5*2}QJa&V)n_jF zs_{sG?wFU7#q@jB{hFiCtxrrESSsW3N)S_MySXoj-*+T~hj}?`ah3ABwZ(bxn`!C^ zDV*b>CnF?RF87^;YTHx~DgJ~OZf^+hw{9j!BO-Ris|0b~EqkcX^HBYfvk4H~j4jZ( zKK*)-tR*wCDGIBDO=Rz^vC{M+H;>}ir1CW8-cDcpBuGriP*iS*(PtEnzTaTIw3-;; zfA?m?x157`Fv(V`7S>D*+6%?{hm1Qpq)VI7&$F!_GOFHgq2IWXxZqVe2S#DZPLUO+ zZ}SiMq};K$g7f)5rJq7?1qnYAtZE4J1jNco{2^w%+K3q{Hx*Lfn$J0MCG&OIpDr2Q zgF8UqKpKf-{_G|}lUbSqf9(t}JToTC_$?gVKF&;s`MT+ZQQ!`WuG;n`1J4Nb%O zrM=Fx4J5hwZ9xd>^3t3gQd*4hUXLJ&*-yO;Q{NeZSf!MS$vc)`=iO3nc)963TAEDg z^`5|-ggQU4`m4LNR|j!8rTI9Arz4pYVX;f*>#-QD@`=xNTKi@f06UHA?d>bCUb0j< zVf~+j%t^eL(`^>@C!+W~V5?m6xC=w80CpS{Y81k7N9XjPt#J@~mp@>KpDqdCS>*In z+*aHhfs6yveAFCM9|6eg6NM*EKOhY{fGq=;%mWe{>GKtg(PE~zy33-}gfcv^k;dWY zzTqeyV#qq3Z>-}4-?T;&!~3`GxEw-n`>u6JcKz^jyT&8K9L~~#-}OyD_@?@cdn<^m zm46Yhk+?InV;OVh6$8>P385R2OS({4T>FJg?T|AX8!>|qM}HGXh}aFT9tnK09W0c< zVbOeS4RN1vYCSxO-|D+=XeiOUSBGsUIR;;DE+1O0G`6vByL{T_a*4<3_^m5|m^XG` zy7p@MXSXlKT@SgnE^=~>hjzeE?t^l99U_c$^f7eYH|6AMTH^L&Qy#5W45G*F&KJ8I zrx=c?u5p(&C7cY1sq{6w;oS-npNR!a4jLWodMxJ^W|)QR#pgC zGZ?kiMa_xg6r&{k^&BW`jaaetSYlC7$%$wA2t#hcYfPs@!dP9l55`b$uri{t+WCcr zlF>;BR!N=CVewbbu0$_y&RygF+-59n5QQlTJyv5!krHKJ#@IcfA}#WneR*ge`Cfwi zz#YQ%s!;4WQ02>m0CHC_)*6R!dj^-J$M-wb9r?=~1_F8I3#b)DAuzdmJ6$im{vY%` zxDDLK;}#;hqS9n6xPlUk7KYDIWl zOgU8E6Hr`URi1QSV7wQfzox<{?akj>YpJ@nYKR$X+GF2Z@6UfgId{+61r<-tyPcrV zJga}&MSN9aRTsOXbb#X(^qlmkb#K05AS5y{*~#~VbDB?#x7T6&;YG>7kp>Y`mT~YQ z=R+d+6L?MZSgk^xMOZ${O0cl;X)hiY3qiI4=SL30Cs)`>gU6``x7#Tu z)K~RoJ>gIn36MXqY}x`P&mmlG8z`|ac7%I*siI`{jQ;!@7kU| z%VGC(#PUYG;(`TfpFtt&I7quQcPjlv}d@bx8t$jFnu z1LWozt_RsHjW_TiKB%_ohWEhjV!FxkVfd<6_E1?sCfNjQ z`556Bkn1;i*cNDQ7vZ?^QXS+F&%>?K=DJ&j_g1>2^JB4{L&SerUqFc46?1K9mxC0X z)5Pk9CR!^_cT4pM8?FbauBci)HJV;zb1g8f#(UY zBiWKSz&S$aGw7zuS$^4!u6M*?Lmt$clAYjm`UvcgbY@0ZZQeBIj0PEmemuwP@$*!O*;ctZzBlz4^3qD6yI-=M@lr&=zM$jRV& zq>jdKST_DTbLGckg;=$z^B^Cr43a`*2B5Yb#Q1zyKkYack9ty z23V&!NCTy*p)gwgiT7+FE;JXR(T$vXJuj|WiC(GkCmSuRkVM^WydRI*46Xf$x-O?* zNbJp>td*bL#dUNEr@?LS$;=*5nUV7+y?TN=C?*CUd8Gv4gC-p z?tH)Ka#X`clGxZ7ETD`)GR*ubGyja&%y@&_c?f|R=ON4=Ya?FpgO6d%owm6kayZp! z7+zPgrQR~7NZn4#ZjhfuA2}3wbp_1x+KrgCz(REGswr~N%ORN!B%?+Yn6!9CT&OQ;e?B_F$J{F?(~vXq^If7Z)m1c0pw;|$ zH0Q=~aam12_rT6)fHnNHloywFysaz4l_b|PU%HY4^0&wGNKenfU z_%}L)xOfbf{fTGOND>s_qkwHyJ-6u$86sU>(HOBx<$(UNBjxHtr~2(C8@ z^1@59HQ0Rfb=5eSG-vc?jS|*wdk1}VC4G<-e#9#Ss)aA?jNyHnPYdzN_Bi8ovk}_k zWmOaB@2gP>276P672b8jOm&h6MJh;S_vQWyBz!DY9y<9pJ`9HTu07q;i z;lmhcvsX}6d)tBMjog^#ciq;fhpv7Su(iQCE>dVe>*i24z-IdqaRq}*DH2a`qjs;* zHVti#u}K)=0(T@6NaROox2xcz0zp1XMC2lB?QIPIaihV8pgUwN)F};@g1YPjKsG^- z+Zv_mN9szTAtz~YYV#g;7|bzZAxcLV?wiSL@G(-x(%62?KhT{j32Mg{P0Tk-Ce%|X zWscznJ+$jd^jbF%Yf@7fw>g3TaHm{yYiljoy*1qQ{K(^Jjt;-Y1_S<$_>Oat;Ar2( z=xH|=VgUZRC3;;qR+>Wzu0E(OJqw*Sdp`9BZjzF}Ga6&uP3J&I| zj>%|qo?Q=RLoV>8#@|_^3@vj_6i)#n_b<5S&wGt5`8Ik7zeI{yke%ASCuMk>F52%l z&l9jXeaw+{O@`cS?f8f6mUqES64Fqbli=P6I6@l82``X_lbv3UjCNa{H~*!!tKlgn zUYJ{e=%bvNd-jC&D0zOqD+1%Pj%R-#nrk+Jp!Xn=rH{T&pJty`uZcw42gEg((Tbox z_430TMsfQe)o+x1J20pWR)ts8PFKbzvf-nsy)sUd%6aL{{ov;vTO)nUBx++iZs>9wcjuiGhsk86rniKAs)_p#e*oJH6KEO8x{F>%uLu+G3$?RBp3k z-e2^V{DzPtE{d2`#1g&aTc7y;zoKqF*r?`!pkIP=|R@2~W-0%Gu28_O_0 zHvxk4o?MxcQ^JNWc~M-Nt_sI?d0GV`ecoKcP?|k)wPmCsqq&$@$p;*9H`$nO9E*aU zSV%E;bdjfhOgHmlpZuGuOF-7W!X%*}ER5|8$(%ug$0S{h+J106MpXRU!%>Fm z>)0}*g){AlT`jeF>yLCN?cXm>_T-`8`v`owk)9=rr4|k{x~5q17Q0SyIpo}0X(zws zu+%#Z3+g0{^*^r6sC|Qq_3&&m$-*65f{#L#SF5h8GC|zG=#PruqDl1X+BY>(CTU`Z zQ%qW~tnCKucAY}6P16zoIgc^h^S_Ftb2~599O0`)!)makj>aHvRN{*qB*%FaKQK~L zmRZHzNO(nG-(b1aXkkNJPe&j#S|QoXaI$VoEdH4~k--gcI}#woAxSSVM_qK9Kq%R0OoyF>h4%DuqKX*{S!~%&3rUKl7(^lDh z$84z{)lu27*NaTcI&?Ym$Df)4`kH?y&QhJ94_fgSrVoC6p-?xV6Q(;w%GnS8yLcHG zX-w0+F8gui7Y=s4;smxt5qF4ax4(~(*REjda>p!??F`LUnwzqkjCb#Z>vAZ(s6Xxy zNd1>wGyOkuO?o2XDF6sxe{Bd*SY`nG;Y66{^Dhz%2Qnq}zVi_GKGhA+RVTrIv6&6w zQF_;2iGJj5{yz-(x}?_ea*M-^er(Meea$-N-b9|_p!K%rDdmqMX_aFgI-&}t!e2Ix zUC~GS$|X8?PU}1_;Ne$w_v>LB&Z{X;*LCK_qU3q^Zfa4bG4K!q9F;E;Z1A$%skv z=zmXN_&a@brrkI=fA9T8ERyDXriUCxNV%VE^uMuO2!B-=J?lfC`I6@&ER_H%_Ka(; zrsWr#lZ%CwXQ#N&4sz|~BTA&dCnMsr%Xt9sy37c_gWbF$4HH$7hi5VAKs-wqmjmt6 zWl7qpYRHK+KsHcGCmI17#fmH`+d45l+s1RDw_;MiK9pP@%rQ8211y9=n?G8W{iJf& z?&QNBQK?$d9Abn}t4Cuufu=}xB9g85(olfXS>1yIpgy&i@pT}?HXU3X@TS};H1)CVO`Ko6uPBY9c4V9A-GMAv(a^BuR0LBz&$3!$ajOP_1%E$d*YtoIU-fpxgnV z+=}ntnlso3+h9+zS)={pw9P62w7HqHW@K-ebbM*b&Y^5VW7i^g)!7a{Rxgg={6-1I z)7=92B<@<}CZs5ko62mnk+Kdl#-tQL%qq!Vl|XhV7nD{!T+ZQp)A`BXUCvq79o-h~ z!mrazP8QbE(roCESS_7JnpY9gnPn=(!=%sHJ{BMO(=9;+aSxfTsh5Lu)oqtJ@WG;n zd-ILS56sSu_GYoPo&exci6D>7CC6g?v@M|2uzFZ>xfgdMJ0uL5FkTv_ z?*3VfdVKL_oWkWGJ-$*p7S`dwV{1lQ^+OdFsuEjVJez4r!)1%4%f`bSHjfod7oBAl z$`j!xVJ8%P?j;SBFqemP<;srRPzh+|FrT()!AI#^RqnY`Rb8Bk=DV&LeBz-}i#+yn z29IC5*@We#2X#VyM$9l*+&ptQjX%%#XU6Co&PTobELHq0(0GiGaPMVR(5u236YLKy z5@3@njsHa^;#=0Qz)l5Z!NcHax?1;u$PT7YXT_)=0Tg`jeeJWB+8**AM4jjJ=k0K_ z7gQ2xqWYxtR)`H9X^`iv|8viXKe4-!Zf8hIq%>FVmr?Sc0L3|_QemPGMAsUbnSflG zH1%1xN^e~FZsV#M3auPqF7oC}&w!~k2k>V5G5R_r;lmy`8?klb(J)BZ({xqyI^)W) zD^HJCO$ETdT?yzuDYH^`;v5&9x{|P3j{vT~1mC>;XA*!t(>Ipa`(n8Qgxo?q(v6z! z0Bc*_XE#IHE8s{Vo-t4$+o7MS3-_R;s=WT1(bYzJJ@oil#=GEpHsA+Zz0&W4|Ls#M z!PAZ)V;NAzz6efV8)RKpxj##J3VHqNfk~3D;bt?z0pOCke=k^47hNrRG!eRAuK@IF z-Oa!umiAgJ5`!I3j_zatm{9TDa_bK%~ z_oQaPz#oMQxjY7VoA;E1AE0I#0(}^w(5N`w)ZbOe^Z#M>GMmc<&U+0l1h+9( z8xq)blRh&P&vN~P-~L24Iy7lG2W^1ndf%J8{qb4cZ8{YPt@9z2fDGWK_nR4 zf-Jy!i`wX@{!Cg?1H<}V-S`rKYU_o>g2XTA6C6kj7fFxhp4voRytXCy^uJ4iekYxb z{U8-mTOye+%5c-CeVr_8PP(eWX|L>){(6_EX6?-`%5Q-BsyC(kC7s^smHj_CdT{T2iVt_UZ{-wk3o% zkFwl)Rg;5lm&t}yFf2O;yzIs%Pj^-ZUm{B8;O4mp+0cY%4^7!rEcnU52P&*);RhDjbKd7Ks%OUI?f z|BYQVJM^SRwvtRNlgzoZUEaS)a5|~w900Ciw!O{;7CUJCg2F` zeolH=>Az(0BTXJ2?6R`}@AZ6C(snT{Ad=N|vK*+u zV)lF{761Nejrhrm>kZApL8 zKqqV!#rI<@2JSfv_&@PtG*)^SvBx}NFnXX7KHDYJ@wv~K%VAB|vtodIJJqMP<#Xej zc2KH6+}oSrpi-WeW@u$QZfk=*LiGqG821($zdbvE7UZ;S0K_HD8i0mCsU&7kI-Zi& zn=R5)hVLfhSye&rA23yp0;saX^4K56KR7TeI|8SdIjs?YzuVPz1E@|;FUMw22uiXj z{)2KbMs{_|^HJtXV7hXbED*=O8^{3X1~>9y;#~OkutT%Y1c{KuI_v_J3>^%dXF2*` zV91jJF4lvd7K`}YHW!NLonZa5tUBRUI?CtZ*y@N~^(+U{+{Ke}m@NXfLmEKVZw!uX z#CZ!rwYR;*sEy4E?U?-Pw`-4r8l<}`ir4Yg>)1~x$xfRZt|!T=@ivdcp3QyhGb5R+ z0>uxj2Z4u0TaAPV!;cG}4~DL~&YSVi#v$60;P%QarC7!r8{Jgqyxdf3-pUYjv`+5M zH!t4m#SoTDn9g}*D|G5U3^ni$P)TX7$G~u`{kWV=Gd8CtN_M`mnkzF5)_Sr-O3tZr z=dP-(^fG+Am~XrFpv?`NoZ5BN6oK_R&?Q|<*^Aw$+7S9?njZa4psDL};OtYD9Gm4M zX5D(_$Rxw3t7Z}zZBGUm2SQlr5s`$RH?{(zbB2 zhm`ZFvN_(|9=$f@_#I;f#Efrf|uXTZ9$+)&ppbkLH{ulUwY8Qwul@E{SK8 zUDUJtOE`CC({-BtmH0$>mf+P(Zp)mc*`g%X(ckiQwe(BS^XoE~>5%TG_PvMJQxVYb zSCD+fd{sgQDZ}-OQLM4i=pU!UKJAI95*K2Y#wuchCk&XI?!B&>^?Q6Nj#OZmz<4V^ zy_h@~0kSyBkN^sU$f={xN06P!6|6`q&6JvI+c_|eB!;Yl0eAAJ{qD6)9&n6y76GsQ zS)s%g&I*g(zR?lRfu>wf2UaFuMe;$e7X)Wu%X~99wya)(jU8HWTv~9l5oznR|38Gi zcTiL9`>(5j5TpfAI-yBXO6a|VAc7#G^p1dZkQTahkS2nJ-mCOp3_XBSrAY5pN+9$O zXYt*$e`lXL-}l`!{=qnzA*`$?>v`_``dl{)pHp%FHU1bFfBP-kt2R+ z!h}CcoB(}q619e07j%Gpe}@+_5gud;i5D%N3M_jriC%m|3UxKU-ZlVR*&W${_2OhF z7~NoSX&HLt{b&(S?myoFieM+eqfG4nm+e;`>{xR21Q9PCrT-qqgcs~9|7f|HMH_Y~ zlgIv|jv4F(F-dGOp0`aZQZ6>+uHlSCXGyN41+#rpUjQq393}83Cy;k&-0nxq#v9*D z|Me4Z{u_mOheFb;UOW#|>7Q62lvq0QYIt6L^p%*=NKVT&uv*;boneyyVYF8v|PzC+-VMZH(cA$9Afv9|wufA}PzOvEaUp)DlGe>Qo@)j%# z%R^RnUyMElvMxW!nkTlCtR4UTApMSSp$yp3DTv=>STO^)8zX;E6>~F_X=<qWtJsTGT@r02ry+kW*dbkV(S@ZKnyIeliHX>u%r3)FVrpDko>R8dN9P&Mr0NY!N`XAn~ zYt~~Q+e|-bKe@&zA1(AbBQUYlxy{k4)Eh*RfVv_AWR|O5uU=p2DNOCo)K!GuI1t$U zK}27lcKxc{=hCkk8@!nK*~VTIEa8gMwfC7$fk2GDrb=aeCGhiYy#5vL4;CM?mn@G+ z^K-%Zv61vkP(YY&CkxIh=5)Azs99Z&8A}l}nb+Pbd};Al-)5vR0d*OZVGED*t2o4n zy{K=SAtTa*W)h0$#L-f6MEx$5dF4*X;g0T#agNLbjT0cvkw+>C-R!7I2(@5kH6fm; z$15AkiTB&Pukig_H7+mm2yLiRJ8Ptkx#Z30hhBB!RfJU;BA(MoXPQ&{1+`z@$cob) z1l8LnKM&UJIKZfSB3E;`97z;-in&agO&S<$Q{Fr-IQ5?Na-#KSP4rScI`m7oF~$-I z${Wf=)755uF5s($9j)Jq?7AvMV}Y1$U}v!5#C7KjtgdN6)rQ1Aoz$ibPQ zL-Q{oikJz=?!6UH`pKn%Zh8WfN5kMiDMMk9H|&k`H5#i+nUaVK+ha)lFcL2%f$&@| zlF>KX?02Ye3_}g0?ha5Am76tEmKJ7pE$h?flX=*GhCtweQS z2lA%B4=W1`%2hkh#}AFFi}4~wrY<-M2w*bpa9hq@iHP+2QC^FocJT84V%MJ(G?cfHE36A_J@_bwrV>x16&%1eBS(CEP>2Hw^`YSocE6y9G!8? z<1cs9=gP)hw6T7(_)l!#7prxH4#>XUX|a&I!!3>q4g9r7uvW#8(O>P=2{jUFfU# zuOk`22h=7BQZ3a{gj^<{Wh$E;V-NgPjUYyL7+7;W)nLEl|`0c`eylp)k+ zMPf%dqk_RSmPkI+7QbAf7ZNDi3xRkC8?T0%BMRhYAxQCZ=%k`+?SXpQL7MW#l5oxf-`ay%(Tk69A83Xbh7Z?f z!4soMlp_rY4QpM=if&6e`pRSYEJRW4IKuN247JB8Daq1MC{A((L;I zsd{j@@c0&imuKjq{527r?l$evr1a%rQeEmVf?Xz+x9860rG_n|WV|vsqAw`yy{>`3TUhi3cQ!3*_hYbR1m6~gb zu=M2TmHdTl;L?fsdfeDmo~}09oo#aCdd<>kxzxz&T1=kPKMZ~zTEU%*!D1%8yhDFn zwe(0pNar22#LC3(wE$QzG&&avlPax^AgNOc_aoXXb4dD5;|^-$F*$=c;#@ktvyepwv| z#?f%03p5u!P&WgA(%!(K*x3-m3=x*wF=mTlva1p%N{D!=k8{V^+2qpyQF2&^XW+cUSIdO>vVS35L!QjTo zUzxGScV$+Cf3(=PnBgPNwafZcn=IJXVSn;f#DtfMh9z4d!VVIf@r&=^>(hans|)A% z0RgPyY}8ecW27ZfkJbn)6L2LRkewv5DzX(P9Hfj#u2(@U&~`Ii%<=trnURkFds2@&Y$^a zNJhlmBc!B}3 z+tKtS(DvhCTtPeGl}u!8x2fUEzs2f(r>VQXx!$>UD8lig-exh6%Fg_u4LRr|4s>;K%HH^(fgXa4}ydRKPY+j?vbIZfTP4 zq?>?F540e(jqpE=KZa7uRm_z*yVtXn8s!Kkt3@mEifSQT7!ibQF}kC_Or<|5nvKQ9 zD*96z!~baiAafuVm+M%D&L`vZ!kkDltP+$bYhd3^>6TeAImN$tq)OUzDjqx+MzV}2 zK7XWQ8aScnp+ltwS>S-0IVL?>hig$iCMKGfgMki~8Mi=w___PS1(G~JRhv}ReV4YW z;ac*X5U=OIV?N93JCDb9UQ(%*+H!1CY%yTEh)-VUf?OrUbx7glEP92lSLcc|#2!nX zOovu|51y;qooIrpMbW2IuL-w*t)U^nrK}mZu1HH`pqbA*ev*;hUDL zQDWQEwWRBKr-YeyaCyh2c2yg7&(c5`IF$@~&VavvQ7f9WpP!eMV06$S&oVzezpHb$>E6Z%#8P`24bu<*AqU@_5Qp2YOt52S^&2Qb1?*qLg;mwC=RKK zc4r7dM3ut^Z}}}y6D`T7h-Yb2Mt-c+?=0xL-mmXVkTCX{-oq@P=4|@r>mMkU;A*pQ z1VjglAkQ!7z^5W}sUQ_cHB z#;Cd2b`N?_qKD!UI)V&?WDhoeiPj<-@H@RImQu|rcy*}CReBUo9qJ6t>RF)Bv_Oe% z@QTTt$66KM$#6~Y1$F!5PB|Po#9GV(92ZuIHI(v!dFem`3utWlWg+ z|2d2tA?^yFrzAoNdSRZ0jA(|jDt(Z$auDYgg$K-Qng@S}7$1yF@weI=H{1X8p;|8< zBpVhOd|TLx5GTqKd&}!+e4d~{5gS&EH_-?n z2@blH`w$b98O>g&hW1afqMyHKP&E{kPn6}GrXTN9K<7vD4NG4ud}-HC4!*qKO(FzR z1l83HA-sP_l#k(T^#fV3zu0JWV2&Tvr!|SfOHgjjVIXGvOu0{F+d$te7?TYy?nxjd zzW`KIYqh^)sfXFf&d&CDBdo0QyS7yY&xTs*KjSJ+eO98^5PMuk2VRCKHY#}@Wq9Gn zL?60flN!{VgKAVr)KA%NYph_QhcyJ<1xBh5h~p)xl%Kr5*)&ooPm<3CkfCLB%1_81+zRn<{TxN{qwQoaE6*$To&M}4)+KCnJY0*AZ~P4kXx{B( zlw{Jy!WqueJ?20w&-LaetUrj^@55g=DpsVU*4(3T$NlV3h9AgOjZ+9xMEy_xD*M)U zw@1{2;l!^hf#&lEOWOJQvL?%t!#Ab0;z@2V7rrSUP(+=6SW(P3(TZ7R zxqe9DkW6=fi}T|`%B*zHLkb6a^W7X)wo3ylpKO=$9cneNxkjaC1lD@l%p}upv$h!9 z4xw}>e@Z(_+cg?(Q=L>r#NY`39L9#YcccP1P;X z>2k&-5C?fUQD6|*+^kuLMdRO4j`96GPJgllqsXjCbHvw2S1fxDH}?FbHmt|DKnp72 zQAgK*Z{b5fq=w+ciy=Ss(sqyq5ZVZMnnZGkk(6xq8E@Z})&qn!gKCsBd<0r&{pRLf{O}63>W7w1^*PK+#a*UwdrTQ!e3vBG;$L~MU z9nYIx>yBRr@&xes&4aJOMX)52n97i-`)@~h#6x^Xlvt+#L)7B*%aOHURhYJkl}CFZ z?=P_T%E8x1^A2i#c(X|_mU+(kmBow_=$C?FW6#AWMt|ABwS@5rj4PskCRuwKv@!h0lx4tmuo_XV!ZIL>RbB+*8E z)97v-w9(T36&m&~RS0@5%iRI1!sPju=P(%(w%z=tNUO%L=DdcOx%Fe zkt|sp4(Oe>ZZaU`c!PaEl3co1(-y-D^|rxRI@~jQiIQ4CUV96)%)Whod7vh2SlC;I zqk^F>m59||OqR0?Bn8ZNAulN{X?2aH{+6rbSV7NMTJ9R5@Ub5?u+jgtCranlLz8N< zZdr;Yf#(xC_kN$pZ_B(ZG&07vi?K~RN=j#+w2+v8+^^ThT#6Esw-#G{z`x+%FOYH$ zare3%LPgZGTKGJqc32oxrK;F*sw^S;jP;%}5OeTm%CN@84_ zs~PjGNs12NmHK;?wfn$)#(r!}+c#64zDXepRpO=hLlZ%jT9+|$thYGT8!?Jg9vr%+nbqZJ6sbq%FYHiVxv{TOp-I3zb*sDJzECz1TF_jp)Ty8DyqRNwsl zfv4FoLH*J+GWdtp-Zu&zZOMuwXGpN~wPbyk^z9`F*fOT^HnrsA#zfTs3wL1f6DY&9 zEW#x2#FA8Mn}%_m@t*mQ1ZEG}$pgD$wa^t}5y~Sk<%ip+(~HBi|e& z0`C=F@E(ylQ`|X>5w|prfzwVTI&%uHh&QU@2d!}QLT9$A$+hZcC?;2;C=AJYX3TIy zSs5{GYv$KTLW%TMP6 zi)Rf@SngYIMv&6Fh*{_dz2!1J-Q-m%&iRJJ(YnWR()uFd#dPLbayofKq!TCVQibkL zr}`%hndi8BA0WJ4DVlrB(+}wUbej)Nj&zzquh)|%m91}!C#C=yT_zfLwK$He-2U-R zqvi)fubsvnrvzKlUjp9wFpqZ4I#cFltGW);e3vOY*0BrJ-O=khEEgyf&pPGCq8gnM zLJ9I=q}EdlHR^7%oF+QOg5>GvpxRu_4gU0?U*jWZ7SxL7!Ez;eIiFugeCVS~SGn>a zCXY?v?xG*-AOKt|ll+~RCsK-rHdcnU!z-gf+VUwi*TW6G*U{9GJ+yMkP65O{p~1S* zuO3p9J5<`5Ej||FF77`Z2xaehF`LN~F4p12^RPCRXo9s={6Ad;rIM;C)wFgEK_j!* zbXt1{;pL16U*EER955=||GkdH^=-`8iv$=#XBw%qT!OLDUNSrBHwM`>%(%5^l?b?S z+}+hpTITHDE)>_o_is2L*OKb}Drzq9RCP;HpO#tTB)0X*|9Wt}jH?;hGI(QH)7*gj zT5@#yc3DzvO_KC}YQ`1CjF_yssKO;)oOJt-C+JB!n_j2DzCTw?fsHg1b}4n^&I{2e zgVi~F5C!25_#ZIUC05Md1R=7cw+_<&2HP$W{_dB2W)+Bj`XKb2`bA|}HP>kDE!gDT ztaKX93ks_#v5Cmk`_P@@NBQXI2yO*3$cT_ZBKL%h$-lRuMSR(aP4k7v(?=}5|A+JA zEh~nw%sG$X#=HKNoh=uG0}*(D5w?Hr6wDjkE=Vp{xHkN5eEU}qgbHo<-8ka4*N0pC zbNS|39*LN9`$oHIdW>@t*!J!*K9SvjHK4S8mbq813ms=1J6Qc*VvW1{178+!2fPbE zo;hWm`9j_^2{n4<4y9c&H?^^y9MQhW6c@KHKA4UFC*7g+G5x34+i(hwBXTz_qUEF^ zSyMX;g@-`M|DUnK$Ex&3m6}kBVzhG3KZAXDMhHHj_?blcH0Zx@Z~p`Gy%h#t^f+1@ z_kaH9zx*I=0RVt}d~ZPijP~6pE_4K5#7R)P1#aFIh5E+8XFGEA+-=p+&h8`n9H1O+Yu{~&(c5MSyP7-|HVE{& z;~8sF+gZ*puTP{jjcWZlbrDFeMxRIm;pRAjLh?QOL4M?E=elm3LT$-t@A1_Vb4F+P zVEyxmgz0Gms(4#Gza+jrqLfz0qFQTBF~k%ZcF&i3paSl|nVc|BPIz|s)8ho4`PA+F z7p<}&5v{-wi>luHREN+ySLM2G<$aQ{&oxe#A5F{Us>$r`%oIW(?qMo+Dv4#{Ng>2pIm}nOMDjj=VDg-X;0pj#L=<9RE?aD=wr1B;wV8W>32(+OHPp z@wf5MRC|Em%ocD&+4UavZw#iD$^@QWrp65U-I)~0xb!~^qzIj+ z%v$OizI{0N3=|ZV*_NX=2Z6jhTx4*aWZ&){X1he0wG)ISV*LY_HDt z>qcI(cg_AwlOiMjurz%mb@ zgW%INT6=$OLV6|$s0BF+Ps+w{|Mq;u`a3#;t+1EZRMwZJyCi4}NIXJ}BN)b70LXFo z$4#*)#`)*kUoSQOq(!{pqQ^LukeYJa9`4Yz{+43QZ?eTmEnwm=7FREoX3o^jS#hHX z-5wLa?&iNE5XTX}jJo!&j2D`U1vo?wnf-rX{O-$>_)gAgXl-=E-vdE>B(Qnin(uRb zm@4g!i_twfsf33r*;ds3LBGJ~r)PV-HD04PWHhifRjolBjcIDuJ56@A*%xb$w|&WexC~$RzU;b^ELZ7=IsFw z-1<1{DO|~j4;F53>W@(~6K;T}9(lw^dkFA(4bHM>K%dpf?oKTMxJl-^;0bWYfIDqv z3XCBGnEGG=(<<7-9|vv#J*I&>v+A8XG)LkjA%UDOL~Sq}L3{)J-ikY6Czt>+2S@}? zZZ7x4_P<;!(l`a(3FG&;&Q-)(RjlIWNiI58M}b#b-AQCfr?0t28i82?C?LNLC=O-@ z9NaQM)Z~V+tQ77*qj!4w2PA`Z7>_oZfY}Oi4ev2l01Jg%97WKIr-ooUknqTnKx}<4 zei?oUYKXVQCcBe-(mJ-?XC7w4p9AFayP`}V+kXN`gw*Yov-{4~`5`KjZ?NLd;!C$B zzmm%zx15Ksp577iyX#3oIs1~&g<&)9>WJzb3~186$2Mhm4X*nT!CB{s($r15qlBG| zm^rte*!fI;-BQdK6kLEUcFBcd>X-t7?iCaqC7VYsDNq5+CtWvv%nuGvN zS_?V7Kj4uOW3d1Eg`vy=ysNdVkw?VC7x?-YxYIVHKYhL7ftU!@obAtu!?-osNFbO>4V&oh2sw z%N^JHf1;|p_2K5Dc%SHsk9c3uul;DgcJ_1wN3H(Vgx&iMno5-y66+&IsJS7_nP@%! zYOnO9_cl1D#pE%!$89jL&1Ooss~dI^hSOc+TP{7JdLxAa=;J*#rgf3~=leF}AH~7` zM8aZ!AECylH4UshhUU=itQyWm2%zjGtZ_AVk zOila(Gk&}a19)8k;kN(wj0L{tLfJ;|`7Zd*B>XL%!}L*l=O2TT=Z<;+D}VxcB9}6ue?0{Teh~;0Y zn|&fyStj|swXT9LWKv)gt=*2LP!kLRk(5ZSpW`OkHT}E5sLthrfXn%>khID z(>R#>sj)-j8!Mceg%?QG#E0#C0gK42&nhsO!nu{W+=?2o`p{@OT*ZCZd0(}g9l=$V z<~~0KtYpL+My|{9AG@NgXhI5ZW0Ac-Kygj8@#M~TVb^lLZvQp#tU|?#rAyi1=r|Tu zD5TQj5&PM$>T#en7wQqjqQ*mL&L!?p_!gjecJkncA5HD@d5_OlV#G2*Kf>znqy|=Y z98Rm$djR08`wS0827IfXkLM^7)k~DlFX{A?KB(!eT4rp+!n~$t>CevJredx#y*XL2 z8I-*8>B?1~wM=UyH=5mmyYEW1tl?a)eXGoiR;-}CzIk$5-(6NbNmjtQ@FGE!zK?Fy zlCw?&2j3l1GSnh45HKHjj2fLL+!^ida=d6G9*jFIRkN6q`B77M?QU%jZ5HcH_wq{C zvo>C-rJkY;t#OZ8RZF8py8aJ+xbbuv3Ra$}D=~xp8YFV2^a_hOR! zu+_8kai0MNX7%#f{Ibe{lB6_8X)&AWvvOu~U7>P2DJ)rz@jM@i9{@A3E4I9KTkj4r z)_kNx&Gm6m1QdwHlUerpvq*LsaaY4)l>$C{$YV^KXdCS*fh@c-T>S}-GSd!pO0}ti z^mATDkqEzLfwAACqjN$kP44SrL(UuXn`LEXo`5!I8`yLODl`vLj#a*B=;otXavetq zsFoO5&V5vr5baT^Fzk%7eG!ThoLg)^T^n1=x4CTMlJKzq;X6lEt(EkuDPKrJKs&+v zGU~&p*Ae@fh|sjJ?Yu}G@rJVsn%MD(I!S}F@yF%UZ`V8NU_|<|EJ%-u5tFu2P!?GZ z=ACd`Y)Dv?QDxN3dK8!+vMhdv2Fav$kOqYE*>@WXXODgs4%1?uFAvh31XSAnz;fR{ z@BxOJq-ufEH5T+7qz{bWOZLhoSk)j(i1#1FJ|KFOpSV)g3%UWA{^ec|V9hi=tzW89 zCyFa-c(>{)Dg+{ec`iLxk|4eXbD&?#7@Iyk2Uf;vG(kWbxm3>NYa_&ablC1hdBz=~ zm@|s3F{z^i<{_CW;zU~=B0Z%5z4UBlM`!E`Am>qhX+$$pGtR#B*5_z)$40<2I=63b&3h~YXu=18N5dMHbhOZh z9v6k_JI85B8nEc~*!UA)rJjii0-|RK7vdxYk1Irh;&MgzD4TLy7-y1;>q>e=P=5$i zqi3)4H+<+gg!|xXyg_-#(@(?!qgvL>uIwHoco zfNdf=9|&RW>V&O{JM*;lxbDq3W=MLgI?Wm5zYOl0&fR7ziehfHJv5)mWDq?RN6iG~ zZbx56I>a?_TeDqH7dLhNiN^7Wc#0x;`pDGI4>RY( zEu4q}3AI6|ePhL_zxH&6QH{ul-zhu>s`ia>DY@6HAnVA^>ExLkzP?2UFI192|2s<; z9$1>M-Aa+^>VV6Q-XJ@-pG0Hzd0WUm*m-j>SNC?a*0JLfw!-9E=$-)^&5x+@TaT5M zF4cRI?v9G&rk})(Za=7VhhM%zOh1U2Ami+*cAIrR?B5%@8o?JnwP6(}FO59k-_2#_ zQnhO>O@5?Yi438l@Ue1t9{E&f(I4kLpwdT7U7^L!H~(-tY_kLf1&Nt8t$!upYCG{9cEm=%N4^azO9SpyvgF)gF~+?C-^U6uRW!Vr)f zlR~5;tj)CE#476&gr7yr&)%>g`a0Hhp5gD73+*(LwwX;QS@YXT}Q3`yH@Z62}KB z<*qu>U$O&JKO4>N6O4yOLrpVcYj#)dcUBZ8?yseZ`r z<8Ft9f@gS}jk{Cn?ExdYZ*+cA@0pZEyj75-iOJ-7kg?|YJk$I)ecHu?!xzWY!;fRZ z^QjA|oiaJ7m3jHgzft~_ri7sqq1P9^XR4w`9cv6VL6)zsh3T%PF(DKBGyTiPi(xX1 z*mpc3sp$Q3eQf#<{{aef{x?v_0}f(dQ>FyH>yX&VjZ^)2&d3W>p0<3`;_Gz_m>|{Q z;FW7pA1ynKqUj=I6y>BDeF`^7a`hqRib+<29j{2i*Rq(!8@}r_`xfybEK8!?rri%j zQz6@luetFj+tm{muw3*HFVBjO7{M&B`4E{+w`zX_D-`W$NJnB@N+K)n!E7LPmde#0 zry=Xj&;&B`SCgMOOAFW_Sup$Esrsa;HTPF@%HLI`zBecFdo(5qR&y(GP}4R#*uWED z#h^IMQ!A8d6^V^&u_3|0r9Wc7!_H1ZKD@Q5_0{dohB69N%yYa=mJV^<9^g_kQdFBW@yy*)jgl&&9KR1uQbo#l=Ij|Twos3hqxh8I+7@cO7 zphqx}IEo@&eWQhKH*TQg%j?;n`M7#+QaOpu4V zS;eIx*K|(J->ES;LN-{_O!57{+!9-yqf^XX0PiDF0P1qPOIKb;QYM1LT4%2wH~+Ym zcqkhH0zDCIIsg!uMsNoNnr3-+&Je5Pgt#;iZ~fJ0T4pE&!fn|8l=n$^DAXhn!o;N= z3W}8bcs%&+iBG2x!k7+xQm#pGmr5=!PTFjJ9#IRImMfh7drVwR7&K{*<+<|jdC7sg zItkl7fjqCOBXho)=y4uqke1$9Uq1LE@q46pq!#<7$)|wCVs9q4(W#fUbO&uSUp_j| z?t9Bcnq;_0iU>MEi{y~mG%ot1l|TS#FwHA$Ce;h3ydIzq=^&vG;d4#7qXCt>pu1d=B5vncJCvlA|y2!drFZQ-`%zx`|sP)mN?T7WJ9@~Lo73KNm<_OisudSV2GB9xJf@N z5}q~NR7Nk=bNJzo4K8HE)qaJmr-a7>#rcVo-SIYWdkG<8!L$eRy+u-`HQCl#aJFq2 z4;kYg&9?f?fTth!z={7(U#)dw$VssNJZx%QmmQ)&S*g!jLc|U>j3MXJxTzcvG_+)y z5iUWjZtJi^ylmzW(YKVBw2|+W!>~_I#|t6|psNqp_xR3wJ-gm)w5HAdfquT)oyz+{ zX?pt2UCliUTUD&8S?6-w!Q$-a-hhwVkN_*;hmE>Y^LGm~J9s@{b&vgts1xV4&CTXK zpRVt&f~|D@zJg_(9VPWjaqy%SBKz%I&J%(|bImV=)6Y`W0M>d{>yPlPbY{ZLW=I;{`9<2 z?^z5de;UZw;bD}jt@ZvBCM$5?D6MmP^U|l5?lFdc)DvZE=MYc3=rOw(x<2<1cK}FcF_1cf6pD%P0jX6I$0VVQNORF3Seo}GG!^%h`l3VJ7=FL|{QCYxCQ<}gJ zSP3gM z{mO$i1N_*SSR<)m*N|#UYAJ5KKayrl^EzMm5Uuh76Jgbv9*MvYBXqNhnyiH_FUJx; z^4lMmf4|hFMEDdBsUkygkLtDVzy0{w<`Q}s_T5PM)yOTVdRDGMbHD7QyHT3vrMNQBh)GyNO|8m#&x|-lKnL?r ze<`O8<+sD={vMkzu&IjE5Xat3`VDvnZF}wBkma~@G^ss~92Q;~(|doRx=y|1I^L&7 zWilgQ%46Qq3<4((rL{Z`{P)XtJ4Tu7eFb}4a`bU-pKUM?hg4x^qYAvE-9scl*Hf3N z!hT4;bbd1O3-14P_pvX1@O56(F03W@nzI6v<&LR4JfLiQZ0PqH^tXX5>o*Z`AqkeX zX1H#=2{SBQnBXjhcJ%(XiVqV+#gwQ zLh?283w})NQ8lTmlHys&FR3#AANHgCANfqrNpu;WS#VKD`k}~ANdyv?re3n)%&wj z>zx+XX#_~ozfiel{=$BK(u45utME86J9*6C0K#DNgU#>DLILa{p@bo@zR+7}JqFi; z$=-~yZoN9;4Q11ONp%|43 zu+PXPrh@uLL0Bnto zinmrk7yY_Zxw=qng9#{M?6VWMfIpZ-Z#;zXy4XMa-jX*()YGGeQVoXqQ>jNBMmKp@ zLjSmZYo7PYWvXGgVQ66(`AD;g$X_OR_wGDS7Iw<^3a4Ho>wtU*d4p<$@EB_3e+P{I z4D5@mlYtqdv16c!dR(J`FN@6`E4=}se}Bd@Mff!MdU!x&?euj5)*jxf0{e_8mqDIs z^67{2F7i%gJo#Ouw^W^aX=~d0*R#F+xAe)cxeRsJCH|hx1I_~FAX)l8ySk}vkFbLP z$jE9ka~d@M+i#9pB5uSl+&}Go7)iKb{xY1hr2Um1F%#@^vuRf;>Q88h{(m^Sa^3cs z9!Z788{lxEkFXe2&X4UR7#4bJ6r>~j4Gak~lbr`hx)!BNmEMVozSg(l_Z4-HpM3eA z_Rx@Sb|_q8Xc6ZW4;$5>h244I+*y^`fG$tuS<%6o?<)`7z{fRjCulJRfdOf5L!ZR) zMp!OwcoSoMH^}ePPF#r^uD#RU+mZL1IqmGTG3u@N#{F3>h^z;S)j%UT;mF(^k!h29 z%$RmE`s^~GBw_H7@7Z4Gj;JA{;L+C8{PZYR<^D-(+pv4`y|(!6r>QCVNkc6L_I1;_ zSyNu21R9(~Rr|KMPaSt|O6Twf0?z5CTE8XzFpd&A{{A#S+($>Vy1RD3ynxKl{{^>1 z>(iB~K+z`hIAT7xytz{)L4FxC_GN{s&Y>aqeGfeyhvkac2_Vyz5;%0xTT;P*O<21` z)TBc;85SqRdm58r6WzE-y)i7)fN>H?a4DzAcM&dP@jL9-u3ykt7Us;41tq_hvE9a{`F`$3WT7T|GjZr8P~Y!OZ{=F zg)(0~P=EU**lCWUs(r*uK+ovgR+z{EX-Xe+WB=7cq#A zn3(iJ)AS8_i3R1KCd5T+TfnrP5CFMp-nmUy-t3Qw!;(+1VL*Ay$0_*%_SJEoLBV;R z!It4KxOf=PELSF;J|Aj23M{jo>G&stz84;jMpHMc(`w(vH&>k`89ENKtDofL^zxf7 zD~1Dw2sfRi{ZWBK1g$0&WHF z^%a6)qS(*yr{`81zO=L}4R%sio38in@y093%e+kt@dX-DNqY8;UA5nK;N+r9Cc(_# zg5hpmlmQNJb(Tji*Obit#W0m5S-vl@T)pyoe)-%}GIav@CU%}6!BjwiV{C}`PHeZy zXIW3dlT6xJ1SVhL=-?TZLF!#Kl>tTiegS3I7IcRcVhh&zWP@IoJxe;EyHUC1<1TfO zx6ESJJmRQt=!vlfj_I>32knqBWw8J!gW;Ru^-%@zGk+VxK<76ti2g_v8z$rm9|Fsg zysKWBzYe*7&k**scpmHPbkX9m%#Z#y$dU}`v9!JiwxG9OEK1~?i%FfvV_>%s18rWOW(#$I;?^aB>#j>8nDLel7W|O`1t!{qm zx4d(lpDGbaD7SK>!=m&3fe!d!u0{-(SUum5s22{sX8OKCJQQbehyGjpxiS8VO7rlX zWb0ljkEN`}=@24NcTUud;_QJL^kN(xP-TGX>rE)twtS9EX8u~0+E9;p&(RB0BFkpS zc+XO2+f`oIQmb;7XOgrkql+*T3C~GvFR0_x90*Hmi>c0d4KlYW|` zdV{o(D|vqJrBDL1`G2Ul1)>?Ik0YC>WbpCwA-IFMJ(wTtv9FoQM}8ui&-M7-g6&Bl zI%aaCK)m3Ndh3~V*Vo7!$~zv5TL}VFn49=+LKX@UBL)D37$f39!sD?@@fgd-Wx>hUgH(4=B3EwZ)&XdMU-dD_Y|VQYRXJw zA%~G9)=HfRz|-+KUPL3Sj(cGQuXeSw@5gfD(gSMdu_{{EXEs?0-f=J;f?VB)wT?0+ zYbK;wWWxzW=_=lcu!tAR$=+b=BGUQVunMJ+Q&b{@Q1TPNoOq%n%aQ$R zQqitcG6##Pj_6^nEj3*5*@Cx@LMWGN{P)5Oi)y-{SQXJ6(B3)Q*hh$y_t(LVyTWrl z{|4IxetOjuRkp>LEBux3ntDQm4P%*iJOjtrSHV&5<%9CFzUn2oi~TS*(m2WzCN@z?R({C%6_#&B=T8HZY$u8wY(@`14O$#HOXbZ* zy5vGj83a%ECT1nuEDU=_c2me5YHOjbz_5zv#L_iufdItT5>{RIJ4-68!aM98HGbfy z7s~p2SabT-=@MS+rSw{{9JoA7oeXbReQV1^U#wEnNtvR@})p}!xSa>{b@X_6HV5AZz8-lDWIe_Kc3^s{SB-_7nb%nSuOk*YUn?>+PlDevZX{kiQCYf zGAg9T1OT&HE4=&{LW&+^O%+H60x%o)_~&2re)L1Y zpD}QVS-AXfdOrf-sug!Tz2*GJ%OQ<*kP6J8`oF)WFNdb6*4t}`(K?6-fCdDqs8uQqmEs1)ZLq-C&i%i zzD=S$-ojk>xr{pIoE)*Wkt#s(2*=sR|FS6GRE3VJJbb95o$ZO)3x8@ z-nH~;URi6F-3-)NYa;Y*#&0Yn&n=QRE7h+_%nj>I(|`hSlGS)_-KN3sHcy!1-bKou zR$}GQAfHzHe4czj#(IgM&3GB#oai5@F8rBMoleDvsHtpACJnxtAqUQdDXDUvIk5?$ zwCQoqUJa>d>C)P{QP0|cOJL`5PM2v{cHGTs&Ic5J`2Y@-ubQ?qcwG1~w|WN{4<&Tx zG7woM8^|+6}D!ihCs^<8Spl(|>_~-|PHf%>tMao$qnD-ZjGfjXBW~ zpw{d&*tN=BCN$54pZpO@R1-bUNp)WD=K(Zl>W-r%t>nOnr-mC~ZnCEB9JKXnd5}C| zqP%r!euwmK8tND@zokD7?0qxkt+Afrd-2N_u$AO%mgq$s6-m zU5NdC66e88!u3ySdXLujfwtiP)81MCMYZ*RTY8W_2q+yA(jna;sDz@Zl%#-^12VwS z64IeacMB5I4H830cMJ^zO2bIkv*w(0KleHJb044YAMoXunb*AbuxGEe_sm{vU7zdS z(^*p1lQTnoU1=cMYkVY0GPjuJ0xX!XB++~kZ78@aX?Ak8?3_})XIU{*(my2j+^R9I zw4TH8(sUuRVx`>rW1M=e2)}5u-aIAya$O(gQ2|X({2f&<$Bezun*84V)>;YUm@sIy zv%kxanL9ZTBK%jDKF7w(GE6lSPUL5Ii!xkbtDO3XMxw1e`T+Vme)u*+aMn`VQl;U< zXz?%5m#CAez|%nT0pgeW;;km(H=gK#hzz5Q|mZnZdOYy{~qrYwt-G&A;F-~ zX@qrAM6aMN+?ZNR?O`ELMXkqrk{tTP?_9>NMgV+kN-b4pCJCTq%QcB6E+A89JqJYd znu9bIzRLPf=e}=T(Q-K}do^*d2#GZghfbz|D4i#0noY=QLz^nb5|C*{rd}YXyjDIM zyfYph1ZLMeq|*qN2YbNBumz^Q)QR(t$Q874%n@K}V8Gk6?Rh9EWHZ4ZYxqdVN78*U zh<&<=V0iPNF^$E^$(4{c;+K5OkOKD+>~6SKPkFOR|FD3toRcYYX9p*CB;BC#OtM^N>>=BaxxBXYn@b?<*AmWF4U)q`aC7NBkBr%0^UJMF z&88~`NZbdB-M6QoO;U~Dv~8%;+MENh>$Fo~$t;{cEzI~KS`g}lbszQ};DPHI2Qp6@ z(R85oGg07dL4kbiTBmf7W&4T>^q~+sMVj2JMg<&5-wIf5N7Xsc6{j`K%iO2o>Gr@R zL=1W@PVCu`x)&)WedME{eYm18gX$4-_yUGUkU%+n%nDWRK_6z!53@+E$~4QJhXKJ2g_V;<`(D(K<#N-wI2^Ai58?by7om&Ss7%)r5X3m$RKZykL+zU0tz zAt@~S!Ao0b(JsuxsZksicVM1Mmyl>k`B_+n=(3&g#DiwI(u|HPwC}c;L_wjr`IoU* zGrO^BBJg^rSI?@HWb6qpjt97h(_ZcF4BR_z6tc{l8lSKxkQ}cu^Ru&Jn-X{2tSuHL@%x>Cy zl9=#^AUBCbrv}h8eG8xPrm}vSb%fsp5tn@%xSnJ| zh+XPciVv!fpCg#)ZMa&ACBCO!o#R2XX;08>+LTEMPYPFJRVNHcmeP<(W(dnzWM>jsDoKIC<>S!#h@Kt7pT-6^WKCNppmi@Kplc$xkIlJ z?*~Z#KB1N!ppxB^7>7$IgD*G@zZ}z4Ey^!^xVYbY1m-_0n)hnV&?}~X%_%1-~8~AN2}-oI1wY`RawBLup*FJ1(S2?qBGyHW5BPM zO!h|!dP@ibJ~!0ve@6#f9k;Wkn_VL*uF@IyPTu9boy&g-TTf~WU6u+?m42n*ia>Q( zp9h}j&+mi&C$vB}xco$FnF}-eK6=vCNAOMS`7W??IUf>lw*`wzWAge!2^EG9hKLzl zzo+K_D8=8J!1NvVpvcR6n+!j8g4t7ZEin8>03$S1Xqwt%V-f?Q|8Vb^d1LRar;J&M zUk0$9sKYOpGZpzg64!w*)>;U1tMdY_GJNKr7QkcctAL9e@nY-Jyf}Q72A8X~-QGM0 z5Ss-jzwMk`L$JM`88RR1p++9;9_v*-iau*zFO2<4E=$>s)qIfo%)G9D3#&Fxp6cLM zrOcEUCAY41U%lril!AxJV!yx3fyyD&HhgWa%S^n-Rp*t7v?P0DRVQtom$d^zrm#oo z^QB3O`?7_QMKg7o`nK}QhAa#=e7MS2?($r8l(ycDpw82~IsaNZ*#a5g9u+)46&A&^oh;_sN_cf*JBi>iWzd5Tb=I3pYu}pV9dp>ft0!8o;}usk zwDEE{S)#ZB#e6QCIF8MwEK*=NTz~%Kx8YANwn7`#;pU0z1RwEn4kPf^a#dF2p3NG* zkVM2xdKUI9xKV7$#ue#xxUOueUj2x{48z@gme}!gGGFcV>DxtXFiW@5XdVB<;c1P8 zX5&odtt1WAy8*VYq9WTGx+ZPNrvwxRs>}Kxi7d%a^v}+bBFeJSk^~QTT*XZrgX5 zAgixgU>V$Q*I1mnkjmDhR=<$kpdzm)Kz4_|+D|PIRg?{qA8{0TlvfwPH@{QzRREv4 zm|2l8;{Bn;8@#o;5Smy*6;A5*opk<5&>>gK4G{eEVZGo3A2YrIlhpSQvD*_WE}e_4 z9o^Jljur{VZ!{%_SbD=h3Hn4FDTtFu?Z~ZaQfbia^T!A4-6{I`!RNaoJMQPZYe7fO zUs+?t>_fvzA9u>0d}ndCA>F7yy3yu;vgqNf0{h~5o^&N5fS#TYmNG#PrJ7)BNb?h?UOJ zP_&4m9tM60-M)0X$Q={&0_ZG@_WJOe_`j($*hJNI=>U1^I?!)zbD^oTVlP0Q72*<6 zfAf1D!+OK4xs1?PWkAegsY@aLJE`Cmg;R}V?G?qUrjTZ~Py*t$hk zpcJD!x%=M!l6$)Yb^ml)fNpV5^BiP#!y1S-ad*YIyB--J<#ME!+Cnwyw;n(QA#629 zAm2Un$5hqlI=+YWTjYfIA3pV(J$}6?PL(7trz?eR#{BRJK3G)u*oBE7$$VH8`l>QKw}tn74W-)0xhIKN&TV8$&)6SMlyzRZMp+VN_#tR^-2rup?nXAi!+}HO0oWmpP96m8YcxlS6ms4ZD7x6YFKd`jBa3G|& zq~r-cpWU#_jQ;gE`$1hLW zV;>y#Cr9*Dp6VH2xX6n-4B{vEeD7FIIIfh5*XH5cdsH>Wkg|!`Nn|K4$+wZ3=CMth zsC5K$qE!jzMHGmJ1*;f!j3XJmt8Y-0L&TWcj-j z?*_w=k*7z|7{Na=Y_$Kuut`T50u!{xUB#YkA#cF2vCvba#mz4NX4n>leV7^tCg(Be z4x|Qlo9IRSt|;lypG~kY($X_R+M?bLy5&3_Ob=B33%{mxS3(fN^lyHRj!i-njJERD zjWi7rv{KZ?Stq68OWsBS&zO>L`DrRC5?-Lw7RirxvQ~~3@PlIolezxJt}%O6Dt^0@ z)Iq9C7EU5Ie}!GY_N}*U)B<3~1wi#iwIs>+roDH-i8q3Q+FzhGyD!Fs`*(P4sH-IN zL+}Dd2qgXwcnz#YD*xEM+11EpH&k%sV!3xiEQrar4#!w7d7^ zr&Lbc99xZ)l9+0@fWU9-y1l0ZG2XvqdAR0*;-T*nx!R4eeW8BxO)R8P3*4{3in-iI zL((r^pJfOD1ooz3;!dTsPruzA)xcTrEx_7>`Y9XlM0r2L?G`}hf)bG`4sk5SX`101 zFfa=3BsGp|)<77waLx|~Gm)a0eg=yc86zdQZ_VvU3=MCUCRIjR2zv@a)Tg?82$3ex zX`vdgp`5NyxENawQd6&?HxVRQB}rIds>S>4=0hRbkmsJ+Y(M2EGDTiNvsgTICI`o` zDr-D~Mvcb^98g5owJY5Nv$9e}=FA{Y7A4rLl{Uikd+sF@o)&ZLPmo8Xs9$jD|S>&I~#Nj$0gGIStCj*LGog)Bc!=&7(insd5X?a;KWXAx?RRT-L$6ed*I($EvOXTHrx(pq*r80+bTDRNx=x#2{_ z(UOBJgKxTiW`DwFHCmN>IxV}aDtX^$_ElN?%EIbu;;GGs;n7ljC{xS$)#nw$8<&AU zYpfu}*Hcnq!2;L{7b7p#j#{|W54v4pG=N93+2*im*@2~#ud&vNEhT@-hm>0pg~mPlE!Le zILE0{=fZ?tXa*r!$ZJN-Y*YA#{T9Vdk znAbnB^$!ltbBU}|jb{ z-7gdv4Nwkkn#Ng63rK*R*!ya?mv&z0oBEfJum3a^!R*)CAZLoJ43dZA_S{fgj1PAU zWS~{+5(RQqzxCcGsZ$d>N~VPU>!I=${2jGUfn^WEp1{S)82H!HwL5Pv zZhPR%oW(5jc_WYYe(v287qAi%7>ZRn(^cy2|DyMbc(TFHDg;zkKlakKyfeCTr%lQ4 z&bMS!X{$O%-Yv(k(lrJ%)Vu|6=F(CqjK_5c`MO;71f0PgM5tiBokcI#Pr(_RCC8N` zUu3O&U&FB*mEe5Zb3G;3d4H72CDGa_r>=f~@Ox{|FkTW%(JiexcVivct7jaMR=y2Bjb~r>~GNHeK{pUIZ60d<^x4OcheXQGQKLK3L zO6ytf9qMu0E&NiuhqD^D=}jAQcas`7Tf#O=3M^v^548_Y^((BBmvx3O6CLIZ;<7Wt zj}!f=>~-3qLoO^l3`2Vh&b(8WZiG&4EdI7h4--F~1;W(chM-nj+_>wB`S2y^jhBpR zOZsFt%2F;}iX#Li_Co939^6mGeV`>FT%W}D62=i{LRGFdgj<#0IamMl7NCk z5Z+gq&0IWwERnwygatd2Ej;Y-HX)P&jAKjo=sEy6NA(wQE{scW`nAO8Kft+G7d37( zDZX7@jW#~*i}_vPyemcfdPXHVFW9kYU9~TvTn*2&R!bA1!ArkENJkbd zUTcp*2`&TgSzIQT^$N!h@xOv?xTZqww;AjjZ&FNq!h=a6j~@7nWB~-ef+dQGEE?k` zu2bBM&tM9_YLI*ungiic-Dl4`H`T+|g`Rni%1kY`MZ64k=JVx8q%6CHdYs6QWVyGl zccz};OevTv?HbQVHmhA4U?WYX+;4nSip%l%meTyjOQN5ok@uvJ2UE;wXSZECviMmC ztwX3`*0z=0n^Rfa&?^0HmDAU2bvJ=*fQhYMjhh@)iq^WInDgjPZ-=YRmg?9Fjke3r znAWw7q6z|Q?Y<&SOw7OeTXH7>m~O^3PgzHdexrYEY2U9ESpzTgv#fR>fw5m@Fp_@U z-8H657W?kXf*_gR*a1dw^haj_57*m0uFmit=81!;(s#o4mQ|Gk?y@JZ6v8^-yf(1==wdL2rz@~w3Mh`L4rBLKW6sy=-;wz<)aS%5U5nlBFywK_ z;9vNSU&^cF0-8eZwp;7*deOg+-=Kpw;At-b&WkGf@bd-SW_2uik(bj-XT2=-;jjV{ z*CpJl-nVle1eLIq(hF8oNF{jsTENmH6lo_u$<_z2UR)+^7E4pBSYZm$z`|6S`0(Dc zsL_Ee7Anyl^gM)}H>FN~mkO#bfmJUSw0*PtrtKq{=n`Uka;TEjyLgTMs16PcKUknO z0x0f6O@ZQ0N9VU^Ia9IGgW@6NsBSBU<@Y3X>h+m%x?RF80aRZXgaJ+(Ai5>?_82Du za1ISB96mB8+5=)M2w=ENpz_KxwV$%18(L`hogoRfE5yUI&gg0}A{ggmP`XF>o=Qe7 zzTceFUBYbF@vZP;wP;sQlztzN-#f4-QL_X>?|85&k>z$rRAVZm5zFwP-welST$Y+? z8V$(ks{`x`nIFNI^x@5fY0PugRl45 zFhc6VJ~)d%H&)|ytfw;cyr&jd-O$n5wOQ_YqI|D@wmaZ6;O50)AQ`zqK8Tz(?+X3r&Y52ngBZK+lmIp-3$5{+NE z*yVTxO+w8HxZ)~H(yv%e?O>Lk3;OtUv`-Q3kw9xo$l_ zKQ@sBllG@_Ud`|erf^!vRreQbFn&+g7Aj1h9#8}7vRcxjXm@9{>h|fv(NW$^-T=!( z7AyN%&1ZZE*~p^OakmJVpNK?WAT+R?R>iMR>ufO`T5W)!`o{XoPY>eegZ9`h@+^Nm zINe-e@U)~_kE~3`>YFE;{Q%1V=pD!EU(q{K_CL`(cb&|}$I&aWaXR4MZ?ySH-Z1^yeVr*Y#@{jMUuNa(MRBdzEi0 zzyWgbg!NCTj=UNg${;%%q^-}O&ki4VumwJbu9?*b5pd9X6Bx`^kpXh zwC|rJTNGHok#snF!Tq2sc2xwai_2~)>*B=plLAY4Hb2E%Ftk}{Ygg*aj?|6%XFDdE z*w!)kaviJp4oFr9LCh9W|3MFIA&Dd8Xy1h(D}i$QFz*NS)Dtdz=l$VmQSzZM$I{i? zz*zT!xhWD9&%}C$CGz*HomxhtW^X!KN70!}OS%M24CH^jt^m0PE3kMaR+o>u9dW0s@xTt#Ba(=;nWX=_Via^5W*8|O3UN`h6n=ZCYNiZaDae$9O%$~(Jw6}P&qRtwRFJiqQn zp{N`7()#Tg2r}8lJ-XL-B~MGs#aFVp8XbJNcSHCXbj7l9K3l=3VV8ssnH1f5hh~bR zT_+PN*YbO)j%jdwxLa66i6pxJfbSR{aK!9xLT{THqzQJB8jrb+N3#0x@X@ITtFE`Q zDp6pRgyz{vakD7!3-(3n=iD}xGAg?Wm)x>5p{Dn%7YhwInGd;@u@|n}ep_7PGLQ>) zbLSg`CgWQ7JrxzKe)90xLy1fq_{?I3sNH-MqI{vAYso9;5{D=Ps}C6J`Xv2$u;~nP zR^AE>F+#|OV>JfR1YZ47UJ!7nLrV@ z-N&T|p;AgXey_J(Lg&>I-!dT=v)hjYbHne3ybJDB_!vpGXwS1alcFUX&NI%qN!)U^ zBujkDc!?g^Xtp>lzE$JtpoCRax#X#93k4bStf>7NA|x zfBw=7-=XP>q;K)G?A`pM(G;7}vz)mmqo$pO))^tLRbC;_7o#CpJcNIrscV(l@V{BS z!PWLBjhAN|PboCDxmCZTh#KT78!l=QajO*l_}uWjc;ta{y5C=DJeT*2>OTy(DDLZS z7)o)Gk;Q*XEu?cBtO7c^I$fsM@nQL0Bj?2>e`0u~Zo4Pn+H|rjsAdhm8x#ny?9rPq zTP1$pA1RDyHk8uZba=;O`nA;J5&hbG^^I{YcfnhV{DqF(l^xE{ztuE7QGJPHAQsD~ ztb%sYWdawSLI?G>P@@3$)&CCV)e^`c@T$^($MP7|Xbuyv`v+D)FA;Mmggt#$2r5w7R2vk`zYwDiw^}H^Zua6ybz7WtH#p;O|5~Z8+X#GVKvoB zuCJ567c1+;8P;})a(;%zqV<4%$@_!`AH1P;V^spK?9{5CZ_y3 z%$6B@LC7(_OuBDywxD4d+b5?hv?PZ8nrA!ZzU1@{f@Hb8*cr&4+;uE^O-t48TTf0q zP&9|RYEwV6+-v%7s2yPaSddVvs9Csu^2Lg5K54bOVL-k!-_1^8{6P)Kqd=84$xmkcx#;CH#skW0$0d-LBYurKDD0xnA1cWgUr3 zc?Zd+u=>hQlOD7AXqQP{#9x`tnU_&x^}|Eeb24j}G^N}-fH`9)XW(089*SgR@Y7xM zbLL^TiCplIH17;!`h9;os1%U zL~|Cgi8k0$^Z4fI`&uoPbV@eOaffTNNHhG3dlv7448_ zKr-2bZCTarI{QeZvsPkapG-8H7rtE|`%D0;%AC_h0^81W&%E$vN@R9vLs3`0br#e<*tDcPdBdLUZF|(tMHYxESwasw*M2HRgz$-OuEUn00>BAboob#~UPm5>Tw-i;6HW2}{v=ZJn)?qN)_&ZwqLQQJj> zy_g?P_xb*tX6)VEKd^cCpnby7(9~CKUZqV9u+Z3`{ps3lrl0aU@R6tHGv&~Kz~j6> z$zMqy3%K614@7fBi+R>eB_hHs*}qh^y=H$`sGUz08hx5RNc;;lOo>q#4D0f4+oF0Fz>%}nJrY0h8%J1t}Mb>Ch~Hiso;5C`dDvuxY?SHDh9BAncAUx$3{h$s9$3JLkIlMD(1exgpXtdnRAUiIuqP4D11SW zZ1`21edF;4oAv5l<*feIW?F)vX`|h*l)L?E=WpI1bd&M^iQPoM{tgQW>sFfNACmp5 zA>~X8`~DyD@~@rRi2-;*hk3?a$gg(BSn_oBU+2bu1wX%C0Y@cnN2-PWtD}0So$v5Z z5a;i&zXr#Rq!oj3`uwU1?-q=T{}LJ%mjmiRVp?np`G1B+Y!QVP2s6qnoS3Svy2P}rY-o9zfzuFo4 pw88pSPyX*|^Ut06KRazM7!VfygsP?z2Uoz~y*nC8MG8hf{|BVO97zBG literal 0 HcmV?d00001 diff --git a/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/vertex-notebook.ipynb b/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/vertex-notebook.ipynb index 74d71a46..2ab79158 100644 --- a/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/vertex-notebook.ipynb +++ b/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/vertex-notebook.ipynb @@ -6,7 +6,7 @@ "source": [ "# Evaluate open LLMs with Vertex AI and Gemini\n", "\n", - "The [Gen AI Evaluation Service in Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview) lets us evaluate LLMs or Application using existing or your own evaluation criterias. It supports academic metrics like BLEU, ROUGE, or LLM as a Judge with Pointwise and Pairwise metrics or custom metrics you can define yourself. It is not directly saided but it can be assumed that Gemini is used as default Judge. \n", + "The [Gen AI Evaluation Service in Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview) lets us evaluate LLMs or Application using existing or your own evaluation criterias. It supports academic metrics like BLEU, ROUGE, or LLM as a Judge with Pointwise and Pairwise metrics or custom metrics you can define yourself. As default LLM as a Judge `Gemini 1.5 Pro` is used.\n", "\n", "We can use the Gen AI Evaluation Service to evaluate the performance of open models and finetuned models using Vertex AI Endpoints and compute resources. In this example we will evaluate [meta-llama/Meta-Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) generated summaries from news articles using a Pointwise metric based on [G-Eval](https://arxiv.org/abs/2303.16634) Coherence metric.\n", "\n", @@ -137,6 +137,33 @@ "After we are logged in we can \"upload\" the model i.e. register the model on Vertex AI. If you want to learn more about the arguments you can pass to the `upload` method, check out [Deploy Gemma 7B with TGI on Vertex AI](https://github.com/huggingface/Google-Cloud-Containers/blob/main/examples/vertex-ai/notebooks/deploy-gemma-on-vertex-ai/vertex-notebook.ipynb).\n" ] }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "from google.cloud import aiplatform\n", + "\n", + "aiplatform.init(project=os.getenv(\"PROJECT_ID\"), location=os.getenv(\"LOCATION\"))\n", + "\n", + "endpoint_display_name = \"llama-3-1-8b-instruct-endpoint\" # TODO: change to your endpoint display name\n", + "\n", + "# Iterates over all the Vertex AI Endpoints within the current project and keeps the first match (if any), otherwise set to None\n", + "ENDPOINT_ID = next(\n", + " (endpoint.name for endpoint in aiplatform.Endpoint.list() \n", + " if endpoint.display_name == endpoint_display_name), \n", + " None\n", + ")\n", + "assert ENDPOINT_ID, (\n", + " \"`ENDPOINT_ID` is not set, please make sure that the `endpoint_display_name` is correct at \"\\\n", + " f\"https://console.cloud.google.com/vertex-ai/online-prediction/endpoints?project={os.getenv('PROJECT_ID')}\"\n", + ")\n", + "\n", + "deployed_model = aiplatform.Endpoint(f\"projects/{os.getenv('PROJECT_ID')}/locations/{os.getenv('LOCATION')}/endpoints/{ENDPOINT_ID}\")" + ] + }, { "cell_type": "code", "execution_count": null, @@ -161,26 +188,9 @@ }, { "cell_type": "code", - "execution_count": 67, - "metadata": {}, - "outputs": [ - { - "ename": "ServiceUnavailable", - "evalue": "503 Machine type temporarily unavailable, please deploy with a different machine type or retry. 14: Machine type temporarily unavailable, please deploy with a different machine type or retry.", - "output_type": "error", - "traceback": [ - "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[0;31mServiceUnavailable\u001b[0m Traceback (most recent call last)", - "Cell \u001b[0;32mIn[67], line 23\u001b[0m\n\u001b[1;32m 20\u001b[0m endpoint \u001b[38;5;241m=\u001b[39m aiplatform\u001b[38;5;241m.\u001b[39mEndpoint\u001b[38;5;241m.\u001b[39mcreate(display_name\u001b[38;5;241m=\u001b[39m\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mvertex_model_name\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m-endpoint\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m 22\u001b[0m \u001b[38;5;66;03m# deploy model to 1x NVIDIA L4\u001b[39;00m\n\u001b[0;32m---> 23\u001b[0m deployed_model \u001b[38;5;241m=\u001b[39m \u001b[43mmodel\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdeploy\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 24\u001b[0m \u001b[43m \u001b[49m\u001b[43mendpoint\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mendpoint\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 25\u001b[0m \u001b[43m \u001b[49m\u001b[43mmachine_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mg2-standard-4\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m 26\u001b[0m \u001b[43m \u001b[49m\u001b[43maccelerator_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mNVIDIA_L4\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m 27\u001b[0m \u001b[43m \u001b[49m\u001b[43maccelerator_count\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m1\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m 28\u001b[0m \u001b[43m)\u001b[49m\n", - "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/gcp/lib/python3.11/site-packages/google/cloud/aiplatform/models.py:5253\u001b[0m, in \u001b[0;36mModel.deploy\u001b[0;34m(self, endpoint, deployed_model_display_name, traffic_percentage, traffic_split, machine_type, min_replica_count, max_replica_count, accelerator_type, accelerator_count, tpu_topology, service_account, explanation_metadata, explanation_parameters, metadata, encryption_spec_key_name, network, sync, deploy_request_timeout, autoscaling_target_cpu_utilization, autoscaling_target_accelerator_duty_cycle, enable_access_logging, disable_container_logging, private_service_connect_config, deployment_resource_pool, reservation_affinity_type, reservation_affinity_key, reservation_affinity_values, spot)\u001b[0m\n\u001b[1;32m 5242\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m 5243\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mTraffic splitting is not yet supported for PSA based PrivateEndpoint. \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 5244\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mTry calling deploy() without providing `traffic_split`. \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 5245\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mA maximum of one model can be deployed to each private Endpoint.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 5246\u001b[0m )\n\u001b[1;32m 5248\u001b[0m explanation_spec \u001b[38;5;241m=\u001b[39m _explanation_utils\u001b[38;5;241m.\u001b[39mcreate_and_validate_explanation_spec(\n\u001b[1;32m 5249\u001b[0m explanation_metadata\u001b[38;5;241m=\u001b[39mexplanation_metadata,\n\u001b[1;32m 5250\u001b[0m explanation_parameters\u001b[38;5;241m=\u001b[39mexplanation_parameters,\n\u001b[1;32m 5251\u001b[0m )\n\u001b[0;32m-> 5253\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_deploy\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 5254\u001b[0m \u001b[43m \u001b[49m\u001b[43mendpoint\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mendpoint\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5255\u001b[0m \u001b[43m \u001b[49m\u001b[43mdeployed_model_display_name\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdeployed_model_display_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5256\u001b[0m \u001b[43m \u001b[49m\u001b[43mtraffic_percentage\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtraffic_percentage\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5257\u001b[0m \u001b[43m \u001b[49m\u001b[43mtraffic_split\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtraffic_split\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5258\u001b[0m \u001b[43m \u001b[49m\u001b[43mmachine_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmachine_type\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5259\u001b[0m \u001b[43m \u001b[49m\u001b[43mmin_replica_count\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmin_replica_count\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5260\u001b[0m \u001b[43m \u001b[49m\u001b[43mmax_replica_count\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmax_replica_count\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5261\u001b[0m \u001b[43m \u001b[49m\u001b[43maccelerator_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43maccelerator_type\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5262\u001b[0m \u001b[43m \u001b[49m\u001b[43maccelerator_count\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43maccelerator_count\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5263\u001b[0m \u001b[43m \u001b[49m\u001b[43mtpu_topology\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtpu_topology\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5264\u001b[0m \u001b[43m \u001b[49m\u001b[43mreservation_affinity_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreservation_affinity_type\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5265\u001b[0m \u001b[43m \u001b[49m\u001b[43mreservation_affinity_key\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreservation_affinity_key\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5266\u001b[0m \u001b[43m \u001b[49m\u001b[43mreservation_affinity_values\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreservation_affinity_values\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5267\u001b[0m \u001b[43m \u001b[49m\u001b[43mservice_account\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mservice_account\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5268\u001b[0m \u001b[43m \u001b[49m\u001b[43mexplanation_spec\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mexplanation_spec\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5269\u001b[0m \u001b[43m \u001b[49m\u001b[43mmetadata\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmetadata\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5270\u001b[0m \u001b[43m \u001b[49m\u001b[43mencryption_spec_key_name\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mencryption_spec_key_name\u001b[49m\n\u001b[1;32m 5271\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;129;43;01mor\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43minitializer\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mglobal_config\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mencryption_spec_key_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5272\u001b[0m \u001b[43m \u001b[49m\u001b[43mnetwork\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mnetwork\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5273\u001b[0m \u001b[43m \u001b[49m\u001b[43msync\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msync\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5274\u001b[0m \u001b[43m \u001b[49m\u001b[43mdeploy_request_timeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdeploy_request_timeout\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5275\u001b[0m \u001b[43m \u001b[49m\u001b[43mautoscaling_target_cpu_utilization\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mautoscaling_target_cpu_utilization\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5276\u001b[0m \u001b[43m \u001b[49m\u001b[43mautoscaling_target_accelerator_duty_cycle\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mautoscaling_target_accelerator_duty_cycle\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5277\u001b[0m \u001b[43m \u001b[49m\u001b[43mspot\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mspot\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5278\u001b[0m \u001b[43m \u001b[49m\u001b[43menable_access_logging\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43menable_access_logging\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5279\u001b[0m \u001b[43m \u001b[49m\u001b[43mdisable_container_logging\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdisable_container_logging\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5280\u001b[0m \u001b[43m \u001b[49m\u001b[43mprivate_service_connect_config\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mprivate_service_connect_config\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5281\u001b[0m \u001b[43m \u001b[49m\u001b[43mdeployment_resource_pool\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdeployment_resource_pool\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5282\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n", - "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/gcp/lib/python3.11/site-packages/google/cloud/aiplatform/base.py:863\u001b[0m, in \u001b[0;36moptional_sync..optional_run_in_thread..wrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 861\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m:\n\u001b[1;32m 862\u001b[0m VertexAiResourceNounWithFutureManager\u001b[38;5;241m.\u001b[39mwait(\u001b[38;5;28mself\u001b[39m)\n\u001b[0;32m--> 863\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mmethod\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 865\u001b[0m \u001b[38;5;66;03m# callbacks to call within the Future (in same Thread)\u001b[39;00m\n\u001b[1;32m 866\u001b[0m internal_callbacks \u001b[38;5;241m=\u001b[39m []\n", - "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/gcp/lib/python3.11/site-packages/google/cloud/aiplatform/models.py:5468\u001b[0m, in \u001b[0;36mModel._deploy\u001b[0;34m(self, endpoint, deployed_model_display_name, traffic_percentage, traffic_split, machine_type, min_replica_count, max_replica_count, accelerator_type, accelerator_count, tpu_topology, reservation_affinity_type, reservation_affinity_key, reservation_affinity_values, service_account, explanation_spec, metadata, encryption_spec_key_name, network, sync, deploy_request_timeout, autoscaling_target_cpu_utilization, autoscaling_target_accelerator_duty_cycle, spot, enable_access_logging, disable_container_logging, private_service_connect_config, deployment_resource_pool)\u001b[0m\n\u001b[1;32m 5456\u001b[0m endpoint \u001b[38;5;241m=\u001b[39m PrivateEndpoint\u001b[38;5;241m.\u001b[39mcreate(\n\u001b[1;32m 5457\u001b[0m display_name\u001b[38;5;241m=\u001b[39mdisplay_name,\n\u001b[1;32m 5458\u001b[0m network\u001b[38;5;241m=\u001b[39mnetwork,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 5463\u001b[0m private_service_connect_config\u001b[38;5;241m=\u001b[39mprivate_service_connect_config,\n\u001b[1;32m 5464\u001b[0m )\n\u001b[1;32m 5466\u001b[0m _LOGGER\u001b[38;5;241m.\u001b[39mlog_action_start_against_resource(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mDeploying model to\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m\"\u001b[39m, endpoint)\n\u001b[0;32m-> 5468\u001b[0m \u001b[43mendpoint\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_deploy_call\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 5469\u001b[0m \u001b[43m \u001b[49m\u001b[43mendpoint\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mapi_client\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5470\u001b[0m \u001b[43m \u001b[49m\u001b[43mendpoint\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mresource_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5471\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5472\u001b[0m \u001b[43m \u001b[49m\u001b[43mendpoint\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_gca_resource\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtraffic_split\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5473\u001b[0m \u001b[43m \u001b[49m\u001b[43mnetwork\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mnetwork\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;129;43;01mor\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mendpoint\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mnetwork\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5474\u001b[0m \u001b[43m \u001b[49m\u001b[43mdeployed_model_display_name\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdeployed_model_display_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5475\u001b[0m \u001b[43m \u001b[49m\u001b[43mtraffic_percentage\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtraffic_percentage\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5476\u001b[0m \u001b[43m \u001b[49m\u001b[43mtraffic_split\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtraffic_split\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5477\u001b[0m \u001b[43m \u001b[49m\u001b[43mmachine_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmachine_type\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5478\u001b[0m \u001b[43m \u001b[49m\u001b[43mmin_replica_count\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmin_replica_count\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5479\u001b[0m \u001b[43m \u001b[49m\u001b[43mmax_replica_count\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmax_replica_count\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5480\u001b[0m \u001b[43m \u001b[49m\u001b[43maccelerator_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43maccelerator_type\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5481\u001b[0m \u001b[43m \u001b[49m\u001b[43maccelerator_count\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43maccelerator_count\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5482\u001b[0m \u001b[43m \u001b[49m\u001b[43mtpu_topology\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtpu_topology\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5483\u001b[0m \u001b[43m \u001b[49m\u001b[43mreservation_affinity_type\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreservation_affinity_type\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5484\u001b[0m \u001b[43m \u001b[49m\u001b[43mreservation_affinity_key\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreservation_affinity_key\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5485\u001b[0m \u001b[43m \u001b[49m\u001b[43mreservation_affinity_values\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mreservation_affinity_values\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5486\u001b[0m \u001b[43m \u001b[49m\u001b[43mservice_account\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mservice_account\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5487\u001b[0m \u001b[43m \u001b[49m\u001b[43mexplanation_spec\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mexplanation_spec\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5488\u001b[0m \u001b[43m \u001b[49m\u001b[43mmetadata\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmetadata\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5489\u001b[0m \u001b[43m \u001b[49m\u001b[43mdeploy_request_timeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdeploy_request_timeout\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5490\u001b[0m \u001b[43m \u001b[49m\u001b[43mautoscaling_target_cpu_utilization\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mautoscaling_target_cpu_utilization\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5491\u001b[0m \u001b[43m \u001b[49m\u001b[43mautoscaling_target_accelerator_duty_cycle\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mautoscaling_target_accelerator_duty_cycle\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5492\u001b[0m \u001b[43m \u001b[49m\u001b[43mspot\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mspot\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5493\u001b[0m \u001b[43m \u001b[49m\u001b[43menable_access_logging\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43menable_access_logging\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5494\u001b[0m \u001b[43m \u001b[49m\u001b[43mdisable_container_logging\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdisable_container_logging\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5495\u001b[0m \u001b[43m \u001b[49m\u001b[43mdeployment_resource_pool\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdeployment_resource_pool\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 5496\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 5498\u001b[0m _LOGGER\u001b[38;5;241m.\u001b[39mlog_action_completed_against_resource(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmodel\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdeployed\u001b[39m\u001b[38;5;124m\"\u001b[39m, endpoint)\n\u001b[1;32m 5500\u001b[0m endpoint\u001b[38;5;241m.\u001b[39m_sync_gca_resource()\n", - "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/gcp/lib/python3.11/site-packages/google/cloud/aiplatform/models.py:1957\u001b[0m, in \u001b[0;36mEndpoint._deploy_call\u001b[0;34m(cls, api_client, endpoint_resource_name, model, endpoint_resource_traffic_split, network, deployed_model_display_name, traffic_percentage, traffic_split, machine_type, min_replica_count, max_replica_count, accelerator_type, accelerator_count, tpu_topology, reservation_affinity_type, reservation_affinity_key, reservation_affinity_values, service_account, explanation_spec, metadata, deploy_request_timeout, autoscaling_target_cpu_utilization, autoscaling_target_accelerator_duty_cycle, spot, enable_access_logging, disable_container_logging, deployment_resource_pool)\u001b[0m\n\u001b[1;32m 1945\u001b[0m operation_future \u001b[38;5;241m=\u001b[39m api_client\u001b[38;5;241m.\u001b[39mdeploy_model(\n\u001b[1;32m 1946\u001b[0m endpoint\u001b[38;5;241m=\u001b[39mendpoint_resource_name,\n\u001b[1;32m 1947\u001b[0m deployed_model\u001b[38;5;241m=\u001b[39mdeployed_model,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 1950\u001b[0m timeout\u001b[38;5;241m=\u001b[39mdeploy_request_timeout,\n\u001b[1;32m 1951\u001b[0m )\n\u001b[1;32m 1953\u001b[0m _LOGGER\u001b[38;5;241m.\u001b[39mlog_action_started_against_resource_with_lro(\n\u001b[1;32m 1954\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mDeploy\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmodel\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;28mcls\u001b[39m, operation_future\n\u001b[1;32m 1955\u001b[0m )\n\u001b[0;32m-> 1957\u001b[0m \u001b[43moperation_future\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mresult\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtimeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mNone\u001b[39;49;00m\u001b[43m)\u001b[49m\n", - "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/gcp/lib/python3.11/site-packages/google/api_core/future/polling.py:261\u001b[0m, in \u001b[0;36mPollingFuture.result\u001b[0;34m(self, timeout, retry, polling)\u001b[0m\n\u001b[1;32m 256\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_blocking_poll(timeout\u001b[38;5;241m=\u001b[39mtimeout, retry\u001b[38;5;241m=\u001b[39mretry, polling\u001b[38;5;241m=\u001b[39mpolling)\n\u001b[1;32m 258\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_exception \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m 259\u001b[0m \u001b[38;5;66;03m# pylint: disable=raising-bad-type\u001b[39;00m\n\u001b[1;32m 260\u001b[0m \u001b[38;5;66;03m# Pylint doesn't recognize that this is valid in this case.\u001b[39;00m\n\u001b[0;32m--> 261\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_exception\n\u001b[1;32m 263\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_result\n", - "\u001b[0;31mServiceUnavailable\u001b[0m: 503 Machine type temporarily unavailable, please deploy with a different machine type or retry. 14: Machine type temporarily unavailable, please deploy with a different machine type or retry." - ] - } - ], + "execution_count": null, + "metadata": {}, + "outputs": [], "source": [ "from huggingface_hub import get_token\n", "\n", @@ -198,18 +208,18 @@ " },\n", " serving_container_ports=[8080],\n", ")\n", - "model.wait() # wait for the model to be registered\n", + "# model.wait() # wait for the model to be registered\n", "\n", - "# create endpoint\n", - "endpoint = aiplatform.Endpoint.create(display_name=f\"{vertex_model_name}-endpoint\")\n", + "# # create endpoint\n", + "# endpoint = aiplatform.Endpoint.create(display_name=f\"{vertex_model_name}-endpoint\")\n", "\n", - "# deploy model to 1x NVIDIA L4\n", - "deployed_model = model.deploy(\n", - " endpoint=endpoint,\n", - " machine_type=\"g2-standard-4\",\n", - " accelerator_type=\"NVIDIA_L4\",\n", - " accelerator_count=1,\n", - ")" + "# # deploy model to 1x NVIDIA L4\n", + "# deployed_model = model.deploy(\n", + "# endpoint=endpoint,\n", + "# machine_type=\"g2-standard-4\",\n", + "# accelerator_type=\"NVIDIA_L4\",\n", + "# accelerator_count=1,\n", + "# )" ] }, { @@ -231,8 +241,8 @@ "from transformers import AutoTokenizer\n", "\n", "# grep the model id from the container spec environment variables\n", - "model_id = next((re.search(r'value: \"(.+)\"', str(item)).group(1) for item in list(model.container_spec.env) if 'MODEL_ID' in str(item)), None)\n", - "tokenizer = AutoTokenizer.from_pretrained(model_id)\n", + "# model_id = next((re.search(r'value: \"(.+)\"', str(item)).group(1) for item in list(model.container_spec.env) if 'MODEL_ID' in str(item)), None)\n", + "tokenizer = AutoTokenizer.from_pretrained(\"meta-llama/Meta-Llama-3.1-8B-Instruct\")\n", "\n", "generation_config = {\n", " \"max_new_tokens\": 256,\n", @@ -259,7 +269,8 @@ " return generated_text\n", "\n", "\n", - "generate(\"How many people live in Berlin?\", generation_config)" + "generate(\"How many people live in Berlin?\", generation_config)\n", + "# 'The population of Berlin is approximately 6.578 million as of my cut off data. However, considering it provides real-time updates, the current population might be slightly higher'" ] }, { @@ -334,7 +345,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We are going to use [argilla/news-summary](https://huggingface.co/datasets/argilla/news-summary) dataset consisting of news article from Reuters. We are going to use a random subset of 10 articles to keep the evaluation fast. Feel free to change the dataset and the number of articles to evaluate the model with more data and different topics.\n" + "We are going to use [argilla/news-summary](https://huggingface.co/datasets/argilla/news-summary) dataset consisting of news article from Reuters. We are going to use a random subset of 15 articles to keep the evaluation fast. Feel free to change the dataset and the number of articles to evaluate the model with more data and different topics.\n" ] }, { @@ -345,9 +356,10 @@ "source": [ "from datasets import load_dataset\n", "\n", - "subset_size = 3\n", + "subset_size = 15\n", "dataset = load_dataset(\"argilla/news-summary\", split=f\"train\").shuffle(seed=42).select(range(subset_size))\n", "\n", + "# print first 150 characters of the first article\n", "print(dataset[0][\"text\"][:150])\n" ] }, @@ -398,9 +410,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Now we can iterate over our prompts and create different evaluation tasks, use our coherence metric to evaluate the summaries and collect the results.\n", - "\n", - "https://github.com/googleapis/python-aiplatform/blob/main/vertexai/evaluation/_evaluation.py" + "Now we can iterate over our prompts and create different evaluation tasks, use our coherence metric to evaluate the summaries and collect the results." ] }, { @@ -409,31 +419,46 @@ "metadata": {}, "outputs": [], "source": [ - "# for prompt_name, prompt in summarization_prompts.items():\n", - "prompt_name = \"simple\"\n", - "prompt = summarization_prompts[prompt_name]\n", + "import uuid\n", "\n", - "# 1. add new prompt column\n", - "df[\"prompt\"] = df[\"text\"].apply(lambda x: prompt.format(text=x))\n", "\n", - "# 2. create eval task\n", - "eval_task = EvalTask(\n", - " dataset=df,\n", - " metrics=[metric],\n", - " experiment=f\"llama-3-1-8b-instruct-{prompt_name}\",\n", - ")\n", - "# 3. run eval task\n", - "results = eval_task.evaluate(model=generate)\n", - "\n" + "results = {}\n", + "for prompt_name, prompt in summarization_prompts.items():\n", + " prompt = summarization_prompts[prompt_name]\n", + "\n", + " # 1. add new prompt column\n", + " df[\"prompt\"] = df[\"text\"].apply(lambda x: prompt.format(text=x))\n", + "\n", + " # 2. create eval task\n", + " eval_task = EvalTask(\n", + " dataset=df,\n", + " metrics=[metric],\n", + " experiment=\"llama-3-1-8b-instruct\",\n", + " )\n", + " # 3. run eval task\n", + " # Note: If the last iteration takes > 1 minute you might need to retry the evaluation\n", + " exp_results = eval_task.evaluate(model=generate, experiment_run_name=f\"prompt-{prompt_name}-{str(uuid.uuid4())[:8]}\")\n", + " print(f\"{prompt_name}: {exp_results.summary_metrics['g-eval-coherence/mean']}\")\n", + " results[prompt_name] = exp_results.summary_metrics[\"g-eval-coherence/mean\"]\n", + "\n", + "for prompt_name, score in sorted(results.items(), key=lambda x: x[1], reverse=True):\n", + " print(f\"{prompt_name}: {score}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ + "Nice, it looks like on our limited test the \"simple\" prompt yields the best results. We can inspect and compare the results in the GCP Console at [Vertex AI > Model Development > Experiments](https://console.cloud.google.com/vertex-ai/experiments).\n", "\n", + "![experiment-results](./assets/experiment-results.png)\n", "\n", - "You can find more examples on how to use the Gen AI Evaluation Service in the [Vertex AI Generative AI documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview)." + "The overview allows to compare the results across different experiments and to inspect the individual evaluations. Here we can see that the standard deviation of detailed is quite high. This could be because of the low sample size or that we need to improve the prompt further.\n", + "\n", + "You can find more examples on how to use the Gen AI Evaluation Service in the [Vertex AI Generative AI documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview) including how to:\n", + "* [how to customize the LLM as a Judge](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/bring_your_own_autorater_with_custom_metric.ipynb)\n", + "* [how to use Pairwise metrics and compare different LLMs](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/compare_generative_ai_models.ipynb)\n", + "* [how to evaluate different prompts more efficiently](https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/evaluation/prompt_engineering_gen_ai_evaluation_service_sdk.ipynb)\n" ] }, { @@ -466,17 +491,11 @@ ] }, { - "cell_type": "markdown", + "cell_type": "code", + "execution_count": null, "metadata": {}, - "source": [ - "Alternatively, you can also remove those from the Google Cloud Console following the steps:\n", - "\n", - "* Go to Vertex AI in Google Cloud\n", - "* Go to Deploy and use -> Online prediction\n", - "* Click on the endpoint and then on the deployed model/s to \"Undeploy model from endpoint\"\n", - "* Then go back to the endpoint list and remove the endpoint\n", - "* Finally, go to Deploy and use -> Model Registry, and remove the model" - ] + "outputs": [], + "source": [] } ], "metadata": { From f92fb9195d2e8a0c88c4e59fd784b77d80f47357 Mon Sep 17 00:00:00 2001 From: philschmid Date: Sun, 22 Sep 2024 19:57:30 +0200 Subject: [PATCH 3/6] clean --- .../vertex-notebook.ipynb | 61 ++++--------------- 1 file changed, 13 insertions(+), 48 deletions(-) diff --git a/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/vertex-notebook.ipynb b/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/vertex-notebook.ipynb index 2ab79158..92abb4fc 100644 --- a/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/vertex-notebook.ipynb +++ b/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/vertex-notebook.ipynb @@ -55,9 +55,8 @@ "source": [ "For ease of use we define the following environment variables for GCP.\n", "\n", - "_Note: Make sure to adapt the project ID to your GCP project._\n", - "\n", - "__Note: The Gen AI Evaluation Service is not available in all regions. If you want to use it, you need to select a region that supports it. At the moment of writing this example, only `us-central1` is supported._ " + "_Note 1: Make sure to adapt the project ID to your GCP project._ \n", + "_Note 2: The Gen AI Evaluation Service is not available in all regions. If you want to use it, you need to select a region that supports it. `us-central1` is currently supported._" ] }, { @@ -137,33 +136,6 @@ "After we are logged in we can \"upload\" the model i.e. register the model on Vertex AI. If you want to learn more about the arguments you can pass to the `upload` method, check out [Deploy Gemma 7B with TGI on Vertex AI](https://github.com/huggingface/Google-Cloud-Containers/blob/main/examples/vertex-ai/notebooks/deploy-gemma-on-vertex-ai/vertex-notebook.ipynb).\n" ] }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "from google.cloud import aiplatform\n", - "\n", - "aiplatform.init(project=os.getenv(\"PROJECT_ID\"), location=os.getenv(\"LOCATION\"))\n", - "\n", - "endpoint_display_name = \"llama-3-1-8b-instruct-endpoint\" # TODO: change to your endpoint display name\n", - "\n", - "# Iterates over all the Vertex AI Endpoints within the current project and keeps the first match (if any), otherwise set to None\n", - "ENDPOINT_ID = next(\n", - " (endpoint.name for endpoint in aiplatform.Endpoint.list() \n", - " if endpoint.display_name == endpoint_display_name), \n", - " None\n", - ")\n", - "assert ENDPOINT_ID, (\n", - " \"`ENDPOINT_ID` is not set, please make sure that the `endpoint_display_name` is correct at \"\\\n", - " f\"https://console.cloud.google.com/vertex-ai/online-prediction/endpoints?project={os.getenv('PROJECT_ID')}\"\n", - ")\n", - "\n", - "deployed_model = aiplatform.Endpoint(f\"projects/{os.getenv('PROJECT_ID')}/locations/{os.getenv('LOCATION')}/endpoints/{ENDPOINT_ID}\")" - ] - }, { "cell_type": "code", "execution_count": null, @@ -208,18 +180,18 @@ " },\n", " serving_container_ports=[8080],\n", ")\n", - "# model.wait() # wait for the model to be registered\n", + "model.wait() # wait for the model to be registered\n", "\n", - "# # create endpoint\n", - "# endpoint = aiplatform.Endpoint.create(display_name=f\"{vertex_model_name}-endpoint\")\n", + "# create endpoint\n", + "endpoint = aiplatform.Endpoint.create(display_name=f\"{vertex_model_name}-endpoint\")\n", "\n", - "# # deploy model to 1x NVIDIA L4\n", - "# deployed_model = model.deploy(\n", - "# endpoint=endpoint,\n", - "# machine_type=\"g2-standard-4\",\n", - "# accelerator_type=\"NVIDIA_L4\",\n", - "# accelerator_count=1,\n", - "# )" + "# deploy model to 1x NVIDIA L4\n", + "deployed_model = model.deploy(\n", + " endpoint=endpoint,\n", + " machine_type=\"g2-standard-4\",\n", + " accelerator_type=\"NVIDIA_L4\",\n", + " accelerator_count=1,\n", + ")" ] }, { @@ -241,7 +213,7 @@ "from transformers import AutoTokenizer\n", "\n", "# grep the model id from the container spec environment variables\n", - "# model_id = next((re.search(r'value: \"(.+)\"', str(item)).group(1) for item in list(model.container_spec.env) if 'MODEL_ID' in str(item)), None)\n", + "model_id = next((re.search(r'value: \"(.+)\"', str(item)).group(1) for item in list(model.container_spec.env) if 'MODEL_ID' in str(item)), None)\n", "tokenizer = AutoTokenizer.from_pretrained(\"meta-llama/Meta-Llama-3.1-8B-Instruct\")\n", "\n", "generation_config = {\n", @@ -489,13 +461,6 @@ "deployed_model.delete()\n", "model.delete()" ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] } ], "metadata": { From c40364a5479cc46cf46a91d644ef4cac4b01f881 Mon Sep 17 00:00:00 2001 From: philschmid Date: Mon, 23 Sep 2024 09:48:56 +0200 Subject: [PATCH 4/6] added docs --- README.md | 7 +++++++ examples/vertex-ai/README.md | 6 ++++++ .../evaluate-llms-with-vertex-ai/vertex-notebook.ipynb | 5 +++++ 3 files changed, 18 insertions(+) diff --git a/README.md b/README.md index 08079d8d..045dc1be 100644 --- a/README.md +++ b/README.md @@ -64,3 +64,10 @@ The [`examples`](./examples) directory contains examples for using the container | Vertex AI | [deploy-flux-on-vertex-ai](./examples/vertex-ai/notebooks/deploy-flux-on-vertex-ai) | Deploying FLUX with Hugging Face PyTorch DLCs for Inference on Vertex AI. | | Vertex AI | [deploy-llama-3-1-405b-on-vertex-ai](./examples/vertex-ai/notebooks/deploy-llama-405b-on-vertex-ai/vertex-notebook.ipynb) | Deploying Meta Llama 3.1 405B in FP8 with Hugging Face DLC for TGI on Vertex AI. | | Cloud Run | [tgi-deployment](./examples/cloud-run/tgi-deployment/README.md) | Deploying Meta Llama 3.1 8B with Text Generation Inference on Cloud Run. | + + +### Evaluation + +| Service | Example | Description | +| --------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------- | +| Vertex AI | [evaluate-llms-with-vertex-ai](./examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai) | Evaluating open LLMs with Vertex AI and Gemini. | diff --git a/examples/vertex-ai/README.md b/examples/vertex-ai/README.md index 37673bba..fbcce718 100644 --- a/examples/vertex-ai/README.md +++ b/examples/vertex-ai/README.md @@ -24,6 +24,12 @@ For Google Vertex AI, we differentiate between the executable Jupyter Notebook e | [deploy-flux-on-vertex-ai](./notebooks/deploy-flux-on-vertex-ai) | Deploying FLUX with Hugging Face PyTorch DLCs for Inference on Vertex AI. | | [deploy-llama-3-1-405b-on-vertex-ai](./notebooks/deploy-llama-405b-on-vertex-ai/vertex-notebook.ipynb) | Deploying Meta Llama 3.1 405B in FP8 with Hugging Face DLC for TGI on Vertex AI. | +### Evaluation Examples + +| Example | Description | +| ------------------------------------------------------------------------ | ----------------------------------------------- | +| [evaluate-llms-with-vertex-ai](./notebooks/evaluate-llms-with-vertex-ai) | Evaluating open LLMs with Vertex AI and Gemini. | + ## Pipelines More to come soon! diff --git a/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/vertex-notebook.ipynb b/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/vertex-notebook.ipynb index 92abb4fc..ad768f42 100644 --- a/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/vertex-notebook.ipynb +++ b/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai/vertex-notebook.ipynb @@ -4,6 +4,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "\n", + "\n", "# Evaluate open LLMs with Vertex AI and Gemini\n", "\n", "The [Gen AI Evaluation Service in Vertex AI](https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluation-overview) lets us evaluate LLMs or Application using existing or your own evaluation criterias. It supports academic metrics like BLEU, ROUGE, or LLM as a Judge with Pointwise and Pairwise metrics or custom metrics you can define yourself. As default LLM as a Judge `Gemini 1.5 Pro` is used.\n", From ecdadcb2a3a1285e35ec6d13a9130e781f09fed5 Mon Sep 17 00:00:00 2001 From: philschmid Date: Mon, 23 Sep 2024 09:56:26 +0200 Subject: [PATCH 5/6] add mention on main --- docs/source/index.mdx | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/source/index.mdx b/docs/source/index.mdx index 88bc3388..4ed1719c 100644 --- a/docs/source/index.mdx +++ b/docs/source/index.mdx @@ -111,3 +111,7 @@ Learn how to use Hugging Face in Google Cloud by reading our blog posts, documen - [Deploying Gemma 7B Instruct with Text Generation Inference (TGI) on Vertex AI](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/vertex-ai/notebooks/deploy-gemma-on-vertex-ai) - [Deploying Gemma 7B Instruct with Text Generation Inference (TGI) from a GCS Bucket on Vertex AI](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/vertex-ai/notebooks/deploy-gemma-from-gcs-on-vertex-ai) - [Deploying FLUX with Hugging Face PyTorch DLCs for Inference on Vertex AI](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/vertex-ai/notebooks/deploy-flux-on-vertex-ai) + +- Evaluation + + - [Evaluating open LLMs with Vertex AI and Gemini](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai) From 831544ee7d13d73c48e7e4a67283b448d94dac7f Mon Sep 17 00:00:00 2001 From: philschmid Date: Mon, 23 Sep 2024 10:29:59 +0200 Subject: [PATCH 6/6] added smaples correctly --- docs/source/index.mdx | 93 +-------------------------------------- docs/source/resources.mdx | 8 +++- 2 files changed, 8 insertions(+), 93 deletions(-) diff --git a/docs/source/index.mdx b/docs/source/index.mdx index 4ed1719c..6a3c7264 100644 --- a/docs/source/index.mdx +++ b/docs/source/index.mdx @@ -23,95 +23,4 @@ Hugging Face DLCs are open source and licensed under Apache 2.0 within the [Goog You have two options to take advantage of these DLCs as a Google Cloud customer: 1. To [get started](https://huggingface.co/blog/google-cloud-model-garden), you can use our no-code integrations within Vertex AI or GKE. -2. For more advanced scenarios, you can pull the containers from the Google Cloud Artifact Registry directly in your environment. [Here](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples) is a list of notebooks examples. - -## Features & benefits 🔥 - -The Hugging Face DLCs provide ready-to-use, tested environments to train and deploy Hugging Face models. They can be used in combination with Google Cloud offerings including Google Kubernetes Engine (GKE) and Vertex AI. GKE is a fully-managed Kubernetes service in Google Cloud that can be used to deploy and operate containerized applications at scale using Google Cloud's infrastructure. Vertex AI is a Machine Learning (ML) platform that lets you train and deploy ML models and AI applications, and customize Large Language Models (LLMs). - -### One command is all you need - -With the new Hugging Face DLCs, train cutting-edge Transformers-based NLP models in a single line of code. The Hugging Face PyTorch DLCs for training come with all the libraries installed to run a single command e.g. via TRL CLI to fine-tune LLMs on any setting, either single-GPU, single-node multi-GPU, and more. - -### Accelerate machine learning from science to production - -In addition to Hugging Face DLCs, we created a first-class Hugging Face library for inference, [`huggingface-inference-toolkit`](https://github.com/huggingface/huggingface-inference-toolkit), that comes with the Hugging Face PyTorch DLCs for inference, with full support on serving any PyTorch model on Google Cloud. - -Deploy your trained models for inference with just one more line of code or select [any of the 170,000+ publicly available models from the model Hub](https://huggingface.co/models?library=pytorch,transformers&sort=trending) and deploy them on either Vertex AI or GKE. - -### High-performance text generation and embedding - -Besides the PyTorch-oriented DLCs, Hugging Face also provides high-performance inference for both text generation and embedding models via the Hugging Face DLCs for both [Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference) and [Text Embeddings Inference (TEI)](https://github.com/huggingface/text-embeddings-inference), respectively. - -The Hugging Face DLC for TGI enables you to deploy [any of the +140,000 text generation inference supported models from the Hugging Face Hub](https://huggingface.co/models?other=text-generation-inference&sort=trending), or any custom model as long as [its architecture is supported within TGI](https://huggingface.co/docs/text-generation-inference/supported_models). - -The Hugging Face DLC for TEI enables you to deploy [any of the +10,000 embedding, re-ranking or sequence classification supported models from the Hugging Face Hub](https://huggingface.co/models?other=text-embeddings-inference&sort=trending), or any custom model as long as [its architecture is supported within TEI](https://huggingface.co/docs/text-embeddings-inference/en/supported_models). - -Additionally, these DLCs come with full support for Google Cloud meaning that deploying models from Google Cloud Storage (GCS) is also straight forward and requires no configuration. - -### Built-in performance - -Hugging Face DLCs feature built-in performance optimizations for PyTorch to train models faster. The DLCs also give you the flexibility to choose a training infrastructure that best aligns with the price/performance ratio for your workload. - -The Hugging Face Training DLCs are fully integrated with Google Cloud, enabling the use of [the latest generation of instances available on Google Cloud Compute Engine](https://cloud.google.com/products/compute?hl=en). - -Hugging Face Inference DLCs provide you with production-ready endpoints that scale quickly with your Google Cloud environment, built-in monitoring, and a ton of enterprise features. - ---- - -Read more about both Vertex AI in [their official documentation](https://cloud.google.com/vertex-ai/docs) and GKE in [their official documentation](https://cloud.google.com/kubernetes-engine/docs). - -## Resources, Documentation & Examples 📄 - -Learn how to use Hugging Face in Google Cloud by reading our blog posts, documentation and examples below. - -### Blog posts - -- [Hugging Face and Google partner for open AI collaboration](https://huggingface.co/blog/gcp-partnership) -- [Google Cloud TPUs made available to Hugging Face users](https://huggingface.co/blog/tpu-inference-endpoints-spaces) -- [Making thousands of open LLMs bloom in the Vertex AI Model Garden](https://huggingface.co/blog/google-cloud-model-garden) -- [Deploy Meta Llama 3.1 405B on Google Cloud Vertex AI](https://huggingface.co/blog/llama31-on-vertex-ai) - -### Documentation - -- [Google Cloud Hugging Face Deep Learning Containers](https://cloud.google.com/deep-learning-containers/docs/choosing-container#hugging-face) -- [Google Cloud public Artifact Registry for DLCs](https://console.cloud.google.com/artifacts/docker/deeplearning-platform-release/us/gcr.io) -- [Serve Gemma open models using GPUs on GKE with Hugging Face TGI](https://cloud.google.com/kubernetes-engine/docs/tutorials/serve-gemma-gpu-tgi) -- [Generative AI on Vertex - Use Hugging Face text generation models](https://cloud.google.com/vertex-ai/generative-ai/docs/open-models/use-hugging-face-models) - -### Examples - -- [All examples](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples) - -#### GKE - -- Training - - - [Full SFT fine-tuning of Gemma 2B in a multi-GPU instance with TRL on GKE](https://github.com/huggingface/Google-Cloud-Containers/blob/main/examples/gke/trl-full-fine-tuning) - - [LoRA SFT fine-tuning of Mistral 7B v0.3 in a single GPU instance with TRL on GKE](https://github.com/huggingface/Google-Cloud-Containers/blob/main/examples/gke/trl-lora-fine-tuning) - -- Inference - - - [Deploying Llama3 8B with Text Generation Inference (TGI) on GKE](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/gke/tgi-deployment) - - [Deploying Qwen2 7B Instruct with Text Generation Inference (TGI) from a GCS Bucket on GKE](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/gke/tgi-from-gcs-deployment) - - [Deploying Snowflake's Arctic Embed (M) with Text Embeddings Inference (TEI) on GKE](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/gke/tei-deployment) - - [Deploying BGE Base v1.5 (English) with Text Embeddings Inference (TEI) from a GCS Bucket on GKE](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/gke/tei-from-gcs-deployment) - -#### Vertex AI - -- Training - - - [Full SFT fine-tuning of Mistral 7B v0.3 in a multi-GPU instance with TRL on Vertex AI](https://github.com/huggingface/Google-Cloud-Containers/blob/main/examples/vertex-ai/notebooks/trl-full-sft-fine-tuning-on-vertex-ai) - - [LoRA SFT fine-tuning of Mistral 7B v0.3 in a single GPU instance with TRL on Vertex AI](https://github.com/huggingface/Google-Cloud-Containers/blob/main/examples/vertex-ai/notebooks/trl-lora-sft-fine-tuning-on-vertex-ai) - -- Inference - - - [Deploying a BERT model for a text classification task using huggingface-inference-toolkit for a Custom Prediction Routine (CPR) on Vertex AI](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/vertex-ai/notebooks/deploy-bert-on-vertex-ai) - - [Deploying an embedding model with Text Embeddings Inference (TEI) on Vertex AI](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/vertex-ai/notebooks/deploy-embedding-on-vertex-ai) - - [Deploying Gemma 7B Instruct with Text Generation Inference (TGI) on Vertex AI](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/vertex-ai/notebooks/deploy-gemma-on-vertex-ai) - - [Deploying Gemma 7B Instruct with Text Generation Inference (TGI) from a GCS Bucket on Vertex AI](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/vertex-ai/notebooks/deploy-gemma-from-gcs-on-vertex-ai) - - [Deploying FLUX with Hugging Face PyTorch DLCs for Inference on Vertex AI](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/vertex-ai/notebooks/deploy-flux-on-vertex-ai) - -- Evaluation - - - [Evaluating open LLMs with Vertex AI and Gemini](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai) +2. For more advanced scenarios, you can pull the containers from the Google Cloud Artifact Registry directly in your environment. [Here](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples) is a list of notebooks examples. \ No newline at end of file diff --git a/docs/source/resources.mdx b/docs/source/resources.mdx index 7ff83001..1801394e 100644 --- a/docs/source/resources.mdx +++ b/docs/source/resources.mdx @@ -50,8 +50,14 @@ Learn how to use Hugging Face in Google Cloud by reading our blog posts, Google - [Deploy FLUX with PyTorch Inference DLC on Vertex AI](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/vertex-ai/notebooks/deploy-flux-on-vertex-ai/vertex-notebook.ipynb) - [Deploy Meta Llama 3.1 405B with TGI DLC on Vertex AI](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/vertex-ai/notebooks/deploy-llama-3-1-405b-on-vertex-ai/vertex-notebook.ipynb) + +- Evaluation + + - [Evaluating open LLMs with Vertex AI and Gemini](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/vertex-ai/notebooks/evaluate-llms-with-vertex-ai) + + ### (Preview) Cloud Run - Inference - - [Deploy Meta Llama 3.1 with TGI DLC on Cloud Run](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/cloud-run/tgi-deployment) + - [Deploy Meta Llama 3.1 with TGI DLC on Cloud Run](https://github.com/huggingface/Google-Cloud-Containers/tree/main/examples/cloud-run/tgi-deployment) \ No newline at end of file