AI and ML
Vertex AI — Model Registry and Endpoints
Vertex AI Model Registry is a centralized repository that manages the lifecycle, versioning, and lineage of machine learning models. Vertex AI Endpoints decouple the serving infrastructure from the model artifacts, allowing for scalable, low-latency deployments across various hardware backends. You utilize these components when you need to standardize production ML pipelines, ensure model reproducibility, and enable rapid A/B testing or canary rollouts in a cloud-native environment.
Understanding the Model Registry
The Model Registry acts as the foundational system of record for your machine learning models within Vertex AI. By centralizing models here, you treat them as first-class, versioned assets rather than orphaned files in object storage. Each time you upload a model version, the registry maintains critical metadata, including training data references, lineage information, and framework-specific artifacts. The registry is not merely a storage bucket; it is a governance tool that allows you to transition models through distinct lifecycle stages, such as 'Experimental', 'Staging', or 'Production'. By explicitly versioning models, you gain the ability to roll back to a known-good state instantly if a production deployment underperforms. This design choice prevents configuration drift and provides auditability, as every deployed version is tied back to its specific training run, source code, and data distribution, which is essential for regulated environments and reproducible research workflows.
from google.cloud import aiplatform
# Registering a model into the Vertex AI Model Registry
model = aiplatform.Model.upload(
display_name="customer-churn-model",
artifact_uri="gs://my-bucket/model-artifacts/",
serving_container_image_uri="us-docker.pkg.dev/vertex-ai/prediction/sklearn-cpu.0-24:latest"
)
# After upload, the model receives a unique resource ID and version ID.Deploying to Vertex AI Endpoints
Vertex AI Endpoints provide a managed, highly available interface for serving predictions from models registered in your registry. When you deploy a model to an endpoint, Vertex AI abstracts away the complexities of managing underlying compute instances, load balancers, and scaling policies. The endpoint acts as an HTTPS ingress, handling incoming requests and routing them to the model container. This abstraction allows you to update your underlying model artifacts without changing the client-facing API endpoint URL. When you create an endpoint, you define the traffic split and the compute resources required to handle your expected request volume. By decoupling the endpoint from the model version, you can perform seamless model hot-swapping. This architecture ensures that infrastructure remains stable even while the machine learning logic iterates frequently. The endpoint environment manages auto-scaling based on CPU or memory usage, ensuring that your inference service remains responsive under varying traffic patterns without manual intervention.
from google.cloud import aiplatform
# Create an endpoint for serving
endpoint = aiplatform.Endpoint.create(display_name="churn-prediction-endpoint")
# Deploy the registered model to the endpoint
model.deploy(endpoint=endpoint, machine_type="n1-standard-4", min_replica_count=1)Managing Traffic and Canary Rollouts
One of the most powerful features of Vertex AI Endpoints is the ability to manage traffic splits between multiple models or different versions of the same model. This capability is foundational for robust production ML strategies such as canary rollouts or shadow deployments. Rather than replacing a live model abruptly, you can assign a small percentage of incoming traffic to a new model version while keeping the majority directed toward the established baseline. This allows you to monitor the performance of the new model against real-world data in a controlled fashion. If the new version exhibits unexpected latency or accuracy degradation, you can shift the traffic weights back to the previous version immediately through an API call. This pattern minimizes risk by allowing for live validation of models before a full promotion. It effectively converts the deployment process into a granular, observable, and reversible operational procedure, preventing significant outages during routine model updates.
from google.cloud import aiplatform
# Traffic splitting: 90% to current model, 10% to new version
endpoint.deploy(
model=new_model_version,
traffic_percentage=10,
traffic_split={model.resource_name: 90, new_model_version.resource_name: 10}
)Handling Prediction Requests and Data Types
When a client sends a prediction request to an endpoint, Vertex AI facilitates the interaction between the request payload and the model's container. The endpoint handles deserialization, passing the data into the specific format expected by your model framework. Understanding the input requirements of your container is critical, as the endpoint enforces consistent schema expectations. You can define custom prediction routines to pre-process incoming raw data before it reaches the model core, or post-process predictions before returning the response to the user. This ensures that the model can be exposed as a standardized API, regardless of whether the underlying model was trained using custom code or a pre-built framework container. By normalizing the communication protocol through the endpoint, you ensure that front-end applications remain agnostic to the internal complexities of the machine learning inference logic, focusing solely on the expected input and output structures.
# Sending an inference request to the deployed endpoint
prediction = endpoint.predict(instances=[[5.1, 3.5, 1.4, 0.2]])
# The endpoint handles the container-specific interface automatically
print(f"Prediction result: {prediction.predictions}")Monitoring and Production Observability
Once your model is live on an endpoint, monitoring becomes the primary vehicle for maintaining model health and accuracy. Vertex AI integrates deeply with Cloud Monitoring to provide visibility into latency, error rates, and resource utilization. Beyond basic infrastructure metrics, you should also implement model monitoring to detect 'training-serving skew' or 'concept drift'. These conditions occur when the data distribution in production deviates significantly from the data used during training, leading to performance degradation. By setting up continuous monitoring on your endpoints, you receive automated alerts when the incoming request data deviates from the baseline distribution. This proactive approach ensures that you are notified of potential model decay before it adversely impacts business outcomes. Integrating these observability tools directly into the deployment process allows you to treat ML service health with the same rigor as traditional backend software, creating a comprehensive feedback loop that informs future retraining efforts.
from google.cloud import aiplatform
# Setting up model monitoring for the endpoint
job = aiplatform.ModelDeploymentMonitoringJob.create(
display_name="churn-monitoring-job",
endpoint=endpoint,
# Define the threshold for skew and drift detection
monitoring_interval=3600 # Every hour
)Key points
- The Model Registry provides a centralized and version-controlled home for all machine learning model artifacts.
- Decoupling model deployment from serving infrastructure allows for independent scaling and lifecycle management.
- Vertex AI Endpoints support traffic splitting to facilitate safe, incremental deployments of new model versions.
- Model versions should always be registered with associated lineage metadata to ensure full auditability and reproducibility.
- Auto-scaling configuration on endpoints ensures that your model serves requests efficiently under varying load conditions.
- Custom prediction routines enable developers to embed data pre-processing directly into the serving container's logic.
- Monitoring production endpoints is essential to detect concept drift and training-serving skew in real-time.
- Automated alerts on monitoring jobs help bridge the gap between initial training and long-term production reliability.
Common mistakes
- Mistake: Deploying models directly to endpoints without versioning. Why it's wrong: This makes rollback impossible and creates downtime during updates. Fix: Always use the Model Registry to manage versions and update the endpoint via traffic splitting.
- Mistake: Assuming Vertex AI Endpoints automatically scale to zero. Why it's wrong: Endpoints incur costs as long as they are deployed, regardless of traffic. Fix: Use auto-scaling configurations and shut down unused endpoints.
- Mistake: Over-provisioning machine types for low-traffic models. Why it's wrong: It leads to significant unnecessary cloud spend. Fix: Use smaller machine types with auto-scaling policies to handle spikes only when needed.
- Mistake: Ignoring traffic splitting during deployment. Why it's wrong: Replacing a model version instantly carries high risk if the new model performs poorly. Fix: Use traffic splitting to shift 5-10% of traffic to the new model first to validate performance.
- Mistake: Failing to define custom container health checks for custom models. Why it's wrong: The endpoint might report a model as 'deployed' even if the serving container is crashed or hanging. Fix: Explicitly define liveness and readiness probes in the container spec.
Interview questions
What is the primary purpose of the Vertex AI Model Registry in Google Cloud?
The Vertex AI Model Registry acts as a central repository for managing the lifecycle of your machine learning models in Google Cloud. It allows you to store, version, and organize your models, making it easier to track which iterations are ready for deployment. By using the registry, you ensure that your production-ready models are properly cataloged, allowing teams to collaborate effectively while maintaining full visibility into model metadata, training lineages, and staging status across multiple environments.
How do you deploy a model to a Vertex AI Endpoint?
Deploying a model involves first importing your model artifact into the Model Registry and then creating an endpoint resource. You associate the model with the endpoint by calling the deployment method, which attaches the model to compute resources. For example, using the Python SDK: 'model.deploy(machine_type='n1-standard-4', min_replica_count=1)'. This process automates the provisioning of virtual machines, sets up the HTTP server to handle prediction requests, and ensures your model is serving traffic reliably.
What are the benefits of using traffic splitting on a Vertex AI Endpoint?
Traffic splitting allows you to route a percentage of incoming requests to different model versions deployed on the same endpoint. This is critical for A/B testing and canary deployments, as it lets you safely evaluate new model versions with live traffic without a full cutover. By configuring the 'traffic_split' parameter, you can gradually shift users from a legacy model to a challenger model, monitoring latency and accuracy metrics to ensure stability before fully retiring the previous version.
Compare Batch Predictions versus Online Predictions in Vertex AI. When should you choose one over the other?
Choose Online Predictions when you require low-latency, real-time inference, such as serving recommendations on a website; these are served via HTTP endpoints that scale based on demand. Choose Batch Predictions when you have large datasets that do not require an immediate response, such as nightly analytical reporting or processing millions of images. Batch jobs are highly cost-effective because they leverage ephemeral resources that spin down automatically after the job completes, whereas Online Endpoints require constant uptime.
How does Vertex AI handle automatic scaling for model endpoints?
Vertex AI uses managed auto-scaling to maintain the performance and availability of your models. By setting the 'min_replica_count' and 'max_replica_count' during deployment, Google Cloud automatically adjusts the number of compute instances in response to incoming request volume. This ensures your service remains responsive during traffic spikes while reducing infrastructure costs during quiet periods. It abstracts away the complexity of cluster management, allowing you to focus on model performance rather than manually resizing your infrastructure underlying the service.
Explain the relationship between Model Registry versioning and deployment aliases in Vertex AI.
Versioning allows you to iterate on models while preserving historical data, while aliases act as dynamic pointers. You can assign an alias like 'default' or 'production' to a specific version number. When you update your application to point to the 'production' alias, you can swap the underlying model version without changing your client-side code. This decoupling is essential for CI/CD pipelines, as it allows your team to promote new versions by simply updating the alias in the registry, ensuring that downstream applications always interact with the intended stable model release.
Check yourself
1. When updating a model on an existing Vertex AI Endpoint, what is the safest approach to ensure zero downtime?
- A.Delete the existing model from the endpoint and deploy the new one immediately.
- B.Create a new endpoint and point DNS to the new service.
- C.Use traffic splitting to route a small percentage of requests to the new model version before shifting all traffic.
- D.Update the endpoint configuration to overwrite the current model file path.
Show answer
C. Use traffic splitting to route a small percentage of requests to the new model version before shifting all traffic.
Traffic splitting allows for canary deployments, which is the standard for zero downtime. Deleting the model first causes downtime, creating a new endpoint is unnecessary overhead, and overwriting files is not how the Registry handles immutable versions.
2. Why is it recommended to register a model in the Vertex AI Model Registry before deploying it to an endpoint?
- A.To ensure the model files are compressed automatically for faster inference.
- B.To manage model lineage, versioning, and simplify the deployment workflow across environments.
- C.To allow Vertex AI to automatically retrain the model when data drifts.
- D.To bypass the need for containerization of the model serving logic.
Show answer
B. To manage model lineage, versioning, and simplify the deployment workflow across environments.
The Registry serves as a catalog that tracks versions and metadata, facilitating reproducible deployments. Compression, retraining, and skipping containerization are not the primary functions or benefits of the Registry.
3. You have a model deployed to an endpoint that is experiencing intermittent latency spikes during high traffic. What is the most cost-effective way to address this using Vertex AI features?
- A.Upgrade the machine type to the most expensive GPU available.
- B.Set the minimum replica count to a very high number to handle peak traffic at all times.
- C.Configure auto-scaling with a specific CPU or request-per-second utilization target.
- D.Disable auto-scaling and rely on manual intervention.
Show answer
C. Configure auto-scaling with a specific CPU or request-per-second utilization target.
Auto-scaling is designed to handle spikes efficiently by adding nodes only when necessary. Higher machine types or high minimum replicas increase base costs, and manual intervention is not scalable.
4. Which of the following scenarios is best handled by creating a new version in the Model Registry rather than just updating the existing model artifact?
- A.Correcting a typo in the model's display name.
- B.Re-deploying the exact same model after a temporary endpoint failure.
- C.Deploying a model trained on a new dataset to compare its performance against the current production model.
- D.Updating the tag for the model description.
Show answer
C. Deploying a model trained on a new dataset to compare its performance against the current production model.
Registry versions are intended for tracking model changes resulting from training experiments. Typos and metadata changes don't require new versions, and re-deploying the same model is unnecessary.
5. What is the primary function of a Vertex AI Endpoint?
- A.To provide a managed service environment for serving predictions from models.
- B.To act as a storage bucket for training datasets.
- C.To automatically generate feature engineering pipelines for raw data.
- D.To execute distributed hyperparameter tuning jobs.
Show answer
A. To provide a managed service environment for serving predictions from models.
Endpoints provide the serving infrastructure (HTTP/REST interface). Storage buckets, feature pipelines, and tuning jobs are separate components of the Vertex AI stack.