Skip to main content
CybersecurityVulnerability Management

Google Cloud Vertex AI Vulnerability Exposes Cross-Tenant RCE Risk

Server room with rows of computer servers and a single, highlighted vulnerable system.

"Google completed the fixes to address this issue in v1.148.0, which was released April 15, 2026."

How a predictable staging bucket name opened a cross‑tenant door

Palo Alto Networks Unit 42 discovered a flaw in the Vertex AI Python SDK (google-cloud-aiplatform) that allowed an attacker in a separate Google Cloud project to hijack a model upload. In SDK versions 1.139.0 and 1.140.0, when a user did not specify a staging_bucket, the SDK constructed a deterministic bucket name from the project ID and region (for example, my-project-vertex-staging-us-central1). The SDK called staging_bucket.exists(), and if the bucket existed it proceeded to upload artifacts without verifying ownership. Because Google Cloud bucket names are globally unique, an attacker who knew a victim’s project ID could preemptively create that bucket in their own project — a technique called bucket squatting — and thereby receive uploads intended for the victim.

Attack mechanics: IAM roles, Cloud Functions and a narrow race window

Unit 42 documented a multi‑phase exploit that depends on three operational conditions: the victim has not yet created the default staging bucket in the target region; the victim omits the staging_bucket parameter in SDK calls such as Model.upload(); and the attacker knows the victim’s project ID. On the attacker side, a cloud project, the squatted bucket, and permissive IAM bindings (for example legacyBucketReader so bucket.exists() returns true, plus objectCreator and objectViewer to permit victim upload and service‑agent reads) are sufficient.

The attacker deploys a Cloud Function triggered by google.storage.object.finalize to act within the tight timing window between the victim’s upload and Google’s internal service agent read. Unit 42’s proof‑of‑concept measured the window at approximately 2.5 seconds: the Cloud Function detected a new object at ~804 ms after upload, replaced the artifact at ~1,433 ms, and the Vertex AI Per‑Product, Per‑Project Service Account (P4SA, service-PROJECT_NUMBER@gcp-sa-aiplatform.iam.gserviceaccount[.]com) read the replaced file by ~2,460 ms. The Cloud Function approach, reacting in roughly 800 ms, is what enabled the attacker to win the race.

Pickle deserialization: turning a model file into remote code execution

Unit 42 named the technique "Pickle in the Middle" because it leverages Python’s pickle/joblib deserialization behavior. Many Python ML models are serialized with joblib or pickle. A crafted joblib/pickle artifact can include a __reduce__ method that executes arbitrary Python code the moment joblib.load() or pickle.load() is called — before any type validation. In the proof‑of‑concept the replaced model carried a joblib payload whose __reduce__ executed immediately when the serving container called joblib.load(). The payload queried the Google Compute Engine metadata server for the serving container’s service account email and OAuth access token, captured container environment variables (project number, endpoint ID, Kubernetes metadata) and exfiltrated them to an attacker‑controlled webhook.

Consequences observed in the proof‑of‑concept

Unit 42’s testing showed that the OAuth token exfiltrated belonged to a service account in Google’s managed tenant project — named custom-online-prediction@<tenant-project>.iam.gserviceaccount[.]com — and had cloud-platform scope. That token permitted actions beyond the single deployment: reading GCS buckets used by other model deployments in the same tenant project (including complete TensorFlow model artifacts), enumerating BigQuery datasets and table names in the victim’s project and reading dataset ACLs, and reading Cloud Logging in the Google‑managed tenant project to reveal internal infrastructure details such as GKE cluster names, active prediction deployments, Google‑internal container image URIs and Kubernetes system identities. Unit 42 emphasized these capabilities as pathways to model theft, data reconnaissance and lateral movement.

Fixes, timeline and recommended defensive steps

Palo Alto Networks reported the vulnerability to Google Cloud through the Vulnerability Reward Program on March 5, 2026. Google assigned top priority on March 9, acknowledged and rated it top severity on March 10, and deployed a first mitigation (v1.144.0) on March 31, 2026. A second, permanent fix — adding a random uuid4 suffix to the default bucket name and adding a bucket ownership verification check in gcs_utils.py — was deployed in v1.148.0 on April 15, 2026. Unit 42 recommends that developers upgrade to v1.148.0 or later to ensure the ownership checks are active, and advises that specifying an explicit staging_bucket when artifact_uri is not a gs:// location improves asset isolation.

What this means for developers, tenant‑project operators, and incident responders

  • Developers and security teams: Update the Vertex AI Python SDK to v1.148.0 or later and, where possible, set an explicit staging_bucket. The vulnerability required no social engineering and no initial access to the victim’s project, so patching is an urgent priority.
  • Tenant‑project operators and cloud operators: Review who can create globally named buckets and examine IAM bindings that permit anonymous or broad read/write to buckets intended for managed service staging. The proof‑of‑concept showed that credentials harvested from a tenant service account can reveal other model artifacts and infrastructure metadata.
  • Incident responders and enterprises: Unit 42 notes that their AI Security Assessment, Frontier AI Defense service and Incident Response team can help identify and mitigate complex AI‑specific risks; contact numbers and regional lines are provided in the Unit 42 report for urgent cases.

Palo Alto Networks and Google worked together to resolve the flaw; Unit 42 also reported that it used a large language model to accelerate discovery and code analysis during research. The episode is a concrete reminder that cloud tooling and ML lifecycle code paths — even a default bucket name and a missing ownership check — can become vectors for cross‑tenant compromise. For any organization running Vertex AI model pipelines, the immediate, documentable step is version upgrade and explicit staging_bucket configuration.

Original report