Hero Image generated by ChatGPT
This is a personal blog and all content therein is my personal opinion and not that of my employer.
Enjoying the content? If you value the time, effort, and resources invested in creating it, please consider supporting me on Ko-fi.
Secure Google Cloud Authentication in Python: Avoiding CI/CD Pitfalls with Service Accounts
Authenticating Python applications to Google Cloud, especially with service accounts in automated pipelines, can be deceptively complex. This post will guide you through common pitfalls and reveal a robust, secure pattern.
Why This Matters
Service accounts are the workhorse of Google Cloud authentication, especially for automation and CI/CD. In real-world teams and pipelines, service account usage is often riddled with insecure or fragile patterns–many inherited from sample code, legacy systems, or rushed CI/CD configs.
In this post, I’ll walk through the most common anti-patterns I’ve encountered, explain why they’re risky, and share a clean, flexible pattern I now use that plays nicely with Python, secure variables, and CI/CD environments like Azure DevOps.
Common Anti-Patterns
1. Hardcoding Credentials in Source Code
creds = service_account.Credentials.from_service_account_info({
"type": "service_account",
"project_id": "my-project",
...
})
Even if you’re reading from a hardcoded JSON string, you’re just one git push away from leaking credentials. This also breaks security review processes and violates secret scanning policies.
2. Committing Service Account Key Files into Repos or Docker Images
COPY my-creds.json /app/
ENV GOOGLE_APPLICATION_CREDENTIALS=/app/my-creds.json
Secrets shouldn’t be baked into containers or repos–even in .dockerignore or .gitignore. Files tend to linger and may still be present in image layers even after deletion.
3. Passing Individual Environment Variables Manually With Plaintext Secrets
export GCP_CLIENT_EMAIL="..."
export GCP_PRIVATE_KEY="..."
Splitting up the key file into dozens of plaintext variables is error-prone and inconsistent. Developers often omit required fields, misformat values, or struggle with multi-line secrets like private keys (especially when stored in YAML or injected through CI/CD). Not to mention – the secrets are still in plaintext in code.
4. Using Application Default Credentials (ADC) Blindly
google.auth.default()
ADC can behave differently locally vs CI/CD. You might think everything is working… until your pipeline throws DefaultCredentialsError.
ADC’s behavior depends on the environment: it might pick up credentials from a local gcloud config, an attached service account on a VM, or the GOOGLE_APPLICATION_CREDENTIALS environment variable. This variability can lead to unexpected failures in environments like CI/CD where these sources might not be present or configured as expected.
5. Storing JSON Service Account Keys as Plain Variables in Azure DevOps
Azure DevOps allows you to define variables at the pipeline level:
variables:
GCP_KEY_JSON: '{ "type": "service_account", ... }'
But these values are not encrypted at rest unless you mark them as secret variables–and even then, multi-line JSON often breaks due to quoting, encoding, or newline inconsistencies.
Even secret variables in ADO are only base64 encoded in transit to the agent and can often be retrieved in plain text through logs or variable dumps.
6. Using Azure DevOps Secure Files for Service Account Keys
While more secure than plain variables, Secure Files require temporary staging to disk:
- task: DownloadSecureFile@1
name: DownloadKey
inputs:
secureFile: gcp-key.json
This introduces secret sprawl, with key files landing in temporary agents, staging directories, or logs if not carefully handled.
A Better Pattern
Store Each Credential Field as a Separate Secret in Azure Key Vault
- Use an Azure Key Vault–backed variable group.
- Store each field from the service account JSON as a separate secret:
- GCP_TYPE
- GCP_PROJECT_ID
- GCP_PRIVATE_KEY
- GCP_CLIENT_EMAIL
- etc.
- Reference the variable group in your pipeline:
variables:
- group: my-variable-group
- In the pipeline step where you run your Python script, pass those secrets as environment variables:
env:
TYPE: $(GCP_TYPE)
PROJECT_ID: $(GCP_PROJECT_ID)
PRIVATE_KEY: $(GCP_PRIVATE_KEY)
CLIENT_EMAIL: $(GCP_CLIENT_EMAIL)
Reconstruct the JSON In-Memory at Runtime
import os
import json
from google.oauth2 import service_account
def build_credentials_from_env():
GCP_ENV_PREFIX = "GCP_"
env_to_json_map = {
"TYPE": "type",
"PROJECT_ID": "project_id",
"PRIVATE_KEY_ID": "private_key_id",
"PRIVATE_KEY": "private_key",
"CLIENT_EMAIL": "client_email",
"CLIENT_ID": "client_id",
"AUTH_URI": "auth_uri",
"TOKEN_URI": "token_uri",
"AUTH_PROVIDER_X509_CERT_URL": "auth_provider_x509_cert_url",
"CLIENT_X509_CERT_URL": "client_x509_cert_url",
"UNIVERSE_DOMAIN": "universe_domain"
}
creds_dict = {}
missing_vars = []
for env_suffix, json_key in env_to_json_map.items():
env_var_name = f"{GCP_ENV_PREFIX}{env_suffix}"
value = os.environ.get(env_var_name)
if value is None:
missing_vars.append(env_var_name)
else:
creds_dict[json_key] = value.replace("\\n", "\n") if json_key == "private_key" else value
if missing_vars:
raise EnvironmentError(f"Missing required Google Cloud environment variables: {', '.join(missing_vars)}")
return service_account.Credentials.from_service_account_info(creds_dict)
Bonus: Testing Locally Without Polluting Env Vars
You can also pass a single JSON blob as an env var for local dev:
export GCP_KEY_JSON="$(cat gcp-key.json)"
Then use:
json.loads(os.environ["GCP_KEY_JSON"])
This is handy for Docker or pytest setups.
Classes and Decorators for Secure Credential Handling
To make integration dead-simple, I use two class-and-decorator patterns: one for single JSON blobs, and one for split key/value env vars.
Pattern 1: Single JSON string in one environment variable
import os, json
from google.oauth2 import service_account
class GCPJsonEnvCredential:
def __init__(self, env_var="GCP_KEY_JSON"):
self.env_var = env_var
self.original = None
def __enter__(self):
self.original = os.environ.get(self.env_var)
if not self.original:
raise EnvironmentError(f"Missing env var: {self.env_var}")
creds_json = json.loads(self.original.replace("\\n", "\n")) # Fix double-escaped newlines
self.creds = service_account.Credentials.from_service_account_info(creds_json)
return self.creds
def __exit__(self, exc_type, exc_val, exc_tb):
os.environ.pop(self.env_var, None)
def with_gcp_json_env(env_var="GCP_KEY_JSON"):
def decorator(func):
def wrapper(*args, **kwargs):
with GCPJsonEnvCredential(env_var) as creds:
return func(creds, *args, **kwargs)
return wrapper
return decorator
Usage Example:
from google.cloud import bigquery
@with_gcp_json_env()
def run_query(creds):
client = bigquery.Client(credentials=creds, project=creds.project_id)
results = client.query("SELECT name FROM `my_dataset.my_table`").result()
for row in results:
print(row.name)
Pattern 2: Reconstruct from multiple environment variables
import os
import json
from google.oauth2 import service_account
class GCPEnvSplitCredential:
GCP_ENV_PREFIX = "GCP_"
env_to_json_map = {
"TYPE": "type",
"PROJECT_ID": "project_id",
"PRIVATE_KEY_ID": "private_key_id",
"PRIVATE_KEY": "private_key",
"CLIENT_EMAIL": "client_email",
"CLIENT_ID": "client_id",
"AUTH_URI": "auth_uri",
"TOKEN_URI": "token_uri",
"AUTH_PROVIDER_X509_CERT_URL": "auth_provider_x509_cert_url",
"CLIENT_X509_CERT_URL": "client_x509_cert_url",
"UNIVERSE_DOMAIN": "universe_domain"
}
required_env_suffixes = list(env_to_json_map.keys())
def __enter__(self):
self.originals = {}
creds_dict = {}
missing_vars = []
for env_suffix, json_key in self.env_to_json_map.items():
env_var_name = f"{self.GCP_ENV_PREFIX}{env_suffix}"
value = os.environ.get(env_var_name)
self.originals[env_suffix] = value
if value is None:
missing_vars.append(env_var_name)
else:
creds_dict[json_key] = value.replace("\\n", "\n") if json_key == "private_key" else value
if missing_vars:
raise EnvironmentError(f"Missing required Google Cloud environment variables: {', '.join(missing_vars)}")
self.creds = service_account.Credentials.from_service_account_info(creds_dict)
return self.creds
def __exit__(self, exc_type, exc_val, exc_tb):
for k_suffix in self.required_env_suffixes:
os.environ.pop(f"{self.GCP_ENV_PREFIX}{k_suffix}", None)
def with_gcp_split_env():
def decorator(func):
def wrapper(*args, **kwargs):
with GCPEnvSplitCredential() as creds:
return func(creds, *args, **kwargs)
return wrapper
return decorator
Usage Example:
@with_gcp_split_env()
def run_query(creds):
from google.cloud import bigquery
client = bigquery.Client(credentials=creds, project=creds.project_id)
results = client.query("SELECT COUNT(*) FROM `my_dataset.my_table`").result()
for row in results:
print(row[0])
Built-in Features
- No plaintext files on disk: everything is in memory
- Fine-grained secrets: easier to rotate and manage
- CI/CD compatible: works with Azure DevOps, GitHub Actions, etc.
- Python-native: integrates cleanly with google-auth
- Newline handling: The
replace('\\n', '\n')logic ensures the private key is correctly formatted - Secrets purged: Environment variables are removed after use, reducing exposure risk
- Native Credentials object: Works seamlessly with all Google Cloud SDKs
Conclusion
This approach helps avoid dangerous defaults and messy boilerplate.
Whether you’re dealing with Azure DevOps, GitHub Actions, or local dev environments, this pattern is secure, portable, and readable.
Always adhere to the principle of least privilege when assigning IAM roles to your service accounts.
Have you encountered other anti-patterns or found alternative solutions?
Share your thoughts in the comments below!