Problem

You are trying to use a Python script to retrieve your serving endpoint metrics and save them to a file in the Prometheus format, when you encounter an error.

AttributeError: 'ExportMetricsResponse' object has no attribute 'metrics'
suggests that the ExportMetricsResponse object returned by w.serving_endpoints.export_metrics(name=endpoint_name) does not contain an attribute called metrics.

Cause

You can request metrics from the serving endpoint API in the Prometheus format, but if you are using alternate methods (such as a Python script to retrieve and save the data to a file and then use it in another job) the JSON response is not guaranteed to be in the Prometheus format.

Solution

You can use an example Python script as a base to retrieve the metrics data from a serving endpoint, convert it to Prometheus format, and write it out to a file.

Example code

This python script fetches the data from the serving endpoint metrics API (AWS | Azure | GCP) and writes the metrics from metrics_output.prom out to a file.

The metrics_output.prom file can be read and referred to in any way you require. For example, you can ingest the response into signalfx -> prometheus-exporter.

Before running this example code, you need to replace:

• <workspace-url> - URL of the workspace
• <authentication-token> - Your PAT/OAuth token
• <serving-endpointname> - The serving endpoint you want to collect metrics from

import requests
from databricks.sdk import WorkspaceClient

# Initialize Databricks client
workspace = WorkspaceClient(host="<workspace-url>", token="<authentication-token>")

# Define the endpoint URL
endpoint = "<workspace-url>/api/2.0/serving-endpoints/<serving-endpointname>/metrics"

# Set headers with correct Content-Type
headers = {
"Authorization": f"Bearer {workspace.config.token}",
"Content-Type": "text/plain; version=0.0.4; charset=utf-8"
}

# Make the GET request
response = requests.get(endpoint, headers=headers)

# Check if the request was successful
if response.status_code == 200:
# Save response to a file
with open("metrics_output.prom", "w") as file:
file.write(response.text)

# Print the contents of the file
with open("metrics_output.prom", "r") as file:
print(file.read())
else:
print(f"Error: {response.status_code}, {response.text}")