Comment on page
Metrics
Cortex includes Prometheus for metrics collection and Grafana for visualization. You can monitor your APIs with the default Grafana dashboards, or create custom metrics and dashboards.
The dashboard URL is displayed once you run a
cortex get <api_name>
command.Alternatively, you can access it on
http://<operator_url>/dashboard
. Run the following command to get the operator URL:cortex env list
If your operator load balancer is configured to be internal, there are a few options for accessing the dashboard:
- 1.Access the dashboard from a machine that has VPC Peering configured to your cluster's VPC, or which is inside of your cluster's VPC.
- 2.Run
kubectl port-forward -n default grafana-0 3000:3000
to forward Grafana's port to your local machine, and access the dashboard on http://localhost:3000 (see instructions for setting upkubectl
here). - 3.
The dashboard is protected with username / password authentication, which by default are:
- Username: admin
- Password: admin
You will be prompted to change the admin user password in the first time you log in.
Grafana allows managing the access of several users and managing teams. For more information on this topic check the grafana documentation.
You can select one or more APIs to visualize in the top left corner of the dashboard.

Grafana allows you to select a time range on which the metrics will be visualized. You can do so in the top right corner of the dashboard.

Note: Cortex only retains a maximum of 2 weeks worth of data at any moment in time
There are more than one dashboard available by default. You can view the available dashboards by accessing the Grafana menu:
Dashboards -> Manage -> Cortex folder
.The dashboards that Cortex ships with are the following:
- RealtimeAPI
- BatchAPI
- Cluster resources
- Node resources
Cortex exposes additional metrics with Prometheus. To view all available metrics, navigate to the
Explore
menu in Grafana and click the Metrics
button.
You can use any of these metrics to set up your own dashboards.
You can scrape metrics from the in-cluster Prometheus server via the
/federate
endpoint and push them to monitoring solutions such as Datadog.The steps for exporting metrics from Prometheus will vary based on your monitoring solution. Here are a few high-level steps to get you started. We will be using Datadog as an example; feel free to reach out to us on Slack if you need help setting up your monitoring tool.
Monitoring solutions provide Kubernetes agents that are capable of scraping Prometheus metrics. Follow the appropriate instructions to install the agent onto your cluster (here are the instructions for Datadog).
Some agents require a Prometheus endpoint to scrape directly. You can provide
http://prometheus.default:9090/federate?match[]={job=~".+"}
as the target url to indicate that all metrics should be scraped.Some agents look for targets to scrape via annotations. You can update Cortex's Prometheus server with the correct annotations. First, Create a
patch.yaml
file and add the relevant annotations for your monitoring solution. Below is an example for Datadog. These annotations instruct the Datadog agent to scrape the Prometheus server at the endpoint /federate?match[]={job=~".+"}
and extract cortex_in_flight_requests
. Note that Datadog specifically requires the query params in the Prometheus url to be encoded.spec:
podMetadata:
annotations:
ad.datadoghq.com/prometheus.check_names: |
["prometheus"]
ad.datadoghq.com/prometheus.init_configs: |
[{}]
ad.datadoghq.com/prometheus.instances: |
[
{
"prometheus_url": "http://%%host%%:%%port%%/federate?match[]=%7Bjob%3D~%22.%2B%22%7D",
"namespace": "cortex",
"metrics": [{"cortex_in_flight_requests":"in_flight_requests"}]
}
]
Then, update Prometheus with your annotations:
kubectl patch --namespace prometheus prometheuses.monitoring.coreos.com prometheus --patch-file patch.yaml --type merge
Prometheus can be configured to write metrics to other monitoring solutions or databases for long term storage. You can attach a remote storage adapter to Prometheus that will receive samples from Prometheus and write to your destination. You can find a list of Prometheus remote storage adapters here. Additional remote storage adapters can be found online if yours isn't on the list.
Once you've found an adapter that works for you, follow the steps below:
Define a
patch.yaml
file with your changes to the Prometheus server:spec:
containers: # container for your adapter
...
remoteWrite:
url: "http://localhost:9201/write" # http endpoint for your adapter
Update Prometheus with your changes:
kubectl patch --namespace prometheus prometheuses.monitoring.coreos.com prometheus --patch-file patch.yaml --type merge
Last modified 1yr ago