Uninstall

cortex cluster down

Delete metadata and log groups

Since you may wish to have access to your data after spinning down your cluster, Cortex's bucket, log groups, and Prometheus volume are not automatically deleted when running cortex cluster down.

To delete them:

# set AWS credentials
export AWS_ACCESS_KEY_ID=***
export AWS_SECRET_ACCESS_KEY=***

# identify the name of your cortex S3 bucket
aws s3 ls

# delete the S3 bucket
aws s3 rb --force s3://<bucket>

# delete the log group (replace <cluster_name> with the name of your cluster, default: cortex)
aws logs describe-log-groups --log-group-name-prefix=<cluster_name> --query logGroups[*].[logGroupName] --output text | xargs -I {} aws logs delete-log-group --log-group-name {}

To delete the Prometheus volume, navigate to the EC2 volumes page in the AWS console (be sure to set the appropriate region), select the volume, and click "Actions" and then "Delete Volume". A Prometheus volume that Cortex created has a name that starts with kubernetes-dynamic-pvc, the kubernetes.io/cluster/<cluster name> tag is set to owned, and the kubernetes.io/created-for/pvc/name tag starts with prometheus-.

If you've configured a custom domain for your APIs, you can remove the SSL Certificate and Hosted Zone for the domain by following these instructions.

Troubleshooting

On rare occasions, cortex cluster down may not be able to spin down your Cortex cluster. When this happens, follow these steps:

  1. If you've manually created any AWS networking resources that are pointed to the cluster or its VPC (e.g. API Gateway VPC links, custom domains, etc), delete them from the AWS console.

  2. Replace "" and "" in the following URL, and open it in your browser: https://console.aws.amazon.com/cloudformation/home?region=<region>#/stacks?filteringText=eksctl-<cluster_name>-

  3. For each CloudFormation stack which contains the word "nodegroup", select the stack and click "Delete".

  4. Select the final stack (the one that ends in "-cluster") and click "Delete".

    If deleting the stack fails, navigate to the EC2 dashboard in the AWS console, delete the load balancers that are associated with the cluster, and try again (you can determine which load balancers are associated with the cluster by setting the correct region in the console and checking the cortex.dev/cluster-name tag on all load balancers). If the problem still persists, delete any other AWS resources that are blocking the stack deletion and try again.

  5. In rare cases, you may need to delete other AWS resources associated with your Cortex cluster. For each the following resources, go to the appropriate AWS Dashboard (in the region that your cluster was in), and confirm that there are no resources left behind by the cluster: CloudWatch Dashboard, SQS Queues, S3 Bucket, and CloudWatch LogGroups (the Cortex bucket and log groups are not deleted by cluster down in order to preserve your data).

Last updated