Fixing AWS Control Tower Baseline resources

Problem

When updating AWS Control Tower accounts after a landing zone upgrade, you may find that some dev has messed up the Control Tower resources or attempted to remove a Stack when they shouldn't. This can lead to the account updates failing for various reasons.

You may see something like...

AWS Control Tower failed to deploy one or more stack set instances: StackSet Id: AWSControlTowerBP-BASELINE-CONFIG:4ef4fefe-1234-4659-938c-fd6da95c0e76, Stack instance Id: arn:aws:cloudformation:us-east-1:406111111874:stack/StackSet-AWSControlTowerBP-BASELINE-CONFIG-c46feddd-1234-459c-aa45-132e23e4d36c/43406ac0-928a-11ec-9017-0eb6c2b8c189, Status: OUTDATED, Status Reason: Stack:arn:aws:cloudformation:us-east-1: 406111111874:stack/StackSet-AWSControlTowerBP-BASELINE-CONFIG-c46feddd-1234-459c-aa45-132e23e4d36c/43406ac0-928a-11ec-9017-0eb6c2b8c189 is in DELETE_FAILED state and can not be updated.

Or maybe you hit something like...

AWS Control Tower cannot create the Config deliverychannel aws-controltower-BaselineConfigDeliveryChannel because the delivery channel already exists. To continue, delete the existing Config deliverychannel and try again. 

Sometimes it is easiest to just remove the Stack resources and redeploy.  That being said, trying to do this with the baseline configurations in Control Tower can be tricky.  Removal of baseline configurations are usually prevented by a SCP Guardrail that needs to be temporarily disabled so you can remove the resources before triggering a redeployments. 

Steps

  • Log into the Master Payer and navigate to the AWS Organizations page. 
  • Select the OU that the account resides under.
  • Select the Policies section.
  • Document the policies that have been applied to this OU. You want to find the aws-guardrails-XXXXX policy.  This is important so you can reattach it later.
  • Select the radio button next to the aws-guardrails-XXXXXX policy and detach it. 
  • Log into the account with the broken resources and nagivate to the CloudFormations page.  Make sure you are in the correct region.
  • Depending on the baseline, you should be able to delete the stack.
  • After deleting the stack, you will want to redeploy it by updating Stack Set instance in the master payer. Using the AWS CLI, after making sure you have set your Profile and Region to match the Master Payer's Control Tower region run something like... 
aws cloudformation update-stack-instances --stack-set-name AWSControlTowerBP-BASELINE-CONFIG --accounts 406111111874 --regions us-east-1

  • After confirming that the resources have been redeployed, you will want to re-enable the guardrail on the OU. Navigate to the OU in the AWS Organizations page of the Master Payer and then make sure to attach the aws-guardrails-XXXXXX policy.
  • You should now be OK to try re-registering the OU in Control Tower to complete the update.
Related AWS Documentation


Popular posts from this blog

Ansible - Task executed multiple times due to a Broken Pipe

Why is this a problem? If you are running a command that is non-idempotent, the command will be started over again which may cause a failure in your playbook execution.  Recently I ran into a situation where Ansible was rerunning on a task on a node even though the task was not configured to be retried.  It turns out that the ssh connection will retry actions auto-magically if the ssh connection fails while waiting for the task to execute. Here's how I was able to reproduce the issue.... Environment Setup I started by setting up a fake environment using containers. I have a scale-ssh  project that can be used to launch a set of containers running ssh that can be used to run Ansible against. $ git clone https://github.com/mwhahaha/scale-ssh $ cd scale-ssh $ ./build-container.sh $ cp ~/.ssh/id_rsa.pub authorized_keys $ ./run-containers.sh 10 After the containers have launched, we now have an Ansible inventory file that can be used to execute Ansible via SSH to our fre...
Read more

tcpdump - Show only http headers

Only the headers tcpdump -A -l -s 0 'tcp port 8088 and (((ip[2:2] - ((ip[0]&0xf) >2)) != 0)' | grep ': ' If you want the request listed as well: tcpdump -A -l -s 0 'tcp port 8088 and (((ip[2:2] - ((ip[0]&0xf) >2)) != 0)' | egrep '(GET|: )'
Read more

Vyatta -- SIP Connection Tracking for VOIP

I've your going to be running VOIP devices behind a Vyatta router, you may need to enable some extra connection tracking options on your firewall to handle the SIP traffic correctly. To enable sip tracking, log in to your router and do the following: vyatta@rtr01:~$ configure vyatta@rtr01# set firewall conntrack-options sip enable-indirect-media vyatta@rtr01# set firewall conntrack-options sip enable-indirect-signalling vyatta@rtr01# commit vyatta@rtr01# save
Read more