loading...

AWS-DLM (data Lifecycle management)- Snapshots Not being Created or Policy in "Error" State- Troubleshooting Tips

dineshrathee12 profile image Dinesh Rathee ・4 min read

Issue :
Sometimes You would see that You've created a DLM policy but It is in "Error State" or Not Creating Snapshots.

===== Some useful Tips =====


➜ Explaining the cause why you are not able to see the snapshots being created immediately:

  • It is being said that when you are crating a new policy , It doesn't creates snapshots immediately. Normally it is being take in consideration that "first snapshot begin to be created by a policy within one hour following the specified start time" and for more info you can refer public documentation on this:

[+] Automating the Amazon EBS Snapshot Lifecycle - (Considerations) :https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-lifecycle.html#dlm-considerations

➜ Hence, At this point, I will suggest you to please wait for an hour following the specified start time in policy and then check the snapshots if created in Snapshot page in EC2 Console.


➜ Common Scenario #1- "Duplicate tag key 'Name' specified." - While you have Policy in Error State

Have a look here -
[+] https://stackoverflow.com/questions/52048012/where-to-read-aws-dlm-data-lifecycle-management-error-details

Event Name will look like :


CreateSnapshot:
CreateSnapshot @2018-12-24T20:25:58.000Z UTC
  "errorCode": "Client.InvalidParameterValue",
    "errorMessage": "Duplicate tag key 'Name' specified.", 
    "requestParameters": {
        "volumeId": "vol-xxxxxxxxxxxx",
        "description": "Created for policy: policy-xxxxschedule: First Schedule",


Now, You need to review the resource tagging and update your DLM Policy to update the Duplicate Key entry field under Tag Created snapshots option and try again.

Below are the steps to follow: ( Example)

a). Login to your EC2 console.

b). Go to “Lifecycle Manager” on the left hand side.

c). Go to your DLM policy and “Right click” -- Click “Modify Snapshot LifeclyclePolicy”.

d). Move to the bottom of the policy and you can find “Tag created snapshots” option.

e). Change the Key from tag name for example “Name” to something Like “NameDLMPolicyTest”
[It should be unique than your resource i.e. volume key tags]

f). Click on “Update” and the policy will be updated.


➜ Common Scenario #2 - "errorCode": "Client.InvalidParameterCombination", "errorMessage": "Tag (Name) is already defined in resource id vol-xxxxx" for example

You might see some CloudTrail Event (CreateSnapshots) such as :



CreateSnapshots 
Error :     "errorCode": "Client.InvalidParameterCombination",
"errorMessage": "Tag (Name) is already defined in resource id vol-xxxxxxxxxxxx.",

---------------------------------------------------------------------------------{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "AssumedRole",
   "eventTime": "2020-01-xxxxxxxx",
    "eventSource": "ec2.amazonaws.com",
    "eventName": "CreateSnapshots",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "dlm.amazonaws.com",
    "userAgent": "dlm.amazonaws.com",
    "errorCode": "Client.InvalidParameterCombination",
    "errorMessage": "Tag (Name) is already defined in resource id vol-xxxxxxxx.",
    "requestParameters": {
"requestParameters": {
        "CreateSnapshotsRequest": {
            "Description": "Created for policy: policy-xxxxxxxschedule: Default Schedule",
            "InstanceSpecification": {
                "ExcludeBootVolume": false,
                "InstanceId": "i-xxxxxxx"
            },
---------------------------------------------------


Now, To solve this
The recommendation is to please review your existing policies and tags being used to make sure you're following the requirements / considerations listed here :

[+] DLM - Considerations for Amazon Data Lifecycle Manager : https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-lifecycle.html#dlm-considerations


➜ There are some basic references to check If the policy doesn't trigger any Snapshots after that time as expected :


➜ About the IAm role & Trust Relationships Cause :

Basically, Amazon DLM uses an IAM role to get the permissions that are required to manage snapshots on your behalf. Amazon DLM creates the "AWSDataLifecycleManagerDefaultRole" role the first time that you create a lifecycle policy using the AWS Management Console. For additional information on configuring Data Lifecycle Manager using a Custom IAM Role and about Permissions and trust relationship for Amazon DLM , please refer:

[+] Permissions & Trust relationship for Amazon DLM: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-lifecycle.html#dlm-permissions


➜ Tracking the CloudTrail to see the cause :

Also, You can trace events for DLM Policy State Change — A DLM event emitted when a lifecycle policy enters an "ERROR" state. The event contains a description of what caused the error as well as the state of the policy. To know more, please refer:
[+] DLM - Using CloudWatch Events : https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-lifecycle.html#monitor-cloudwatch-events


➜ Also, basic reference on the DLM Event Details present in CloudTrail which is useful:

Explaining a brief on DLM events in cloudtrail which will be logged using Event Name - "CreateSnapshot".
In general - DLM POLICY related Events

  • CreateLifecyclePolicy - New Policy
  • UpdateLifecyclePolicy - For any update in the policy
  • CreateSnapshot - DLM Creates Snapshots ( Agent Name would be assigned as - "userAgent": "dlm.amazonaws.com") For more Info in this, please refer:

[+] DLM events references : https://docs.aws.amazon.com/cli/latest/reference/dlm/index.html

Other, Some basic info on a fresh setup, You can refer:
[+] Data Lifecycle Manager - Setup New Policy : https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/snapshot-lifecycle.html#dlm-lifecycle-policies
[+] Blog - Data Lifecycle Manager : https://aws.amazon.com/blogs/aws/new-lifecycle-management-for-amazon-ebs-snapshots/

[+] https://stackoverflow.com/questions/59802958/aws-dlm-data-lifecycle-management-snapshots-not-being-created-policy-in-er/59802959#59802959

Posted on by:

dineshrathee12 profile

Dinesh Rathee

@dineshrathee12

Dinesh Rathee , Young IT Professional 💻 Working in Dublin (Ireland) 💚 Enthusiast, Innovative, Loves Travelling 🛣 & Exploring ⛰🚵‍♂ #Fan of @elonmusk 🚀🛰, Proud Son👨‍🎓with Big Dreams.🏁

Discussion

markdown guide