ablog

不器用で落着きのない技術者のメモ

DynamoDB->DynamoDB Streams->Lambda->Amazon Kinesis Firehose->Aurora に連携する

Amazon DynamoDB からのデータストリームを AWS Lambda と Amazon Kinesis Firehose を活用して Amazon Aurora に格納する | Amazon Web Services ブログ を試してみた。

  • 必要なファイルをダウンロードする。
$ git clone https://github.com/awslabs/dynamoDB-data-replication-to-aurora.git
$ aws s3api create-bucket --bucket az-ddb2aurora --create-bucket-configuration LocationConstraint=ap-northeast-1
$ aws s3api put-object --bucket az-ddb2aurora --key processed/
$ aws s3api put-object --bucket az-ddb2aurora --key tranformation_failed_data_backup/
  • リージョン名、AWSアカウントID、バケット名を書き換える。
$ perl -i.org -pe 's/AWS_REGION/ap-northeast-1/g;s/AWS_ACCOUNT_NUMBER/123456789012/g;s/BUCKET_NAME/az-ddb2aurora/g' *.json

iPhone 7 から iPhone Xs に移行した

iPhone 7 (128GB) から iPhone Xs (64GB) に移行したメモ。auショップiPhone Xs を購入して機種変更手続きを行った後にしたことは以下の通り。

手順

AWS CLI で Redshift クラスターを作成する

AWS CLI で Redshift クラスターを作成したメモ。

手順

  • サブネットグループ作成
$ aws redshift create-cluster-subnet-group --cluster-subnet-group-name sys01-dev-subnet001 --description 'test' --subnet-ids subnet-02a4f85a
  • パラメータグループ作成
$ aws redshift create-cluster-parameter-group --parameter-group-name pg-sys01-dev --parameter-group-family redshift-1.0 --description test
  • パラメータ変更
$ vi pg-sys01-dev.json
[
  {
    "ParameterName": "auto_analyze",
    "ParameterValue": "false"
  },
  {
    "ParameterName": "enable_user_activity_logging",
    "ParameterValue": "true"
  }
]

$ aws redshift modify-cluster-parameter-group --parameter-group-name pg-sys01-dev --parameters file://pg-sys01-dev.json
$ aws redshift create-cluster --db-name dev --cluster-identifier sys01-dev --cluster-type multi-node --node-type ds2.8xlarge \
--master-username awsuser --master-user-password ****** --number-of-nodes 2 --port 5439 --cluster-parameter-group-name pg-sys01-dev \
--encrypted --kms-key-id 2ebae6df-****-4831-****-9d24c93cd996 --cluster-subnet-group-name sys01-dev-subnet001 --enhanced-vpc-routing \
--availability-zone ap-northeast-1c --no-publicly-accessible --vpc-security-group-ids sg-086f4*** --maintenance-track-name current \
--iam-roles arn:aws:iam::123456789012:role/redshift-spectrum-s3-fullaccess
  • WLM設定
$ vi wlm-sys01-dev.json
[
    {
        "ParameterName": "wlm_json_configuration",
        "ParameterValue": "[{\"query_concurrency\":10,\"user_group\":[\"bms_group\",\"mm_group\",\"gate_group\"],\"user_group_wild_card\":0,\"memory_percent_to_user\":60},{\"short_query_queue\":true,\"max_execution_time\":20000}]",
        "ApplyType": "dynamic"
    }
]
$ aws redshift modify-cluster-parameter-group --parameter-group-name pg-sys01-dev --parameters file://wlm-sys01-dev.json
$ aws redshift reboot-cluster --cluster-identifier sys01-dev

RDSのパラメータグループ作成してパラメータを変更時の CloudTrail イベント

RDS MySQL でパラメータグループ作成時のイベントは CreateDBParameterGroup、パラメータ変更時のイベントは ModifyDBParameterGroup。発生するイベントは Aurora でも同じ。

  • パラメータグループ作成
    • パラメータグループファミリー: mysql5.7
    • グループ名: mysql
    • 説明: mysql
{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "...",
        "arn": "arn:aws:sts::1234567890123:assumed-role/AdminRole/...",
        "accountId": "1234567890123",
        "accessKeyId": "...",
        "sessionContext": {
            "attributes": {
                "mfaAuthenticated": "false",
                "creationDate": "2019-05-13T12:54:13Z"
            },
            "sessionIssuer": {
                "type": "Role",
                "principalId": "...",
                "arn": "arn:aws:iam::1234567890123:role/AdminRole",
                "accountId": "1234567890123",
                "userName": "AdminRole"
            }
        }
    },
    "eventTime": "2019-05-13T13:00:44Z",
    "eventSource": "rds.amazonaws.com",
    "eventName": "CreateDBParameterGroup",
    "awsRegion": "ap-northeast-1",
    "sourceIPAddress": "**.*.*.***",
    "userAgent": "console.amazonaws.com",
    "requestParameters": {
        "dBParameterGroupFamily": "mysql5.7", ★
        "description": "mysql", ★
        "dBParameterGroupName": "mysql" ★
    },
    "responseElements": {
        "dBParameterGroupName": "mysql",
        "dBParameterGroupArn": "arn:aws:rds:ap-northeast-1:1234567890123:pg:mysql",
        "description": "mysql",
        "dBParameterGroupFamily": "mysql5.7"
    },
    "requestID": "42a8934a-f16f-4aaf-a2a1-c6dabc13d1a0",
    "eventID": "567c91e1-c511-4d50-8de4-33f333fa4086",
    "eventType": "AwsApiCall",
    "recipientAccountId": "1234567890123"
}
  • パラメータ変更
    • autocommit を 0 に設定
{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "...",
        "arn": "arn:aws:sts::1234567890123:assumed-role/AdminRole/...",
        "accountId": "1234567890123",
        "accessKeyId": "...",
        "sessionContext": {
            "attributes": {
                "mfaAuthenticated": "false",
                "creationDate": "2019-05-13T12:54:13Z"
            },
            "sessionIssuer": {
                "type": "Role",
                "principalId": "...",
                "arn": "arn:aws:iam::1234567890123:role/AdminRole",
                "accountId": "1234567890123",
                "userName": "AdminRole"
            }
        }
    },
    "eventTime": "2019-05-13T13:02:06Z",
    "eventSource": "rds.amazonaws.com",
    "eventName": "ModifyDBParameterGroup",
    "awsRegion": "ap-northeast-1",
    "sourceIPAddress": "**.*.*.***",
    "userAgent": "console.amazonaws.com",
    "requestParameters": {
        "parameters": [
            {
                "isModifiable": false,
                "applyMethod": "immediate",
                "parameterName": "autocommit", ★
                "parameterValue": "0" ★
            }
        ],
        "dBParameterGroupName": "mysql"
    },
    "responseElements": {
        "dBParameterGroupName": "mysql"
    },
    "requestID": "238defbb-2b6c-4956-a6b1-1ccd49d0fd2e",
    "eventID": "cb24eb61-a760-4d9f-9e20-019848c4a168",
    "eventType": "AwsApiCall",
    "recipientAccountId": "1234567890123"
}

AWS CLI で Redshift のスナップショットを共有する

AWS CLI で Redshift のスナップショットを共有してみたメモ。

スナップショットを共有する

共有元AWSアカウント
  • スナップショットを作成する。
$ aws redshift create-cluster-snapshot --snapshot-identifier redshift-cluster-1-snapshot --cluster-identifier redshift-cluster-1
  • 作成したスナップショットを共有する。
$ aws redshift authorize-snapshot-access --snapshot-identifier redshift-cluster-1-snapshot --account-with-restore-access 234567890123
共有先AWSアカウント
  • 共有されたスナップショットを確認する。
$ aws redshift describe-cluster-snapshots --owner-account 123456789012
{
    "Snapshots": [
        {
            "EstimatedSecondsToCompletion": 0,
            "OwnerAccount": "123456789012",
            "CurrentBackupRateInMegaBytesPerSecond": 12.6263,
            "ActualIncrementalBackupSizeInMegaBytes": 15.0,
            "SnapshotRetentionStartTime": "2019-05-07T23:30:43.771Z",
            "NumberOfNodes": 2,
            "Status": "available",
            "VpcId": "vpc-...",
            "ClusterVersion": "1.0",
            "Tags": [],
            "EncryptedWithHSM": false,
            "MasterUsername": "...",
            "ManualSnapshotRetentionPeriod": -1,
            "EnhancedVpcRouting": false,
            "TotalBackupSizeInMegaBytes": 85.0,
            "BackupProgressInMegaBytes": 15.0,
            "MaintenanceTrackName": "current",
            "ClusterCreateTime": "2019-01-22T09:01:46.365Z",
            "RestorableNodeTypes": [
                "dc2.large"
            ],
            "ElapsedTimeInSeconds": 1,
            "ClusterIdentifier": "redshift-cluster-1",
            "SnapshotCreateTime": "2019-05-07T23:30:42.583Z",
            "AvailabilityZone": "ap-northeast-1d",
            "NodeType": "dc2.large",
            "Encrypted": false,
            "SnapshotType": "manual",
            "Port": 5439,
            "SnapshotIdentifier": "redshift-cluster-1-snapshot"
        }
    ]
}

スナップショットの共有をやめる

共有元AWSアカウント
  • スナップショットの共有をやめる。
$ aws redshift revoke-snapshot-access --snapshot-identifier redshift-cluster-1-snapshot --account-with-restore-access 234567890123
共有先AWSアカウント
  • スナップショットが共有されてないことを確認する。
$ aws redshift describe-cluster-snapshots --owner-account 123456789012
{
    "Snapshots": []
}

補足

  • CloudTrail でスナップショット共有時のイベント
{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "...:i-...",
        "arn": "arn:aws:sts::123456789012:assumed-role/EC2AdminRole/i-...",
        "accountId": "123456789012",
        "accessKeyId": "...",
        "sessionContext": {
            "attributes": {
                "mfaAuthenticated": "false",
                "creationDate": "2019-05-07T23:22:30Z"
            },
            "sessionIssuer": {
                "type": "Role",
                "principalId": "...",
                "arn": "arn:aws:iam::123456789012:role/EC2AdminRole",
                "accountId": "123456789012",
                "userName": "EC2AdminRole"
            }
        }
    },
    "eventTime": "2019-05-07T23:39:13Z",
    "eventSource": "redshift.amazonaws.com",
    "eventName": "AuthorizeSnapshotAccess",
    "awsRegion": "ap-northeast-1",
    "sourceIPAddress": "*.***.**.***",
    "userAgent": "aws-cli/1.16.86 Python/2.7.14 Linux/4.14.77-81.59.amzn2.x86_64 botocore/1.12.76",
    "requestParameters": {
        "snapshotIdentifier": "redshift-cluster-1-snapshot",
        "accountWithRestoreAccess": "234567890123"
    },
    "responseElements": {
        "estimatedSecondsToCompletion": 0,
        "ownerAccount": "123456789012",
        "availabilityZone": "ap-northeast-1d",
        "backupProgressInMegaBytes": 15,
        "port": 5439,
        "clusterVersion": "1.0",
        "snapshotRetentionStartTime": "May 7, 2019 11:30:43 PM",
        "numberOfNodes": 2,
        "masterUsername": "...",
        "clusterIdentifier": "redshift-cluster-1",
        "actualIncrementalBackupSizeInMegaBytes": 15,
        "status": "available",
        "clusterCreateTime": "Jan 22, 2019 9:01:46 AM",
        "snapshotType": "manual",
        "encryptedWithHSM": false,
        "manualSnapshotRetentionPeriod": -1,
        "snapshotCreateTime": "May 7, 2019 11:30:42 PM",
        "nodeType": "dc2.large",
        "enhancedVpcRouting": false,
        "vpcId": "vpc-...",
        "totalBackupSizeInMegaBytes": 85,
        "maintenanceTrackName": "current",
        "elapsedTimeInSeconds": 1,
        "tags": [],
        "snapshotIdentifier": "redshift-cluster-1-snapshot",
        "encrypted": false,
        "currentBackupRateInMegaBytesPerSecond": 12.6263
    },
    "requestID": "51c2c8d6-7121-11e9-b0f2-cf53f67bbd19",
    "eventID": "982f38ce-e22c-42b6-b64c-c6fd39fde694",
    "eventType": "AwsApiCall",
    "recipientAccountId": "123456789012"
}
  • スナップショット共有削除時のイベント
{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "...:i-...",
        "arn": "arn:aws:sts::123456789012:assumed-role/EC2AdminRole/i-...",
        "accountId": "123456789012",
        "accessKeyId": "...",
        "sessionContext": {
            "attributes": {
                "mfaAuthenticated": "false",
                "creationDate": "2019-05-07T23:22:30Z"
            },
            "sessionIssuer": {
                "type": "Role",
                "principalId": "...",
                "arn": "arn:aws:iam::123456789012:role/EC2AdminRole",
                "accountId": "123456789012",
                "userName": "EC2AdminRole"
            }
        }
    },
    "eventTime": "2019-05-07T23:39:27Z",
    "eventSource": "redshift.amazonaws.com",
    "eventName": "RevokeSnapshotAccess",
    "awsRegion": "ap-northeast-1",
    "sourceIPAddress": "*.***.**.***",
    "userAgent": "aws-cli/1.16.86 Python/2.7.14 Linux/4.14.77-81.59.amzn2.x86_64 botocore/1.12.76",
    "requestParameters": {
        "snapshotIdentifier": "redshift-cluster-1-snapshot",
        "accountWithRestoreAccess": "234567890123"
    },
    "responseElements": {
        "estimatedSecondsToCompletion": 0,
        "ownerAccount": "123456789012",
        "availabilityZone": "ap-northeast-1d",
        "backupProgressInMegaBytes": 15,
        "port": 5439,
        "clusterVersion": "1.0",
        "snapshotRetentionStartTime": "May 7, 2019 11:30:43 PM",
        "numberOfNodes": 2,
        "masterUsername": "awsuser",
        "clusterIdentifier": "redshift-cluster-1",
        "actualIncrementalBackupSizeInMegaBytes": 15,
        "status": "available",
        "clusterCreateTime": "Jan 22, 2019 9:01:46 AM",
        "snapshotType": "manual",
        "encryptedWithHSM": false,
        "manualSnapshotRetentionPeriod": -1,
        "snapshotCreateTime": "May 7, 2019 11:30:42 PM",
        "nodeType": "dc2.large",
        "enhancedVpcRouting": false,
        "vpcId": "vpc-e971228e",
        "totalBackupSizeInMegaBytes": 85,
        "maintenanceTrackName": "current",
        "elapsedTimeInSeconds": 1,
        "tags": [],
        "snapshotIdentifier": "redshift-cluster-1-snapshot",
        "encrypted": false,
        "currentBackupRateInMegaBytesPerSecond": 12.6263
    },
    "requestID": "5a6ddf09-7121-11e9-83f5-ef6166354d39",
    "eventID": "c4643f12-266c-4c12-9d52-f6d2fb3fe60c",
    "eventType": "AwsApiCall",
    "recipientAccountId": "123456789012"
}

AWS CLI で Aurora のスナップショットを共有する

AWS CLI で Aurora PostgreSQL互換 のスナップショットを共有してみたメモ。

Aurora のスナップショットを共有する

共有元AWSアカウント
  • RDSのスナップショットを作成する
$ aws rds create-db-cluster-snapshot --db-cluster-snapshot-identifier aurora-postgres106-snapshot --db-cluster-identifier aurora-postgres106
  • スナップショットをAWSアカウントID「234567890123」に共有する
aws rds modify-db-cluster-snapshot-attribute --db-cluster-snapshot-identifier aurora-postgres106-snapshot --attribute-name restore --values-to-add 234567890123
共有先AWSアカウント
  • 共有されたスナップショットを確認する。
$ aws rds describe-db-cluster-snapshots --include-shared --snapshot-type shared
{
    "DBClusterSnapshots": [
        {
            "Engine": "aurora-postgresql",
            "SnapshotCreateTime": "2019-05-07T14:57:37.491Z",
            "VpcId": "vpc-...",
            "DBClusterIdentifier": "aurora-postgres106",
            "DBClusterSnapshotArn": "arn:aws:rds:ap-northeast-1:123456789012:cluster-snapshot:aurora-postgres106-snapshot",
            "MasterUsername": "...",
            "LicenseModel": "postgresql-license",
            "Status": "available",
            "PercentProgress": 100,
            "DBClusterSnapshotIdentifier": "arn:aws:rds:ap-northeast-1:123456789012:cluster-snapshot:aurora-postgres106-snapshot",
            "KmsKeyId": "arn:aws:kms:ap-northeast-1:123456789012:key...",
            "ClusterCreateTime": "2019-02-19T01:15:26.881Z",
            "StorageEncrypted": true,
            "AllocatedStorage": 0,
            "EngineVersion": "10.6",
            "SnapshotType": "shared",
            "AvailabilityZones": [
                "ap-northeast-1a",
                "ap-northeast-1c",
                "ap-northeast-1d"
            ],
            "IAMDatabaseAuthenticationEnabled": false,
            "Port": 0
        }
    ]
}

スナップショットの共有をやめる

共有元AWSアカウントID
  • スナップショットの共有をやめる。
$ aws rds modify-db-cluster-snapshot-attribute --db-cluster-snapshot-identifier aurora-postgres106-snapshot --attribute-name restore --values-to-remove 234567890123
共有先AWSアカウントID
  • スナップショットが共有されていないことを確認する。
$ aws rds describe-db-cluster-snapshots --include-shared --snapshot-type shared
{
    "DBClusterSnapshots": []
}

補足

  • スナップショット共有時の CloudTrail のイベント
{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "...:i-...",
        "arn": "arn:aws:sts::123456789012:assumed-role/EC2AdminRole/i-...",
        "accountId": "123456789012",
        "accessKeyId": "...",
        "sessionContext": {
            "attributes": {
                "mfaAuthenticated": "false",
                "creationDate": "2019-05-07T21:22:30Z"
            },
            "sessionIssuer": {
                "type": "Role",
                "principalId": "...",
                "arn": "arn:aws:iam::123456789012:role/EC2AdminRole",
                "accountId": "123456789012",
                "userName": "EC2AdminRole"
            }
        }
    },
    "eventTime": "2019-05-07T22:08:10Z",
    "eventSource": "rds.amazonaws.com",
    "eventName": "ModifyDBClusterSnapshotAttribute",
    "awsRegion": "ap-northeast-1",
    "sourceIPAddress": "*.***.**.***",
    "userAgent": "aws-cli/1.16.86 Python/2.7.14 Linux/4.14.77-81.59.amzn2.x86_64 botocore/1.12.76",
    "requestParameters": {
        "valuesToAdd": [
            "234567890123"
        ],
        "attributeName": "restore",
        "dBClusterSnapshotIdentifier": "aurora-postgres106-snapshot"
    },
    "responseElements": {
        "dBClusterSnapshotAttributes": [
            {
                "attributeName": "restore",
                "attributeValues": [
                    "234567890123"
                ]
            }
        ],
        "dBClusterSnapshotIdentifier": "aurora-postgres106-snapshot"
    },
    "requestID": "fcf6a3e7-8d75-4d46-a078-0989020f0216",
    "eventID": "04bfed7d-24dd-4b48-9e35-2373a262cb73",
    "eventType": "AwsApiCall",
    "recipientAccountId": "123456789012"
}
  • スナップショット共有削除時のイベント
{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "...:i-...",
        "arn": "arn:aws:sts::123456789012:assumed-role/EC2AdminRole/i-...",
        "accountId": "123456789012",
        "accessKeyId": "...",
        "sessionContext": {
            "attributes": {
                "mfaAuthenticated": "false",
                "creationDate": "2019-05-07T21:22:30Z"
            },
            "sessionIssuer": {
                "type": "Role",
                "principalId": "...",
                "arn": "arn:aws:iam::123456789012:role/EC2AdminRole",
                "accountId": "123456789012",
                "userName": "EC2AdminRole"
            }
        }
    },
    "eventTime": "2019-05-07T22:10:05Z",
    "eventSource": "rds.amazonaws.com",
    "eventName": "ModifyDBClusterSnapshotAttribute",
    "awsRegion": "ap-northeast-1",
    "sourceIPAddress": "*.***.**.***",
    "userAgent": "aws-cli/1.16.86 Python/2.7.14 Linux/4.14.77-81.59.amzn2.x86_64 botocore/1.12.76",
    "requestParameters": {
        "valuesToRemove": [
            "234567890123"
        ],
        "attributeName": "restore",
        "dBClusterSnapshotIdentifier": "aurora-postgres106-snapshot"
    },
    "responseElements": {
        "dBClusterSnapshotAttributes": [
            {
                "attributeName": "restore",
                "attributeValues": []
            }
        ],
        "dBClusterSnapshotIdentifier": "aurora-postgres106-snapshot"
    },
    "requestID": "acbd52b9-bf14-42cd-9e82-64c03c079ea8",
    "eventID": "df54195f-c799-40ef-ac7c-32b1aafe2ef7",
    "eventType": "AwsApiCall",
    "recipientAccountId": "123456789012"
}