AWS CLI の s3 cp/sync は自動的にチェックサムでデータ整合性チェックしている

AWS CLI の s3 cp/sync は自動的にチェックサムでデータ整合性チェックしている。

Q: Does the AWS CLI validate checksums?

The AWS CLI will perform checksum validation for uploading and downloading files in specific scenarios.

Upload

The AWS CLI will calculate and auto-populate the Content-MD5 header for both standard and multipart uploads. If the checksum that S3 calculates does not match the Content-MD5 provided, S3 will not store the object and instead will return an error message back the AWS CLI. The AWS CLI will retry this error up to 5 times before giving up. On the case that any files fail to transfer successfully to S3, the AWS CLI will exit with a non zero RC. See aws help return-codes for more information.

If the upload request is signed with Signature Version 4, then the AWS CLI uses the x-amz-content-sha256 header as a checksum instead of Content-MD5. The AWS CLI will use Signature Version 4 for S3 in several cases:

• You’re using an AWS region that only supports Signature Version 4. This includes eu-central-1 and ap-northeast-2.
• You explicitly opt in and set signature_version = s3v4 in your ~/.aws/config file.

Note that the AWS CLI will add a Content-MD5 header for both the high level aws s3 commands that perform uploads (aws s3 cp, aws s3 sync) as well as the low level s3api commands including aws s3api put-object and aws s3api upload-part.

AWS CLI S3 FAQ — AWS CLI 2.2.18 Command Reference

aws s3 cp を --debug オプション付きで実行してみると、ちゃんとチェックサムでデータ整合性をチェックしている模様。

$ dd if=/dev/urandom of=1gb.dat bs=1M count=1024
$ aws s3 --debug cp 1gb.dat s3://fujita-san-v-neck-reverse/ 2>&1|grep -i content-md5
 2021-07-13 13:03:45,760 - Thread-3 - botocore.endpoint - DEBUG - Making request for OperationModel(name=UploadPart) with params: {'body': <s3transfer.utils.ReadFileChunk object at 0x7fb15830a650>, 'url': u'https://s3.ap-northeast-1.amazonaws.com/fujita-san-v-neck-reverse/1gb.dat?partNumber=128&uploadId=VLeC0orkEXNBUU_13biYlOHel6ts5PWUN72EI5hBRAPp9V5F6d_jU0wxaa5ZL92dxzrjo3zberY44yEyAYeSr7iFazSRsaSQ_VDUKk3fdN2cgHqFltbj8G1t753dtDP2oI3yyUxMHXjNl4nRgdiL6Y3sNiVpYjrb.JIR_OE6PGE-', 'headers': {'Content-MD5': u'F3byuzaEvR1dKtrMCDkMQg==★', 'Expect': '100-continue', 'User-Agent': 'aws-cli/1.18.147 Python/2.7.18 Linux/4.14.232-177.418.amzn2.x86_64 botocore/1.18.6'}, 'context': {'auth_type': None, 'client_region': 'ap-northeast-1', 'signing': {'bucket': u'fujita-san-v-neck-reverse'}, 'has_streaming_input': True, 'client_config': <botocore.config.Config object at 0x7fb15a5c5110>}, 'query_string': {u'partNumber': 128, u'uploadId': 'VLeC0orkEXNBUU_13biYlOHel6ts5PWUN72EI5hBRAPp9V5F6d_jU0wxaa5ZL92dxzrjo3zberY44yEyAYeSr7iFazSRsaSQ_VDUKk3fdN2cgHqFltbj8G1t753dtDP2oI3yyUxMHXjNl4nRgdiL6Y3sNiVpYjrb.JIR_OE6PGE-'}, 'url_path': u'/fujita-san-v-neck-reverse/1gb.dat', 'method': u'PUT'}
content-md5:F3byuzaEvR1dKtrMCDkMQg==★
content-md5;host;x-amz-content-sha256;x-amz-date;x-amz-security-token
2021-07-13 13:03:45,775 - Thread-3 - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://fujita-san-v-neck-reverse.s3.ap-northeast-1.amazonaws.com/1gb.dat?partNumber=128&uploadId=VLeC0orkEXNBUU_13biYlOHel6ts5PWUN72EI5hBRAPp9V5F6d_jU0wxaa5ZL92dxzrjo3zberY44yEyAYeSr7iFazSRsaSQ_VDUKk3fdN2cgHqFltbj8G1t753dtDP2oI3yyUxMHXjNl4nRgdiL6Y3sNiVpYjrb.JIR_OE6PGE-, headers={'X-Amz-Content-SHA256': 'UNSIGNED-PAYLOAD', 'Content-Length': '8388608', 'X-Amz-Date': '20210713T130345Z', 'User-Agent': 'aws-cli/1.18.147 Python/2.7.18 Linux/4.14.232-177.418.amzn2.x86_64 botocore/1.18.6', 'Expect': '100-continue', 'X-Amz-Security-Token': '...', 'Content-MD5': 'F3byuzaEvR1dKtrMCDkMQg==★', 'Authorization': 'AWS4-HMAC-SHA256 Credential=ASIAX4POU7QUL4BQ4FOY/20210713/ap-northeast-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=...'}>

参考

2020年6月24日以降に作成された新しいバケットは SigV2 署名付きリクエストはサポートされません。ただし、既存のバケットについて引き続き SigV2 がサポートされますが、我々はお客様が古いリクエスト署名方法から移行するよう働きかけます。

Amazon S3 アップデート — SigV2 の廃止時期、延期と変更 | Amazon Web Services ブログ