ablog

不器用で落着きのない技術者のメモ

Aurora PostgreSQL でインスタンスのクラッシュを発生させる

AWS

Aurora PostgreSQL で aurora_inject_crash() を使ってインスタンスをクラッシュさせてみたメモ。

障害挿入クエリ関数 aurora_inject_crash() を使用して、Aurora PostgreSQL インスタンスのクラッシュを強制的に発生させることができます。
この障害挿入クエリでは、フェイルオーバーが発生しません。フェイルオーバーをテストする場合、RDS コンソールで DB クラスターの [フェイルオーバー] インスタンスアクションを選択するか、AWS CLI の failover-db-cluster コマンド、または RDS API の FailoverDBCluster オペレーションを使用します。
障害挿入クエリを使用した Amazon Aurora PostgreSQL のテスト - Amazon Aurora

Aurora PostreSQL に接続する

$ psql "host=cluster-rds001.cluster-********.ap-northeast-1.rds.amazonaws.com user=awsuser dbname=mydb port=5432"
psql (9.2.24, server 12.6)
WARNING: psql version 9.2, server version 12.0.
         Some psql features might not work.
SSL connection (cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256)
Type "help" for help.

SELECT aurora_inject_crash ('node')

クラッシュさせる

mydb=> SELECT aurora_inject_crash ('node');
SSL SYSCALL error: EOF detected
The connection to the server was lost. Attempting reset: Failed.
!> \q

Writer の error/postgres.log

2021-08-30 00:15:22.389 GMT [11470] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:15:22.389 GMT [11470] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:15:22.389 GMT [11470] LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"
2021-08-30 00:15:22 UTC::@:[11470]:WARNING: unrecognized configuration parameter "rds.adaptive_autovacuum"
2021-08-30 00:15:22 UTC::@:[11470]:WARNING: unrecognized configuration parameter "rds.enable_plan_management"
2021-08-30 00:15:22 UTC::@:[11470]:LOG: database system is shut down
Postgres Shared Memory Value: 46636015616 bytes
2021-08-30 00:15:23.899 GMT [11872] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:15:23.899 GMT [11872] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:15:23.899 GMT [11872] LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"
2021-08-30 00:15:23 UTC::@:[11872]:WARNING: unrecognized configuration parameter "rds.adaptive_autovacuum"
2021-08-30 00:15:23 UTC::@:[11872]:WARNING: unrecognized configuration parameter "rds.enable_plan_management"
2021-08-30 00:15:23 UTC::@:[11872]:LOG: starting PostgreSQL 12.6 on aarch64-unknown-linux-gnu, compiled by aarch64-unknown-linux-gnu-gcc (GCC) 7.4.0, 64-bit
2021-08-30 00:15:23 UTC::@:[11872]:LOG: listening on IPv4 address "0.0.0.0", port 5432
2021-08-30 00:15:23 UTC::@:[11872]:LOG: listening on IPv6 address "::", port 5432
2021-08-30 00:15:23 UTC::@:[11872]:LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2021-08-30 00:15:24 UTC::@:[11872]:LOG: redirecting log output to logging collector process
2021-08-30 00:15:24 UTC::@:[11872]:HINT: Future log output will appear in directory "/rdsdbdata/log/error".
2021-08-30 00:15:30 UTC::@:[11906]:LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:15:30 UTC::@:[11906]:LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"

Read Replcia の error/postgres.log

2021-08-30 00:15:28.327 GMT [11065] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:15:28.327 GMT [11065] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:15:28.327 GMT [11065] LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"
2021-08-30 00:15:28 UTC::@:[11065]:WARNING: unrecognized configuration parameter "rds.adaptive_autovacuum"
2021-08-30 00:15:28 UTC::@:[11065]:WARNING: unrecognized configuration parameter "rds.enable_plan_management"
2021-08-30 00:15:28 UTC::@:[11065]:LOG: database system is shut down
Postgres Shared Memory Value: 46636015616 bytes
2021-08-30 00:15:29.348 GMT [11213] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:15:29.348 GMT [11213] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:15:29.348 GMT [11213] LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"
2021-08-30 00:15:29 UTC::@:[11213]:WARNING: unrecognized configuration parameter "rds.adaptive_autovacuum"
2021-08-30 00:15:29 UTC::@:[11213]:WARNING: unrecognized configuration parameter "rds.enable_plan_management"
2021-08-30 00:15:29 UTC::@:[11213]:LOG: starting PostgreSQL 12.6 on aarch64-unknown-linux-gnu, compiled by aarch64-unknown-linux-gnu-gcc (GCC) 7.4.0, 64-bit
2021-08-30 00:15:29 UTC::@:[11213]:LOG: listening on IPv4 address "0.0.0.0", port 5432
2021-08-30 00:15:29 UTC::@:[11213]:LOG: listening on IPv6 address "::", port 5432
2021-08-30 00:15:29 UTC::@:[11213]:LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2021-08-30 00:15:29 UTC::@:[11213]:LOG: redirecting log output to logging collector process
2021-08-30 00:15:29 UTC::@:[11213]:HINT: Future log output will appear in directory "/rdsdbdata/log/error".

SELECT aurora_inject_crash ('instance')

クラッシュさせる

mydb=> SELECT aurora_inject_crash ('instance');
SSL SYSCALL error: EOF detected
The connection to the server was lost. Attempting reset: Failed.
!>

Writer の error/postgres.log

2021-08-30 00:20:10 UTC::@:[11906]:LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:20:10 UTC::@:[11906]:LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"
2021-08-30 00:20:12.258 GMT [26854] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:20:12.258 GMT [26854] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:20:12.258 GMT [26854] LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"
2021-08-30 00:20:12 UTC::@:[26854]:WARNING: unrecognized configuration parameter "rds.adaptive_autovacuum"
2021-08-30 00:20:12 UTC::@:[26854]:WARNING: unrecognized configuration parameter "rds.enable_plan_management"
2021-08-30 00:20:12 UTC::@:[26854]:LOG: database system is shut down
Postgres Shared Memory Value: 46636015616 bytes
2021-08-30 00:20:13.398 GMT [27455] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:20:13.399 GMT [27455] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:20:13.399 GMT [27455] LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"
2021-08-30 00:20:13 UTC::@:[27455]:WARNING: unrecognized configuration parameter "rds.adaptive_autovacuum"
2021-08-30 00:20:13 UTC::@:[27455]:WARNING: unrecognized configuration parameter "rds.enable_plan_management"
2021-08-30 00:20:13 UTC::@:[27455]:LOG: starting PostgreSQL 12.6 on aarch64-unknown-linux-gnu, compiled by aarch64-unknown-linux-gnu-gcc (GCC) 7.4.0, 64-bit
2021-08-30 00:20:13 UTC::@:[27455]:LOG: listening on IPv4 address "0.0.0.0", port 5432
2021-08-30 00:20:13 UTC::@:[27455]:LOG: listening on IPv6 address "::", port 5432
2021-08-30 00:20:13 UTC::@:[27455]:LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2021-08-30 00:20:13 UTC::@:[27455]:LOG: redirecting log output to logging collector process
2021-08-30 00:20:13 UTC::@:[27455]:HINT: Future log output will appear in directory "/rdsdbdata/log/error".
2021-08-30 00:20:30 UTC::@:[27456]:LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:20:30 UTC::@:[27456]:LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"

Read Replcia の error/postgres.log

2021-08-30 00:20:10 UTC::@:[11218]:LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:20:10 UTC::@:[11218]:LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"
2021-08-30 00:20:20.408 GMT [26538] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:20:20.408 GMT [26538] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:20:20.408 GMT [26538] LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"
2021-08-30 00:20:20 UTC::@:[26538]:WARNING: unrecognized configuration parameter "rds.adaptive_autovacuum"
2021-08-30 00:20:20 UTC::@:[26538]:WARNING: unrecognized configuration parameter "rds.enable_plan_management"
2021-08-30 00:20:20 UTC::@:[26538]:LOG: database system is shut down
Postgres Shared Memory Value: 46636015616 bytes
2021-08-30 00:20:21.567 GMT [26654] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:20:21.567 GMT [26654] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:20:21.567 GMT [26654] LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"
2021-08-30 00:20:21 UTC::@:[26654]:WARNING: unrecognized configuration parameter "rds.adaptive_autovacuum"
2021-08-30 00:20:21 UTC::@:[26654]:WARNING: unrecognized configuration parameter "rds.enable_plan_management"
2021-08-30 00:20:21 UTC::@:[26654]:LOG: starting PostgreSQL 12.6 on aarch64-unknown-linux-gnu, compiled by aarch64-unknown-linux-gnu-gcc (GCC) 7.4.0, 64-bit
2021-08-30 00:20:21 UTC::@:[26654]:LOG: listening on IPv4 address "0.0.0.0", port 5432
2021-08-30 00:20:21 UTC::@:[26654]:LOG: listening on IPv6 address "::", port 5432
2021-08-30 00:20:21 UTC::@:[26654]:LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2021-08-30 00:20:21 UTC::@:[26654]:LOG: redirecting log output to logging collector process
2021-08-30 00:20:21 UTC::@:[26654]:HINT: Future log output will appear in directory "/rdsdbdata/log/error".

SELECT aurora_inject_crash ('dispatcher')

クラッシュさせる

mydb=> SELECT aurora_inject_crash ('dispatcher');
              aurora_inject_crash
------------------------------------------------
 fault injection has been executed successfully
(1 row)

Writer の error/postgres.log

2021-08-30 00:23:42.319 GMT [6194] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:23:42.319 GMT [6194] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:23:42.319 GMT [6194] LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"
2021-08-30 00:23:42 UTC::@:[6194]:WARNING: unrecognized configuration parameter "rds.adaptive_autovacuum"
2021-08-30 00:23:42 UTC::@:[6194]:WARNING: unrecognized configuration parameter "rds.enable_plan_management"
2021-08-30 00:23:42 UTC::@:[6194]:LOG: database system is shut down
Postgres Shared Memory Value: 46636015616 bytes
2021-08-30 00:23:43.320 GMT [6985] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:23:43.320 GMT [6985] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:23:43.320 GMT [6985] LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"
2021-08-30 00:23:43 UTC::@:[6985]:WARNING: unrecognized configuration parameter "rds.adaptive_autovacuum"
2021-08-30 00:23:43 UTC::@:[6985]:WARNING: unrecognized configuration parameter "rds.enable_plan_management"
2021-08-30 00:23:43 UTC::@:[6985]:LOG: starting PostgreSQL 12.6 on aarch64-unknown-linux-gnu, compiled by aarch64-unknown-linux-gnu-gcc (GCC) 7.4.0, 64-bit
2021-08-30 00:23:43 UTC::@:[6985]:LOG: listening on IPv4 address "0.0.0.0", port 5432
2021-08-30 00:23:43 UTC::@:[6985]:LOG: listening on IPv6 address "::", port 5432
2021-08-30 00:23:43 UTC::@:[6985]:LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2021-08-30 00:23:43 UTC::@:[6985]:LOG: redirecting log output to logging collector process
2021-08-30 00:23:43 UTC::@:[6985]:HINT: Future log output will appear in directory "/rdsdbdata/log/error".

Read Replcia の error/postgres.log

2021-08-30 00:23:48.508 GMT [5403] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:23:48.508 GMT [5403] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:23:48.508 GMT [5403] LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"
2021-08-30 00:23:48 UTC::@:[5403]:WARNING: unrecognized configuration parameter "rds.adaptive_autovacuum"
2021-08-30 00:23:48 UTC::@:[5403]:WARNING: unrecognized configuration parameter "rds.enable_plan_management"
2021-08-30 00:23:48 UTC::@:[5403]:LOG: database system is shut down
Postgres Shared Memory Value: 46636015616 bytes
2021-08-30 00:23:49.667 GMT [5882] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:23:49.667 GMT [5882] LOG: skipping missing configuration file "/rdsdbdata/config/recovery.conf"
2021-08-30 00:23:49.667 GMT [5882] LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"
2021-08-30 00:23:49 UTC::@:[5882]:WARNING: unrecognized configuration parameter "rds.adaptive_autovacuum"
2021-08-30 00:23:49 UTC::@:[5882]:WARNING: unrecognized configuration parameter "rds.enable_plan_management"
2021-08-30 00:23:49 UTC::@:[5882]:LOG: starting PostgreSQL 12.6 on aarch64-unknown-linux-gnu, compiled by aarch64-unknown-linux-gnu-gcc (GCC) 7.4.0, 64-bit
2021-08-30 00:23:49 UTC::@:[5882]:LOG: listening on IPv4 address "0.0.0.0", port 5432
2021-08-30 00:23:49 UTC::@:[5882]:LOG: listening on IPv6 address "::", port 5432
2021-08-30 00:23:49 UTC::@:[5882]:LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
2021-08-30 00:23:49 UTC::@:[5882]:LOG: redirecting log output to logging collector process
2021-08-30 00:23:49 UTC::@:[5882]:HINT: Future log output will appear in directory "/rdsdbdata/log/error".