[Greenplum] 6.12.0 버전에서 PANIC 상황을 발생시키는 방법

Greenplum 6.12.0 버전에서 PANIC 상황을 발생시키는 방법

실제 운영 사이트에서 PANIC을 발생시키는 것은 말도 안되는 상황이지만 개발/테스트 환경에서는 여러가지 이유로 PANIC상황을 발생시키는 이유는 다양하다.

PANIC상황을 발생시키는 이유

• Pglog 에 PANIC 로그가 어떻게 남는지 확인
• 분석을 위한 코어덤프 파일이 생성되는지 테스트
• 장애환경발생시 대응방법 습득

실행환경

• RedHat 7.5
• Greenplum 6.12.0

 

Greenplum 6.14.0 에서 해결된 이슈(Resolved Issues) 30974를 참고하여 PANIC을 발생시켜보자.

URL : https://gpdb.docs.pivotal.io/6-14/relnotes/gpdb-614-release-notes.html

정리하자면 Greenplum MasterNode에 SSH접속 후 유틸리티모드로 gp_param_setting() 함수를 실행하면 PANIC 상황이 발생한다는 것이다.

 

간략히 명령어 정리(SSH로 Greenplum Master Node 접속 후 실행)

su - gpadmin
gpstop -af
gpstart -m
PGOPTIONS='-c gp_session_role=utility' psql -d postgres
SELECT * FROM gp_toolkit.gp_param_setting('');

 

SSH 실행로그

[root@mdw ~]# su - gpadmin
Last login: Tue Jul 13 13:21:37 KST 2021 on pts/4
[gpadmin@mdw ~]$
[gpadmin@mdw ~]$
[gpadmin@mdw ~]$ gpstart -m
20210713:13:42:24:001388 gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args: -m
20210713:13:42:24:001388 gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20210713:13:42:24:001388 gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 6.12.0 build commit:4c176763c7619fb678ce38095e6b3e8fb9548186'
20210713:13:42:24:001388 gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '301908232'
20210713:13:42:24:001388 gpstart:mdw:gpadmin-[WARNING]:-****************************************************************************
20210713:13:42:24:001388 gpstart:mdw:gpadmin-[WARNING]:-Master-only start requested. Disruptive action if standby master configured.
20210713:13:42:24:001388 gpstart:mdw:gpadmin-[WARNING]:-This is advisable only under the direct supervision of Greenplum support.
20210713:13:42:24:001388 gpstart:mdw:gpadmin-[WARNING]:-This mode of operation is not supported in a production environment and
20210713:13:42:24:001388 gpstart:mdw:gpadmin-[WARNING]:-may lead to a split-brain condition and possible unrecoverable data loss.
20210713:13:42:24:001388 gpstart:mdw:gpadmin-[WARNING]:-****************************************************************************

Continue with master-only startup Yy|Nn (default=N):
> y
20210713:13:42:26:001388 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode
20210713:13:42:27:001388 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20210713:13:42:27:001388 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20210713:13:42:28:001388 gpstart:mdw:gpadmin-[INFO]:-Setting new master era
20210713:13:42:28:001388 gpstart:mdw:gpadmin-[INFO]:-Master Started...
[gpadmin@mdw ~]$ PGOPTIONS='-c gp_session_role=utility' psql -d postgres
psql (9.4.24)
Type "help" for help.

postgres=#
postgres=#
postgres=# SELECT * FROM gp_toolkit.gp_param_setting('');
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
!>

Greenplum PANIC 발생 쿼리실행

 

PGLOG PANIC 내용

2021-07-13 13:43:22.918731 KST,"gpadmin","postgres",p1410,th854493312,"[local]",,2021-07-13 13:42:44 KST,0,,cmd1,seg-1,,,,sx1,"LOG","00000","statement: SELECT * FROM gp_toolkit.gp_param_setting('');",,,,,,"SELECT * FROM gp_toolkit.gp_param_setting('');",0,,"postgres.c",1639,
2021-07-13 13:43:23.717428 KST,,,p1410,th0,,,2021-07-13 13:42:44 KST,0,,cmd3,seg-1,,,,,"PANIC","XX000","Unexpected internal error: Segment process received signal SIGSEGV",,,,,,,0,,,,"1    0x7f87304f0680 libpthread.so.0 <symbol not found> + 0x304f0680
2    0x8bebcd postgres ExecInitNode (execProcnode.c:755)
3    0x8d6332 postgres ExecInitAppend (discriminator 3)
4    0x8bd7bc postgres ExecInitNode (execProcnode.c:336)
5    0x8b96f1 postgres standard_ExecutorStart (execMain.c:2098)
6    0x8d4fb6 postgres fmgr_sql (functions.c:979)
7    0x8c839c postgres ExecMakeTableFunctionResult (execQual.c:2339)
8    0x8eb69e postgres <symbol not found> (nodeFunctionscan.c:141)
9    0x8bf4b8 postgres ExecProcNode (execProcnode.c:1043)
10   0x8b67b9 postgres <symbol not found> (tuptable.h:159)
11   0x8b747c postgres standard_ExecutorRun (execMain.c:2962)
12   0xa8b327 postgres <symbol not found> (pquery.c:1152)
13   0xa8d311 postgres PortalRun (pquery.c:999)
14   0xa878f0 postgres <symbol not found> (postgres.c:1827)
15   0xa8a0af postgres PostgresMain (postgres.c:5242)
"
2021-07-13 13:43:23.718281 KST,,,p1398,th854493312,,,,0,,,seg-1,,,,,"LOG","00000","server process (PID 1410) was terminated by signal 11: Segmentation fault","Failed process was running: SELECT * FROM gp_toolkit.gp_param_setting('');",,,,,,0,,"postmaster.c",3970,
2021-07-13 13:43:23.718322 KST,,,p1398,th854493312,,,,0,,,seg-1,,,,,"LOG","00000","terminating any other active server processes",,,,,,,0,,"postmaster.c",3687,
2021-07-13 13:43:23.718359 KST,,,p1406,th854493312,,,,0,,,seg-1,,,,,"WARNING","01000","ic-proxy-server: received signal 3",,,,,,,0,,"ic_proxy_main.c",379,
2021-07-13 13:43:23.718392 KST,"gpadmin","postgres",p1435,th854493312,"[local]",,2021-07-13 13:43:23 KST,0,,,seg-1,,,,,"FATAL","57P03","the database system is in recovery mode","last replayed record at B/C1E1E290",,,,,,0,,"postmaster.c",2553,
2021-07-13 13:43:23.718427 KST,,,p1398,th854493312,,,,0,,,seg-1,,,,,"LOG","00000","worker process: ic proxy process (PID 1406) exited with exit code 1",,,,,,,0,,"postmaster.c",3948,
2021-07-13 13:43:23.718460 KST,,,p1398,th854493312,,,,0,,,seg-1,,,,,"LOG","00000","all server processes terminated; reinitializing",,,,,,,0,,"postmaster.c",4222,
2021-07-13 13:43:23.718492 KST,,,p1436,th854493312,,,,0,,,seg-1,,,,,"LOG","00000","database system was interrupted; last known up at 2021-07-13 13:42:27 KST",,,,,,,0,,"xlog.c",6443,
2021-07-13 13:43:23.995094 KST,,,p1436,th854493312,,,,0,,,seg-1,,,,,"LOG","00000","database system was not properly shut down; automatic recovery in progress",,,,,,,0,,"xlog.c",6849,
2021-07-13 13:43:23.997309 KST,,,p1436,th854493312,,,,0,,,seg-1,,,,,"LOG","00000","record with zero length at B/C1E1E310",,,,,,,0,,"xlog.c",4285,
2021-07-13 13:43:23.997341 KST,,,p1436,th854493312,,,,0,,,seg-1,,,,,"LOG","00000","redo is not required",,,,,,,0,,"xlog.c",7389,
2021-07-13 13:43:24.160157 KST,,,p1436,th854493312,,,,0,,,seg-1,,,,,"LOG","00000","end of transaction log location is B/C1E1E310",,,,,,,0,,"xlog.c",7451,
2021-07-13 13:43:24.161915 KST,,,p1436,th854493312,,,,0,,,seg-1,,,,,"LOG","00000","latest completed transaction id is 2040749 and next transaction id is 2040750",,,,,,,0,,"xlog.c",7739,
2021-07-13 13:43:24.163765 KST,,,p1436,th854493312,,,,0,,,seg-1,,,,,"LOG","00000","MultiXact member wraparound protections are now enabled",,,,,,,0,,"multixact.c",2619,
2021-07-13 13:43:24.163801 KST,,,p1436,th854493312,,,,0,,,seg-1,,,,,"LOG","00000","database system is ready",,,,,,,0,,"xlog.c",7763,
2021-07-13 13:43:24.165213 KST,,,p1398,th854493312,,,,0,,,seg-1,,,,,"LOG","00000","starting background worker process ""ic proxy process""",,,,,,,0,,"postmaster.c",6152,
2021-07-13 13:43:24.165249 KST,,,p1398,th854493312,,,,0,,,seg-1,,,,,"LOG","00000","PostgreSQL 9.4.24 (Greenplum Database 6.12.0 build commit:4c176763c7619fb678ce38095e6b3e8fb9548186) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Oct 28 2020 19:41:17",,,,,,,0,,"postmaster.c",3273,
2021-07-13 13:43:24.165276 KST,,,p1398,th854493312,,,,0,,,seg-1,,,,,"LOG","00000","database system is ready to accept connections","PostgreSQL 9.4.24 (Greenplum Database 6.12.0 build commit:4c176763c7619fb678ce38095e6b3e8fb9548186) on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 6.4.0, 64-bit compiled on Oct 28 2020 19:41:17",,,,,,0,,"postmaster.c",3277,

 

PANIC Pglog - [XX000] Unexpected internal error: Segment process received signal SIGSEGV