7 Replies Latest reply: Oct 9, 2012 1:00 AM by 965015 RSS

    LIBTUX_CAT:749: ERROR: Cannot unlock Bulletin Board, not the locker

    965015
      there are two server using tuexedo. the model is MP. one day all tpcall failed.we found lots of "LIBTUX_CAT:1122: ERROR: No space in Bulletin Board","LIBTUX_CAT:582: ERROR: Unable to register, registry table full" mseeage int the ULOG file. before these error appears, there are 5 error message "LIBTUX_CAT:749: ERROR: Cannot unlock Bulletin Board, not the locker". but the message did not appears on the same day.and the number of server process is 5 ,too. thank you!
        • 2. Re: LIBTUX_CAT:749: ERROR: Cannot unlock Bulletin Board, not the locker
          Bobfinan - Oracle-Oracle
          Hi:
          The first two message are usually symptoms of resource management problems. Either internal to Tuxedo (e.g. MAXACCESSERS,MAXSERVERS configuration) or external operating system (e.g. ipc resources).
          The last message is seen rarely. Sometimes when there is a syncronization problem for a CPU in a multicore platform.
          http://docs.oracle.com/cd/E35855_01/tuxedo/docs12c/messages/lbtux/index.html

          You should review the ULOGs to see when problems started to occur. At that point were there corresponding spikes in load or were servers shutdown/rebooted properly (e.g cleanup of ipc resources).
          Make sure that the ubb configuration parameters are properly set for the load that occurred and that Tuxedo is updated to use the latest rolling patch available for that platform.

          Otherwise, open a service request. Provide ulogs(from day of stability to day of distress), configuration files and platform architecture(e.g. type and number of CPUs).

          Regards,
          Bob Finan
          • 3. Re: LIBTUX_CAT:749: ERROR: Cannot unlock Bulletin Board, not the locker
            965015
            UBB file:
            *RESOURCES
            IPCKEY 200703

            DOMAINID mbps #一般生产机为mbps 开发机为mbps2
            MASTER MBPS1,MBPS2
            MAXACCESSERS 1000
            MAXSERVERS 300
            MAXSERVICES 200
            MODEL MP
            OPTIONS LAN
            BBLQUERY 30
            BLOCKTIME 30
            SCANUNIT 10

            LDBAL Y

            *MACHINES
            "jxnxmbps1" LMID=MBPS1
                      APPDIR="/home/switch/bin"
                      TUXCONFIG="/home/switch/etc/tuxconfig"
                      TUXDIR="/home/tuxedo/tuxedo9.1"
                      ULOGPFX="/home/switch/log/TUX/ULOG"
                      TLOGDEVICE = "/home/switch/log/TUX/TLOG"
                      TLOGNAME=TLOG
                      TLOGSIZE=200
                      MAXWSCLIENTS=100

            "jxnxmbps2" LMID=MBPS2
            APPDIR="/home/switch/bin"
            TUXCONFIG="/home/switch/etc/tuxconfig"
            TUXDIR="/home/tuxedo/tuxedo9.1"
            ULOGPFX="/home/switch/log/TUX/ULOG"
            TLOGDEVICE = "/home/switch/log/TUX/TLOG"
            TLOGNAME=TLOG
            TLOGSIZE=200
            MAXWSCLIENTS=10


            *GROUPS
            MBPSGRP1 #MBPS 主服务组
            LMID=MBPS1 GRPNO=2
            TMSNAME="TMS_ORACLE64" TMSCOUNT=20
            # OPENINFO="Oracle_XA64:Oracle_XA+Acc=P/jxnxmbps/jxnxmbps+SesTm=600+MaxCur=5+LogDir=."
            OPENINFO="Oracle_XA64:Oracle_XA+Acc=P/jxnxmbps/jxnxmbps+SesTm=600+MaxCur=5+LogDir=.+SqlNet=ORCL147"

            MBPSGRP2 #MBPS 主服务组
            LMID=MBPS2 GRPNO=8
            TMSNAME="TMS_ORACLE64" TMSCOUNT=20
            # OPENINFO="Oracle_XA64:Oracle_XA+Acc=P/jxnxmbps/jxnxmbps+SesTm=600+MaxCur=5+LogDir=."
            OPENINFO="Oracle_XA64:Oracle_XA+Acc=P/jxnxmbps/jxnxmbps+SesTm=600+MaxCur=5+LogDir=.+SqlNet=ORCL147"
            WSL
            LMID=MBPS1 GRPNO=3

            FRONT
            LMID=MBPS1 GRPNO=4
            FFILE
            LMID=MBPS1 GRPNO=5

            MBPSDMADM
            LMID=MBPS1 GRPNO=20

            GWGRP1
            LMID=MBPS1 GRPNO=21

            GWGRP2
            LMID=MBPS1 GRPNO=22

            *NETWORK
            MBPS1 NADDR="//10.0.135.151:6000"
            NLSADDR="//10.0.135.151:6001"

            MBPS2 NADDR="//10.0.135.152:6000"
            NLSADDR="//10.0.135.152:6001"


            *SERVERS
            DEFAULT:
                      CLOPT="-A -r"

            MRSVR SRVGRP=MBPSGRP1 SRVID=100 MIN=5 MAX=5 RESTART=Y MAXGEN=255
            MRSVR SRVGRP=MBPSGRP2 SRVID=150 MIN=30 MAX=30 RESTART=Y MAXGEN=255


            *SERVICES
            DEFAULT: LOAD=50

            MRSVR



            ULOG:(part)

            112046.jxnxmbps1!MRSVR.3928.1435417904.0: Successful [jxnx_atm] channel total dealing time:[419]ms
            112046.jxnxmbps1!?proc.28661.3567914000.0: 09-21-2012: Tuxedo Version 9.1, 64-bit
            112046.jxnxmbps1!?proc.28661.3567914000.0: LIBTUX_CAT:749: ERROR: Cannot unlock Bulletin Board, not the locker
            112046.jxnxmbps1!MRSVR.3925.455720240.0: Successful [jxnx_atm] channel total dealing time:[209]ms
            ------------------successful channel dealing log-----------------------
            112046.jxnxmbps1!MRSVR.3928.1435417904.0: Successful [jxnx_atm] channel total dealing time:[419]ms
            112046.jxnxmbps1!?proc.28661.3567914000.0: 09-21-2012: Tuxedo Version 9.1, 64-bit
            112046.jxnxmbps1!?proc.28661.3567914000.0: LIBTUX_CAT:749: ERROR: Cannot unlock Bulletin Board, not the locker
            112046.jxnxmbps1!MRSVR.3925.455720240.0: Successful [jxnx_atm] channel total dealing time:[209]ms
            ------------------successful channel dealing log-----------------------
            112046.jxnxmbps1!MRSVR.3928.1435417904.0: Successful [jxnx_atm] channel total dealing time:[419]ms
            112046.jxnxmbps1!?proc.28661.3567914000.0: 09-21-2012: Tuxedo Version 9.1, 64-bit
            112046.jxnxmbps1!?proc.28661.3567914000.0: LIBTUX_CAT:749: ERROR: Cannot unlock Bulletin Board, not the locker
            112046.jxnxmbps1!MRSVR.3925.455720240.0: Successful [jxnx_atm] channel total dealing time:[209]ms
            ------------------successful channel dealing log-----------------------
            112046.jxnxmbps1!MRSVR.3928.1435417904.0: Successful [jxnx_atm] channel total dealing time:[419]ms
            112046.jxnxmbps1!?proc.28661.3567914000.0: 09-21-2012: Tuxedo Version 9.1, 64-bit
            112046.jxnxmbps1!?proc.28661.3567914000.0: LIBTUX_CAT:749: ERROR: Cannot unlock Bulletin Board, not the locker
            112046.jxnxmbps1!MRSVR.3925.455720240.0: Successful [jxnx_atm] channel total dealing time:[209]ms
            ------------------successful channel dealing log-----------------------
            112046.jxnxmbps1!MRSVR.3928.1435417904.0: Successful [jxnx_atm] channel total dealing time:[419]ms
            112046.jxnxmbps1!?proc.28661.3567914000.0: 09-21-2012: Tuxedo Version 9.1, 64-bit
            112046.jxnxmbps1!?proc.28661.3567914000.0: LIBTUX_CAT:749: ERROR: Cannot unlock Bulletin Board, not the locker
            112046.jxnxmbps1!MRSVR.3925.455720240.0: Successful [jxnx_atm] channel total dealing time:[209]ms
            ------------------successful channel dealing log-----------------------
            NOTE:the ULOG top didn't occured on the same day
            after the ULOG top appeared, then did some channel request suceese, the appeared the ULOGS following.

            121923.jxnxmbps1!?proc.29030.1482980368.0: LIBTUX_CAT:582: ERROR: Unable to register, registry table full
            121923.jxnxmbps1!?proc.28157.2587999248.0: LIBTUX_CAT:1122: ERROR: No space in Bulletin Board
            121923.jxnxmbps1!?proc.28157.2587999248.0: LIBTUX_CAT:582: ERROR: Unable to register, registry table full
            121923.jxnxmbps1!?proc.29003.2013240336.0: LIBTUX_CAT:1122: ERROR: No space in Bulletin Board
            121923.jxnxmbps1!?proc.29003.2013240336.0: LIBTUX_CAT:582: ERROR: Unable to register, registry table full
            121923.jxnxmbps1!?proc.28757.4286983184.0: LIBTUX_CAT:1122: ERROR: No space in Bulletin Board
            121923.jxnxmbps1!?proc.28757.4286983184.0: LIBTUX_CAT:582: ERROR: Unable to register, registry table full
            121923.jxnxmbps1!?proc.28632.3356662800.0: LIBTUX_CAT:1122: ERROR: No space in Bulletin Board
            121923.jxnxmbps1!?proc.28632.3356662800.0: LIBTUX_CAT:582: ERROR: Unable to register, registry table full
            121923.jxnxmbps1!?proc.29111.3293035536.0: LIBTUX_CAT:1122: ERROR: No space in Bulletin Board
            121923.jxnxmbps1!?proc.29111.3293035536.0: LIBTUX_CAT:582: ERROR: Unable to register, registry table full
            121923.jxnxmbps1!?proc.28677.911870992.0: LIBTUX_CAT:1122: ERROR: No space in Bulletin Board
            121923.jxnxmbps1!?proc.28677.911870992.0: LIBTUX_CAT:582: ERROR: Unable to register, registry table full
            ----------------------------loop--------------------------------------------------

            NOTE:after the ULOGS top appeared, all request failed.reboot the tuexedo using "tmshutdown -y" and "tmboot -y". it worked normally again.

            Archive:
            [root@eastcom ~]# uname -a
            Linux eastcom 2.6.18-194.el5xen #1 SMP Tue Mar 16 22:01:26 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

            TUXEDO VERSION:
            9.1

            ORACLE version:
            10g


            i'll wait on line.thank you!

            Edited by: 962012 on 2012-9-28 下午6:27

            Edited by: 962012 on 2012-9-29 上午1:00

            Edited by: 962012 on 2012-9-29 上午1:03
            • 4. Re: LIBTUX_CAT:749: ERROR: Cannot unlock Bulletin Board, not the locker
              Bobfinan - Oracle-Oracle
              The information provided is not enough to determine what occurred at the time of the error. I suggest you open a service request to get a more effective and timely response.

              The MAXACCESSERS is 1000 but I can not tell what is the maximum nubmer of servers (including WSHs) and clients you have configured(you have 22 groups defined but not all the servers)

              How do the clients shutdown? Do clients use tpterm ? Do clients immediately try to reconnect (i.e. are they running in a loop or do they exit)?
              Are there processes which die? Are there processes which are still running in the OS but were supposed to have shutdown/exited?
              Are there processes which hang?


              I do not see what patch you have installed. The latest Tuxedo 9.1 rolling patch for Linux is RP160.

              I do not see SYSTEM_ACCESS defined in the ubb. By default it is set to FASTPATH. Setting it to PROTECTED may help but that may effect your performance (or be overwritten by tpinit).

              See the following link for information on MAXACCESSERS and SYSTEM_ACCESS ubb configuration parameters:
              http://docs.oracle.com/cd/E13203_01/tuxedo/tux91/rf5/rf579.htm#1531941

              Edited by: Bob Finan on Oct 1, 2012 12:48 PM
              • 5. Re: LIBTUX_CAT:749: ERROR: Cannot unlock Bulletin Board, not the locker
                965015
                How do the clients shutdown? Do clients use tpterm ? Do clients immediately try to reconnect (i.e. are they running in a loop or do they exit)?
                ---------------clients use tpterm() to shutdown------------------------
                ---------------they exit.there are one thousand process which are receiving request from client using TCP socket.if the process received a request, it called service of tuxedo,using tpinit/tplloc/tpcall/tpterm,eg:
                /* 初始连接tuxedo系统 */
                if (tpinit((TPINIT *) NULL) == -1)
                {
                ErrLogF(__FILE__,__LINE__, COMMANALYZECL_INITFAIL,tperrno,tpstrerror(tperrno));
                return NULL;
                }
                /* 分配系统内存 */
                sendlen=*len+sizeof(struct upanaintf);
                rcvlen=sizeof(struct downanaintf);
                if((sendbuf = (char *) tpalloc("CARRAY", NULL, sendlen+1)) == NULL)
                {
                ErrLogF(__FILE__,__LINE__, COMMANALYZECL_TPALLOCFAIL,tperrno,tpstrerror(tperrno));
                tpterm();
                return NULL;
                }
                if((rcvbuf = (char *) tpalloc("CARRAY", NULL, rcvlen)) == NULL)
                {
                ErrLogF(__FILE__,__LINE__, COMMANALYZECL_TPALLOCFAIL,tperrno,tpstrerror(tperrno));
                tpfree(sendbuf);
                tpterm();
                return NULL;
                }
                /* 封报 */
                memcpy(sendbuf,upinf,sizeof(struct upanaintf));
                memcpy(sendbuf+sizeof(struct upanaintf),commbuf,*len);
                /* 提交 */
                ret = tpcall(svrname, (char *)sendbuf,sendlen , (char **)&rcvbuf, &rcvlen, TPNOTIME);
                .................
                tpfree(sendbuf);
                tpfree(rcvbuf);
                tpterm();
                ------------------------------

                Are there processes which die? Are there processes which are still running in the OS but were supposed to have shutdown/exited?
                Are there processes which hang?
                processed are normal.
                • 6. Re: LIBTUX_CAT:749: ERROR: Cannot unlock Bulletin Board, not the locker
                  Bobfinan - Oracle-Oracle
                  If I understand you correctly, you can have 1000 processes which are potentially doing concurrent client calls to Tuxedo.
                  If these are native processes then you will run out of accessors and get the error message:
                  121923.jxnxmbps1!?proc.29030.1482980368.0: LIBTUX_CAT:582: ERROR: Unable to register, registry table full
                  121923.jxnxmbps1!?proc.28157.2587999248.0: LIBTUX_CAT:1122: ERROR: No space in Bulletin Board

                  Tuxedo maintains tables of resources as part of the bulletin board. The size of the tables it maintains is dependent
                  upon the configuration information you set in the UBB configuration file.

                  In the above link for MAXACCESSERS it explains what is counted as an accesser:
                  "System administration processes, such as the BBL, restartsrv, cleanupsrv, tmshutdown(), and tmadmin(), need not be accounted for in this value, but the DBBL, all bridge processes, all system-supplied and application server processes, and all potential client processes at a particular site need to be counted. (Examples of system-supplied servers are AUTHSVR, TMQUEUE, TMQFORWARD, TMUSREVT, TMSYSEVT, TMS—see TMSNAME parameter in GROUPS section, TMS_QM, GWTDOMAIN, and WSL.) If the application is booting workstation listeners (WSLs) at a particular site, both the WSLs and the number of potential workstation handlers (WSHs) that may be booted need to be counted."

                  You will need to increase the number of MAXACCESSERS to be the 1000 native clients plus the other processes(Tuxedo admin serveirs, application servers, WSL+WSHs) you have configued in the UBB.
                  • 7. Re: LIBTUX_CAT:749: ERROR: Cannot unlock Bulletin Board, not the locker
                    965015
                    If these are native processes then you will run out of accessors and get the error message:
                    there are one thousand process receiving client request,but the number of concurrent request could not be one thousand.It should only be dozens .

                    ULOG
                    112450.jxnxmbps1!MRSVR.3927.307264816.0: Successful [jxnx_atm] channel total dealing time:[595]ms
                    112450.jxnxmbps1!MRSVR.3925.455720240.0: Successful [jxnx_atm] channel total dealing time:[264]ms
                    112451.jxnxmbps1!?proc.28160.4171639824.0: 09-21-2012: Tuxedo Version 9.1, 64-bit
                    112451.jxnxmbps1!?proc.28160.4171639824.0: LIBTUX_CAT:1122: ERROR: No space in Bulletin Board
                    112451.jxnxmbps1!?proc.28160.4171639824.0: LIBTUX_CAT:582: ERROR: Unable to register, registry table full
                    112452.jxnxmbps1!?proc.28298.1302678544.0: 09-21-2012: Tuxedo Version 9.1, 64-bit
                    112452.jxnxmbps1!?proc.28298.1302678544.0: LIBTUX_CAT:1122: ERROR: No space in Bulletin Board
                    112452.jxnxmbps1!?proc.28298.1302678544.0: LIBTUX_CAT:582: ERROR: Unable to register, registry table full
                    112452.jxnxmbps1!MRSVR.3929.4262587696.0: Successful [jxnx_atm] channel total dealing time:[174]ms
                    112452.jxnxmbps1!MRSVR.3928.1435417904.0: Successful [jxnx_atm] channel total dealing time:[171]ms
                    112452.jxnxmbps1!?proc.28932.35171344.0: 09-21-2012: Tuxedo Version 9.1, 64-bit
                    112452.jxnxmbps1!?proc.28932.35171344.0: LIBTUX_CAT:1122: ERROR: No space in Bulletin Board
                    112452.jxnxmbps1!?proc.28932.35171344.0: LIBTUX_CAT:582: ERROR: Unable to register, registry table full
                    112452.jxnxmbps1!MRSVR.3925.455720240.0: Successful [jxnx_atm] channel total dealing time:[1988]ms
                    112452.jxnxmbps1!MRSVR.3928.1435417904.0: Successful [jxnx_atm] channel total dealing time:[198]ms
                    112452.jxnxmbps1!MRSVR.3927.307264816.0: Successful [jxnx_atm] channel total dealing time:[953]ms
                    112452.jxnxmbps1!MRSVR.3928.1435417904.0: Successful [jxnx_atm] channel total dealing time:[47]ms
                    112452.jxnxmbps1!MRSVR.3926.3325127984.0: Successful [jxnx_atm] channel total dealing time:[533]ms
                    112453.jxnxmbps1!MRSVR.3927.307264816.0: Successful [jxnx_atm] channel total dealing time:[79]ms
                    112453.jxnxmbps1!?proc.28169.3756653584.0: 09-21-2012: Tuxedo Version 9.1, 64-bit
                    112453.jxnxmbps1!?proc.28169.3756653584.0: LIBTUX_CAT:1122: ERROR: No space in Bulletin Board
                    112453.jxnxmbps1!?proc.28169.3756653584.0: LIBTUX_CAT:582: ERROR: Unable to register, registry table full
                    112453.jxnxmbps1!?proc.29029.4283046928.0: 09-21-2012: Tuxedo Version 9.1, 64-bit
                    112453.jxnxmbps1!?proc.29029.4283046928.0: LIBTUX_CAT:1122: ERROR: No space in Bulletin Board
                    112453.jxnxmbps1!?proc.29029.4283046928.0: LIBTUX_CAT:582: ERROR: Unable to register, registry table full
                    112453.jxnxmbps1!?proc.28895.3282689040.0: 09-21-2012: Tuxedo Version 9.1, 64-bit
                    112453.jxnxmbps1!?proc.28895.3282689040.0: LIBTUX_CAT:1122: ERROR: No space in Bulletin Board
                    112453.jxnxmbps1!?proc.28895.3282689040.0: LIBTUX_CAT:582: ERROR: Unable to register, registry table full
                    -------------------------------there is not any success like "Successful [jxnx_atm] channel total dealing time:[198]ms " any more-----------------------

                    Edited by: 962012 on 2012-10-8 下午10:55