12 Replies Latest reply on Feb 4, 2020 10:09 AM by 2937321

    OL 7.7 systemd: New main PID xxxx does not belong to service...

    Dude!

      Hi,

       

      I noticed the following since the recent 7.7 update:

       

      Aug 16 19:52:01 vm7031.example.com systemd[1]: Starting Remote desktop service (VNC)...

      Aug 16 19:52:05 vm7031.example.com systemd[1]: New main PID 2241 does not belong to service, and PID file is not owned by root. Refusing.

      Aug 16 19:52:05 vm7031.example.com systemd[1]: Failed to start Remote desktop service (VNC).

      Aug 16 19:52:05 vm7031.example.com systemd[1]: Unit vncserver@:95.service entered failed state.

      Aug 16 19:52:05 vm7031.example.com systemd[1]: vncserver@:95.service failed.

       

      Apparently there's been some change how systemd handles PID files.

       

      This affects the vncserver service when using /lib/systemd/system/vncserver@.service. The file was already modified with the release of 7.4, but has not been updated to work for 7.7. Starting with release 8, vncserver has been changed again using systemctl --user, which further complicates the matter as outlined in OL8: Failed to connect to bus: No such file or directory

       

      According to the TigerVNC deverloper forum the changes are not made by TigerVNC. So it's probably Red Hat making these changes. Anyway, I decided not use the vncserver template anymore and create my one vncservice unit.

       

      I have updated vncpilot to work with 7.0 - 7.7 and also created a new version for release 8.

       

      Btw, release 8 no longer ships with xorg-x11-apps and requires to enable the CodeReady Builder repository, in case you need xclock, etc. To my experience, xterm and other ancient X tools are no longer functioning correctly anymore.

       

      I don't know if anyone else uses vncpilot, but it certainly makes my life easier whenever I need to run an X app using secure VNC ad hoc.

       

      VNCpilot - Simple And Secure Remote Access

       

      Cheers.

        • 1. Re: OL 7.7 systemd: New main PID xxxx does not belong to service...
          Avi Miller-Oracle

          Thanks, I'll ask the development team to investigate.

          • 2. Re: OL 7.7 systemd: New main PID xxxx does not belong to service...
            Dude!

            I think the following is related:

            https://github.com/systemd/systemd/issues/8085

            https://github.com/systemd/systemd/commit/db256aab13d8a89d583ecd2bacf0aca87c66effc

             

            In case of the vncserver service unit and OS release 8, the PID file (or variable) was removed.

             

            EL7: /usr/lib/systemd/system/vncserver\@.service

             

            ExecStartPre=/bin/sh -c '/usr/bin/vncserver -kill %i > /dev/null 2>&1 || :'

            ExecStart=/usr/sbin/runuser -l <USER> -c "/usr/bin/vncserver %i"

            PIDFile=/home/<USER>/.vnc/%H%i.pid

            ExecStop=/bin/sh -c '/usr/bin/vncserver -kill %i > /dev/null 2>&1 || :'

             

            EL8: /usr/lib/systemd/user/vncserver@.service

             

            ExecStartPre=/bin/sh -c '/usr/bin/vncserver -kill %i > /dev/null 2>&1 || :'

            ExecStart=/usr/bin/vncserver %i

            ExecStop=/usr/bin/vncserver -kill %i

            • 3. Re: OL 7.7 systemd: New main PID xxxx does not belong to service...
              Tomáš Glozar

              The entire service file seems to be Red Hat-specific, as you can see here: https://git.centos.org/rpms/tigervnc/blob/c7/f/SOURCES/vncserver.service.

               

              Unfortunately you cannot view the history for the file because there is an error - the new CentOS/RHEL source repository uses Pagure, and it breaks quite a lot.

               

              Edit: By Red Hat-specific I didn't necessarily mean written by Red Hat; I meant that it is provided as a whole file in the repo instead of a patch to an existing file.

              • 4. Re: OL 7.7 systemd: New main PID xxxx does not belong to service...
                Marcelo Marques

                I am having the same problem with OEL 7.7

                 

                # systemctl start vncserver@:3.service

                Job for vncserver@:3.service failed because a configured resource limit was exceeded. See "systemctl status vncserver@:3.service" and "journalctl -xe" for details.

                 

                # systemctl status vncserver@:3.service

                ...

                Sep 11 14:30:10 mcsdboralnx5.esri.com systemd[1]: Starting Remote desktop service (VNC)...

                Sep 11 14:30:13 mcsdboralnx5.esri.com systemd[1]: New main PID 20733 does not belong to service, and PID file is not owned by root. Refusing.

                ...

                 

                I opened a ticket with RedHat and they pointed me to the bugzilla bug below.

                Once it is OEL 7.7 OS the RedHat Support closed my case. I decided to open a SR to notify Oracle Support as well.

                The vncserver daemon is still active and can accept connections, but it cannot be managed by systemd.

                 

                ------

                https://bugzilla.redhat.com/show_bug.cgi?id=1747191

                "Bug 1747191 - tigervnc service via unit file fails with PID file error"

                Actual results: Service fails with the error "New main PID $PIDNUM does not belong to service, and PID file is not owned by root.

                Refusing." The VNC daemon is still active and can accept connections; however, it cannot be managed by systemd at this point.

                Expected results: Service runs normally without issue.

                Additional info: This appears to be the same bug as bug 1583159 filed against Fedora 27. While replacement unit files were provided in that bug, by themselves they don't appear to work completely in RHEL 7.7; the daemons start and systemd reads it as started, but attempting to log in via GDM fails. There may be additional work to be done.

                -----

                 

                I hope this helps.

                 

                Thanks,

                Marcelo Marques

                Technical Manager, OCP

                Esri - www.esri.com

                • 5. Re: OL 7.7 systemd: New main PID xxxx does not belong to service...
                  user8707233

                  Did a little more digging as I just stepped in this same issue on tigervnc-server 1.8.0 when I upgrade to centos 7.7 earlier today.

                   

                  I think we may be looking in the wrong place. Output from my journalctl -xe shows this:

                   

                  -- The start-up result is done.

                  Oct 03 17:48:08 berne runuser[23923]: pam_limits(runuser-l:session): invalid line 'ulimit -u 50000' - skipped    <<<<<<<<=================

                  Oct 03 17:48:08 berne runuser[23923]: pam_unix(runuser-l:session): session opened for user virt by (uid=0)

                  Oct 03 17:48:08 berne runuser[23923]: Warning: berne:1 is taken because of /tmp/.X1-lock

                  Oct 03 17:48:08 berne runuser[23923]: Remove this file if there is no X server berne:1

                  Oct 03 17:48:08 berne runuser[23923]: A VNC server is already running as :1

                  Oct 03 17:48:08 berne runuser[23923]: Warning: berne:1 is taken because of /tmp/.X1-lock

                  Oct 03 17:48:08 berne runuser[23923]: Remove this file if there is no X server berne:1

                  Oct 03 17:48:08 berne runuser[23923]: Warning: berne:2 is taken because of /tmp/.X2-lock

                  Oct 03 17:48:08 berne runuser[23923]: Remove this file if there is no X server berne:2

                  Oct 03 17:48:08 berne runuser[23923]: Warning: berne:3 is taken because of /tmp/.X3-lock

                  Oct 03 17:48:08 berne runuser[23923]: Remove this file if there is no X server berne:3

                  Oct 03 17:48:08 berne runuser[23923]: Warning: berne:4 is taken because of /tmp/.X4-lock

                  Oct 03 17:48:08 berne runuser[23923]: Remove this file if there is no X server berne:4

                  Oct 03 17:48:08 berne runuser[23923]: Warning: berne:5 is taken because of /tmp/.X5-lock

                  Oct 03 17:48:08 berne runuser[23923]: Remove this file if there is no X server berne:5

                  Oct 03 17:48:08 berne runuser[23923]: Warning: berne:6 is taken because of /tmp/.X6-lock

                  Oct 03 17:48:08 berne runuser[23923]: Remove this file if there is no X server berne:6

                  Oct 03 17:48:08 berne runuser[23923]: Warning: berne:7 is taken because of /tmp/.X7-lock

                  Oct 03 17:48:08 berne runuser[23923]: Remove this file if there is no X server berne:7

                  Oct 03 17:48:08 berne runuser[23923]: Warning: berne:8 is taken because of /tmp/.X8-lock

                  Oct 03 17:48:08 berne runuser[23923]: Remove this file if there is no X server berne:8

                  Oct 03 17:48:11 berne runuser[23923]: New 'berne:10 (virt)' desktop is berne:10

                  Oct 03 17:48:11 berne runuser[23923]: Starting applications specified in /home/virt/.vnc/xstartup

                  Oct 03 17:48:11 berne runuser[23923]: Log file is /home/virt/.vnc/berne:10.log

                  Oct 03 17:48:11 berne runuser[23923]: pam_unix(runuser-l:session): session closed for user virt

                  Oct 03 17:48:11 berne systemd[1]: Can't open PID file /run/virt.vnc.berne:1.pid (yet?) after start: No such file or directory

                  Oct 03 17:48:11 berne systemd[1]: Failed to start Remote desktop service (VNC).

                  -- Subject: Unit vncserver@:1.service has failed

                  • 6. Re: OL 7.7 systemd: New main PID xxxx does not belong to service...
                    Dude!

                    I'd say you are experiencing a different problem. The error you are seeing occurs when you start the  Xvnc server using a port that is already in use. Your Xvnc or TigerVNC server could be freezing or crashing for several reasons. You need to quit the previous Xvnc process before you an start a new process using the same configuration. The -lock files contain the process ID of the Xvnc process. When you stop the Xvnc process, the corresponding -lock file is usually removed.

                     

                    vncpilot handles the situation as following, for example:

                     

                    [root@vm7051 tmp]# ls -al .*-lock

                    ls: cannot access .*-lock: No such file or directory

                     

                    [root@vm7051 tmp]# vncpilot -c -size=1280x800

                    [root@vm7051 tmp]# cat .X95-lock

                          2538

                     

                    [root@vm7051 tmp]# ps -ef | grep 2538

                    vnc-410   2538     1  0 18:36 ?        00:00:00 /usr/bin/Xvnc :95 -auth /home/vnc-410/.Xauthority -depth 16 -desktop vm7051.example.com:95 (vnc-410) -fp catalogue:/etc/X11/fontpath.d -geometry 1280x800 -pn -rfbauth /home/vnc-410/.vnc/passwd -rfbport 5995 -rfbwait 30000 -nevershared -localhost -SecurityTypes=vncauth

                     

                    [root@vm7051 tmp]# vncpilot -c -size=1280x800

                    VNCpilot 1.0.2. Copyright (c) 2018, 2019 Dude! @ Oracle Community.

                    Free under the GNU General Public License. See vncpilot --help for info.

                     

                    %vncpilot-E-169, TCP port 5995 already in use.

                     

                    [root@vm7051 tmp]# vncpilot -d

                    [root@vm7051 tmp]# ls -al .*-lock

                    ls: cannot access .*-lock: No such file or directory

                    • 7. Re: OL 7.7 systemd: New main PID xxxx does not belong to service...
                      user8707233

                      You are correct WRT shutting down previous process. As systemd apparently no longer controls it, I manually killed it and wiped locks  (/tmp/.X*lock).

                       

                      The current lock points to the process:

                       

                      [root@berne virt]# cat /tmp/.X14-lock

                           32739

                      [root@berne virt]# ps -ef|grep 32739

                      root      1540 32269  0 20:15 pts/0    00:00:00 grep --color=auto 32739

                      virt     32739     1  0 20:10 ?        00:00:00 /usr/bin/Xvnc :14 -geometry 1600x1024 -auth /home/virt/.Xauthority -desktop berne:14 (virt) -fp catalogue:/etc/X11/fontpath.d -pn -rfbauth /home/virt/.vnc/passwd -rfbport 5914 -rfbwait 30000 -Log *:stderr:100

                       

                      Noting that VNC is running on :14 and not on the expected :1 or :2 (I have two VNC services), I quickly adjusted the port before my F/W mapping and was able to successfully access my box.

                       

                      I suspect there is no problem with the pid files as they are all present:

                       

                      [root@berne virt]# ls -ltr .vnc/*

                      -rw-rw-r--. 1 virt virt 1008 Jan 29  2018 .vnc/berne:2.log

                      -rw-r--r--. 1 virt virt  330 Jan 29  2018 .vnc/config

                      -rw-------. 1 virt virt    8 Jul 19  2018 .vnc/passwd

                      -rwxr-xr-x. 1 virt virt  950 Jul 19  2018 .vnc/xstartup

                      -rw-rw-r--  1 virt virt 1740 Oct  3 17:11 .vnc/berne:1.log

                      -rw-rw-r--  1 virt virt    6 Oct  3 17:16 .vnc/berne:4.pid

                      -rw-rw-r--  1 virt virt 1740 Oct  3 17:21 .vnc/berne:4.log

                      -rw-rw-r--  1 virt virt    6 Oct  3 17:39 .vnc/berne:6.pid

                      -rw-rw-r--  1 virt virt    6 Oct  3 17:39 .vnc/berne:7.pid

                      -rw-rw-r--  1 virt virt 1740 Oct  3 17:44 .vnc/berne:6.log

                      -rw-rw-r--  1 virt virt 1740 Oct  3 17:45 .vnc/berne:7.log

                      -rw-rw-r--  1 virt virt    6 Oct  3 17:45 .vnc/berne:8.pid

                      -rw-rw-r--  1 virt virt 1986 Oct  3 17:47 .vnc/berne:8.log

                      -rw-rw-r--  1 virt virt    6 Oct  3 17:47 .vnc/berne:9.pid

                      -rw-rw-r--  1 virt virt    6 Oct  3 17:48 .vnc/berne:10.pid

                      -rw-rw-r--  1 virt virt 2845 Oct  3 17:53 .vnc/berne:10.log

                      -rw-rw-r--  1 virt virt 2753 Oct  3 17:53 .vnc/berne:9.log

                      -rw-rw-r--  1 virt virt    6 Oct  3 17:53 .vnc/berne:11.pid

                      -rw-rw-r--  1 virt virt 2757 Oct  3 20:00 .vnc/berne:11.log

                      -rw-rw-r--  1 virt virt    6 Oct  3 20:01 .vnc/berne:12.pid

                      -rw-rw-r--  1 virt virt 2078 Oct  3 20:02 .vnc/berne:12.log

                      -rw-rw-r--  1 virt virt  310 Oct  3 20:02 .vnc/berne:13.log

                      -rw-rw-r--  1 virt virt    6 Oct  3 20:10 .vnc/berne:14.pid

                      -rw-rw-r--  1 virt virt  974 Oct  3 20:10 .vnc/berne:14.log

                       

                      which makes no sense as I had clearly set the PIDFile to be created elsewhere:

                       

                      [root@berne virt]# grep PID /etc/systemd/system/vncserver@\:1.service

                      #     PIDFile=/home/<USER>/.vnc/%H%i.pid)

                      #PIDFile=/home/virt/.vnc/%H%i.pid

                      PIDFile=/run/virt.vnc.%H%i.pid

                       

                      and had reloaded systemctl with no result.

                      • 8. Re: OL 7.7 systemd: New main PID xxxx does not belong to service...
                        user8707233

                        I think I finally understand the root symptom that is causing this weird PID issue for tigervnc.

                         

                        I am seeing that everytime I bounce the service from systemctl, I get the next display number instantiated in the .vnc directory:

                         

                        [root@berne .vnc]# ls -ltr

                        total 28

                        -rw-r--r--. 1 virt virt   330 Jan 29  2018 config

                        -rw-------. 1 virt virt     8 Jul 19  2018 passwd

                        -rwxr-xr-x. 1 virt virt   950 Jul 19  2018 xstartup

                        -rw-rw-r--  1 virt virt     5 Oct  3 21:00 berne:17.pid

                        -rw-rw-r--  1 virt virt 11587 Oct  3 21:10 berne:17.log

                         

                        yet, the service still believes it should find the display number as specified in the service naming, a 1 here.

                         

                        [root@berne .vnc]# !syst:p

                        systemctl start vncserver@:1

                        [root@berne .vnc]# systemctl -l status vncserver@:1

                        ● vncserver@:1.service - Remote desktop service (VNC)

                           Loaded: loaded (/etc/systemd/system/vncserver@:1.service; enabled; vendor preset: disabled)

                           Active: failed (Result: resources) since Thu 2019-10-03 21:00:55 EDT; 10min ago

                          Process: 8647 ExecStart=/usr/sbin/runuser -l virt -c /usr/bin/vncserver %i -geometry 1800x900 -Log '*:stderr:100' (code=exited, status=0/SUCCESS)

                          Process: 8589 ExecStartPre=/bin/sh -c /usr/bin/vncserver -kill %i > /dev/null 2>&1 || : (code=exited, status=0/SUCCESS)

                         

                        and issues its beef when it can't find that pid as it some how is the next display number, 17 here.

                         

                        Oct 03 21:00:52 berne systemd[1]: Starting Remote desktop service (VNC)...

                        Oct 03 21:00:55 berne systemd[1]: Can't open PID file /home/virt/.vnc/berne:1.pid (yet?) after start: No such file or directory

                        Oct 03 21:00:55 berne systemd[1]: Failed to start Remote desktop service (VNC).

                        Oct 03 21:00:55 berne systemd[1]: Unit vncserver@:1.service entered failed state.

                        Oct 03 21:00:55 berne systemd[1]: vncserver@:1.service failed.

                         

                        I guess I can try reverting the systemd rpm back to the last one under an intact centos 7.6 box and see if this headache goes away. But I believe this is the root cause, systemd is monotonically receiving the next display number from the service naming and passing this to the vnc startup scripts.

                         

                        This is the root problem:

                         

                        To start or enable the service, specify the display number directly in the command. The file configured above works as a template, in which %i is substituted with the display number by systemd.

                        • 9. Re: OL 7.7 systemd: New main PID xxxx does not belong to service...
                          user8707233

                          I have also just confirmed that wiping the sockets under /tmp/.X11-unix will prevent the increasing display number:

                           

                          [root@berne .vnc]# ls -ltr /tmp/.X11-unix

                          total 0

                          srwxrwxrwx 1 root root 0 Oct  3 21:00 X0

                          srwxrwxrwx 1 virt virt 0 Oct  3 21:40 X1

                          [root@berne .vnc]# lsof /tmp/.X11-unix

                          lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/25584/gvfs

                                Output information may be incomplete.

                          [root@berne .vnc]# lsof /tmp/.X11-unix/X?

                          lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/25584/gvfs

                                Output information may be incomplete.

                          COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF   NODE NAME

                          X        8687 root    6u  unix 0xffff9465562a3400      0t0 469090 /tmp/.X11-unix/X0

                          Xvnc    13401 virt    8u  unix 0xffff9464fc771000      0t0 580849 /tmp/.X11-unix/X1

                           

                          But even having created the correct display number:

                           

                          [root@berne .vnc]# ps -ef|grep vnc

                          virt     13401     1  0 21:40 ?        00:00:00 /usr/bin/Xvnc :1 -geometry 1600x1024 -auth /home/virt/.Xauthority -desktop berne:1 (virt) -fp catalogue:/etc/X11/fontpath.d -pn -rfbauth /home/virt/.vnc/passwd -rfbport 5901 -rfbwait 30000 -Log *:stderr:100

                           

                          The status is upset at not finding the PID due to ownership now:

                           

                          [root@berne .vnc]# systemctl -l status vncserver@:1

                          ● vncserver@:1.service - Remote desktop service (VNC)

                             Loaded: loaded (/etc/systemd/system/vncserver@:1.service; enabled; vendor preset: disabled)

                             Active: failed (Result: resources) since Thu 2019-10-03 21:40:11 EDT; 11s ago

                            Process: 13370 ExecStart=/usr/sbin/runuser -l virt -c /usr/bin/vncserver %i -geometry 1800x900 -Log '*:stderr:100' (code=exited, status=0/SUCCESS)

                            Process: 13364 ExecStartPre=/bin/sh -c /usr/bin/vncserver -kill %i > /dev/null 2>&1 || : (code=exited, status=0/SUCCESS)

                           

                           

                          Oct 03 21:40:08 berne systemd[1]: Starting Remote desktop service (VNC)...

                          Oct 03 21:40:11 berne systemd[1]: New main PID 13401 does not belong to service, and PID file is not owned by root. Refusing.

                          Oct 03 21:40:11 berne systemd[1]: New main PID 13401 does not belong to service, and PID file is not owned by root. Refusing.

                          Oct 03 21:40:11 berne systemd[1]: Failed to start Remote desktop service (VNC).

                          Oct 03 21:40:11 berne systemd[1]: Unit vncserver@:1.service entered failed state.

                          Oct 03 21:40:11 berne systemd[1]: vncserver@:1.service failed.

                           

                          and obviously there is no pid file and I can still connect with no problem:

                           

                          [root@berne .vnc]# ls -ltr

                          total 20

                          -rw-r--r--. 1 virt virt  330 Jan 29  2018 config

                          -rw-------. 1 virt virt    8 Jul 19  2018 passwd

                          -rwxr-xr-x. 1 virt virt  950 Jul 19  2018 xstartup

                          -rw-rw-r--  1 virt virt 6093 Oct  3 21:47 berne:1.log

                          • 10. Re: OL 7.7 systemd: New main PID xxxx does not belong to service...
                            user8372473

                            Hello all. What worked for me in 7.7 was to modify ExecStart in my systemd service file, which was created from the default template (/usr/lib/systemd/system/vncserver@.service), to:

                             

                            ExecStart=/usr/bin/vncserver %i

                             

                            I also added to the [Service] section:

                             

                            User=<YOUR_VNC_USER>

                             

                            The root cause seems to be that the upstream vendor hardened systemd security (https://access.redhat.com/errata/RHSA-2019:2091) as a result of a discovered vulnerability (https://access.redhat.com/security/cve/cve-2018-16888). All this affected the ability to execute the tigervnc-server binary as non-root using the runuser command.

                             

                            Hope this helps someone. Cheers.

                            • 11. Re: OL 7.7 systemd: New main PID xxxx does not belong to service...
                              Dude!

                              ExecStart=/usr/bin/vncserver %i

                               

                              I guess that depends on what you wish to accomplish, but for remote access you will probably want some security and allow access only via SSH tunneling and also use VNC authentication.

                               

                              For example adding  -localhost -SecurityTypes=vncauth

                               

                              Also unless you have Gnome installed, you will need some additional software.

                              • 12. Re: OL 7.7 systemd: New main PID xxxx does not belong to service...
                                2937321

                                To make statement made by 'user8372473' more clear, your vncserver-$USER@.service file should look like below

                                 

                                ===================================

                                 

                                [Unit]

                                Description=Remote desktop service (VNC)

                                After=syslog.target network.target

                                 

                                [Service]

                                Type=forking

                                User=<USER>

                                WorkingDirectory=/home/<USER>

                                 

                                # Clean any existing files in /tmp/.X11-unix environment

                                ExecStartPre=/bin/sh -c '/usr/bin/vncserver -kill %i > /dev/null 2>&1 || :'

                                ExecStart=/usr/bin/vncserver %i -geometry 1600x900

                                PIDFile=/home/<USER>/.vnc/%H%i.pid

                                ExecStop=/bin/sh -c '/usr/bin/vncserver -kill %i > /dev/null 2>&1 || :'

                                 

                                [Install]

                                WantedBy=multi-user.target

                                 

                                ===================================

                                Instead of older way of using 'runuser'/'runas'/'su -c', let Systemd run the service under user which should own the service.