5 Replies Latest reply on Jun 1, 2011 3:37 PM by 830663

    Advice on License Queuing Scheme on 6.2u5


      I would like some ideas on how to manage a set of jobs that require a specific license in order to run with some specific restraints on how many licenses can be consumed. I already have a license queuing scheme where I create a request-able complex for each license that can be requested. The problem I am trying to resolve is for a group of users whose jobs individually can either completely overwhelm the cluster or use all the available licenses ( I want to try and leave a few for interactive use). What I want to do is dynamically adjust what each user sees for available licenses, sort of. I want any individual user to be allowed to use only 75% of the available license. Users can queue as many jobs as desired, but will be limited to how many will run. I will try to make this more clear with an example..

      lic_avail_total = 100 ( The total number of available licenses )
      lic_used = 50 ( The total number of licenses currently being used )
      lic_avail = 50 ( The number of licenses currently available to users )
      license_used_by_fred = 35 ( The number of licenses currently used by user fred )

      Given the above numbers, the user fred is using 70% of the available 50 licenses. So user fred should be able to acquire 2 more licenses based on the 75% limit. If the number of available licenses increases, user fred would be able to acquire more. We are currently solving this by basically having each user run a set of scripts that creates a sort of off line queue for their jobs and only submits jobs when licensees are free. In doing this we lose visibility to how many licenses are really needed, and how many jobs are actually queued.

      As a first pass, I am thinking that I will create a holding queue that has no resources and requires a forced complex. Then jobs will sit in the queue until they have been released to run by a script that scans the queue once a minute or so. The script would basically identify all users running jobs and license utilized, then alter the submit options of jobs that can be run. I would need to take into account available licenses, job priority and used licenses by user.

      Is there a better way to do this? Are there some tricks that I can use with rqs, jvs, etc.. To achieve what I want?

      Any ideas or thoughts are welcome.


        • 1. Re: Advice on License Queuing Scheme on 6.2u5
          Hello Michael,
          I think I understand what you are doing and there is probably more than one way to achieve this. The limiting of total number of licenses in the cluster is quite simple and can be done using a complex. Create a new complex using qconf -mc, for example:

          #name shortcut type relop requestable consumable default urgency
          license lic INT <= YES YES 1 0

          Then using qconf -me global, add this complex value to the global host configuration with the number of total licenses available in the system, in your case 1000.

          complex_values license=1000

          Then anytime a user submits a job with a request for a license ex. qsub -l license=1 .... then it will mean only 999 licenses remain to be used.
          This can be seen in the output of qhost -F

          As for the second part of your question regarding dynamically adjusting what each users sees, this is a bit more complex and I need to do a bit more investigation into how this can be done.
          Hope this gives you a good start,

          • 2. Re: Advice on License Queuing Scheme on 6.2u5
            Thanks for the response. As my message stated, I am already using a complex to manage overall license availability.

            • 3. Re: Advice on License Queuing Scheme on 6.2u5
              Ok, I guess I somehow overlooked that. I am still looking into the dynamic part of the issue.

              • 4. Re: Advice on License Queuing Scheme on 6.2u5
                Hello Michael,
                I had been looking around and trying to think of different ways that your goal can be achieved. Unfortunately your requirement to have the dynamically changing resources makes it quite difficult to implement. However I have come up with a solution that works for a fixed number of licenses.
                Since your cluster is already setup to have a maximum number or licenses running, there is no additional modifications for this requirement. And to be able to limit the user to a certain amount of licenses, you can do this by creating a resource quota set (qconf -arqs). Below is an example:

                name user_lic_to_35
                description "limit each user to a maximum of 35 licenses"
                enabled TRUE
                limit users {*} to license=35

                In this example, no user can have running jobs requesting more than 35 licenses. In a simple case of qsub -l license=1 ... this would mean that a single user is limited to 35 running jobs. Any additionally submitted jobs will just be queued until a free licenses is available. In addition, since you have a hard limit of 100 licenses on the global execution host, than if two users are already running jobs with 35 licenses each, than a third user can only submit and run 30 jobs that request a license. Of course the limit "license" would have to be the name of the complex that you created in your cluster.

                I know this does not fully accomplish what you want, but maybe it helps you out a bit with limiting users to a certain amount of licenses.

                I am still looking into how the dynamic part of this can be solved. Stay tuned...

                Edited by: mpospisil on Jun 1, 2011 7:19 AM
                • 5. Re: Advice on License Queuing Scheme on 6.2u5
                  Thanks for the idea of using rqs. Unfortunately, this still allows the users to fairly easily consume all the available license. I really want it to be more like a fair share that always leaves a few licenses on the table for interactive or other use. I am still thinking my best option is to create a 'holding queue' then write a script to parse the queue every so often and release jobs based on license availability and the users current usage. But I am open to ideas. Thanks for the comments so far. I do appreciate them.