Custom Query (101 matches)

with this

Traceback (most recent call last):
  File "/usr/sbin/jobmond", line 2203, in <module>
  File "/usr/sbin/jobmond", line 2198, in main
  File "/usr/sbin/jobmond", line 1073, in run
  File "/usr/sbin/jobmond", line 909, in submitJobData
  File "/usr/sbin/jobmond", line 759, in multicastGmetric
  File "/usr/sbin/jobmond", line 1969, in __init__
  File "/usr/lib/python2.7/socket.py", line 187, in __init__
socket.error: [Errno 24] Too many open files

something

At one time, I had many Torque queues: one for each group of homogeneous systems. Since I couldn't rely on my users to consistently check qstat, showq, or Ganglia before submitting a job to an queue with free CPUs rather than a queue with none, I converted my Torque settings to put all cluster systems into one queue, and use Maui partitions to keep parallel jobs on a group of homogeneous systems. This has worked out great as far as queue efficiency is concerned.

Now that I'm getting Job Monarch integrated into the setup, I've noticed that active jobs in my batch queue show up in all cluster joblists and overviews, even when that particular cluster has no active jobs on its nodes. I'll try to attach screenshots, but if my users still have jobs running when you read this ticket, you can see for yourself on the live server:

• ​Cluster view normal, this one had all the active jobs

• ​Cluster with extra jobs and nodes showing

Ganglia knows, for example, that "ChE Compute Nodes" has 9 systems in it named ch226-11 ... ch226-19. Monarch displays those, but also displays ch226-29 and ch226-31 from the "PNGV Project Compute Nodes" cluster that had active jobs. Any cluster view where Monarch was enabled had this effect.