source: trunk/CHANGES @ 286

Last change on this file since 286 was 286, checked in by bas, 10 years ago

PBSacct framework for accounting records, beta
list some changes

  • Property svn:keywords set to Id
File size: 21.8 KB
Line 
1=========== 4.X.X
2 * PBSQuery bugfix, parsing of node status line with multiple EVENT: settings
3   was wrong
4   Author: Bas van der Vlies
5
6=========== 4.3.4
7 * PBSQuery bugfix. getjob() can now handle short jobname, eg:
8    * p.getjob('6073545')
9
10   Author: Willem Vermin (SARA) & Bas van der Vlies
11
12 * added an example: pbs_jobmonitor shows the output of top
13   -u user on the node
14    - one cycle of top
15    - user: the user the job belongs to
16
17   renamed as pbs_joblogin logs in to the node as the user
18   who invokes this script (os.getenv('USER'))
19
20   Author: Willem Vermin (SARA)
21 
22=========== 4.3.3
23 * New generated files for pbs_wrap.c and pbs.py to support python 3.X versions
24   Reported by: Steve Traylen <steve dot traylen add cern dot ch>
25   Author: Bas van der Vlies
26
27 * Fixed AdvancedParser when using 01-12 range the zero was not appended.
28   Reported by: Ramon Bastiaans
29   Author: Dennis Stam
30
31 * examples/sara_nodes.py: Catch the PBSQuery error if we can not make a
32   connection with the batch server and exit the program.
33   Author: Bas van der Vlies
34
35 * Remove the debian dependency on AdvancedParser. We now have
36   our own PBSAdvanceParser so it does not conflict with other SARA
37   packages.
38   Reported by: Ramon Bastiaans (SARA)
39   Fixed by: Bas van der Vlies
40
41 * Make pbs_torque only dependend on libtorque instead of torque and adjusted
42   the maintainer of the package, closes #30
43   Reported by: Guillermo Marcus  <guillermo dot marcus add@gmail dot com>
44   Fixed by: Bas van der Vlies
45   
46=========== 4.3.0
47 * new_rack_pbsmon.py: For backward compatibility added -a/--all option.
48   Has the same functionality as -s/--summary
49   Author: Bas van der Vlies
50
51 * Added some license info to debian package and include GPL and LGPL 3.0
52   files.
53   Suggested by: W. Trevor King <wking at drexel dot edu>
54
55 * PBSQuery.py: Added support for new feature in node status field:
56    * message=EVENT:<key=value:>+
57    * message=ERROR <text>
58
59  Added the following syntax for node:
60        * node['error']
61    * node['event']
62
63  eg: node: n1
64    state = free
65    np = 8
66    properties = ib,switch1,highmem
67    ntype = cluster
68    jobs = 0/567403.sara.nl, 1/567403.sara.nl
69    status = ..,loadave=0.00,message=EVENT:sample.time=1288864220.003:cputotals.user=0:iconnect.pktout=0,netload=3487600394,state=free,...
70
71  example code:
72        p = PBSQuery()
73    node = p.getnode(n1)
74    print node.event['sample.time']
75    print node['event']['sample.time']
76
77   Requested by: Roy Dragseth <roy dot dragseth add uit dotno>
78   Author: Bas van der Vlies
79
80=========== 4.2.0
81 * Updated LICENSE file to LGPLV3. Thanks to Justin Bronder
82   (jsbronder add gentoo dot org) for the information and
83   explanation.
84
85 * examples/pbsmon.py. Patched so it also works for torque 2.5
86   Author: Stijn De Weirdt stijn dot deweirdt at ugent dot be
87   Applied by: Bas van der Vlies
88
89 * setup.py.in changed determine which torque version is installed
90   so we also support snapshots.
91   Author: Mark Roberts ( mark at gingergeeks dot co dot uk)
92   Applied by: Bas van der Vlies
93
94 * PBSQuery:
95   - get_nodes function for jobs now works for old and new datastructure and
96     gives correct results if unique=True
97     Reported by + idea for patch: Shunjie Lau (eijnuhs add gmail dot com)
98     Author: Bas van der Vlies
99
100 * Added AdvanceParser module. new_rack_pbsmon.py and sara_nodes.py make use of
101   this module
102   Author: Dennis Stam (Dennis dot Stam at sara dot nl)
103
104 * examples/new_rack_pbsmon.py:
105     - Better specification for node expression
106     - added a new option -l/--list to list the state of a nodes and which job
107       is running on it, eg:
108             new_rack_pbsmon.py - l gb-r[1-3]n[3-6] gb-r4[n1-20]
109
110   Authors: Dennis Stam & Bas van der Vlies
111   Applied by: Bas van der Vlies
112
113 * examples/sara_nodes.py:
114   An utility to set/clear notes for multiple nodes, see ..
115   Author: Dennis Stam (Dennis dot Stam at sara dot nl)
116
117 * setup.py.in:
118   Added support for runtime-libraries when torque is installed in a non-default
119   location, closes #22
120   Author: Mattias (slabanja at chalmers dot se)
121   Applied by: Bas van der Vlies
122
123 * Changed build system for debian
124   Authors: Dennis Stam & Bas van der Vlies
125
126=========== 4.1.2
127 * examples/new_rack_pbsmon.py now uses the new PBSQuery data
128   structure.
129   Author: Bas van der Vlies
130
131 * PBSQuery:
132     - fixed and error in get_jobs function for nodes. Did not
133       support new data structure and uniq did not work.
134       Authors: Shunjie Lau <eijnuhs at gmail dot com> and Bas van der Vlies
135
136=========== 4.1.0
137 * Delete function pbs.version() replaced by
138    * pbs.version
139        This is a string variable
140    * pbs.version_info
141        Is a tuple variable. So we can use it in scripts
142
143   This analogue with sys module, closes #20
144
145   Requested by: Dennis Stam ( Dennis dot Stam add sara dot nl )
146   Author: Bas van der Vlies
147
148 * More robust make/configure/setup.py
149   Reported by:
150        - Saidi, Yacine <saidi add linmpi dot mpg dot de>
151        - Shunjie Lau <eijnuhs add gmail dot com>
152   Author: Bas van der Vlies
153
154
155 * Only use the pbs_wrap.c and pbs.py 2.4 version for torque version 2.4.7 and higher.
156   Reported by: Saidi, Yacine <saidi add linmpi dot mpg dot de>
157   Author: Bas van der Vlies
158   
159=========== 4.0.2
160  * Quick fix so we do not get any undefined symbols. Forgot to
161    add the torque library when we compile the package.
162    Reported by: Shunjie Lau ( eijnuhs add gmail dot com)
163    Author: Bas van der Vlies
164
165=========== 4.0.1
166  * Fixed a bug in setting the right library path  for compiling.
167    Reported by: Chad Vizino ( vizino add psc dot edu )
168    Author: Bas van der Vlies
169   
170=========== 4.0.0
171  * pbs.py added some new functions for 2.4 version:
172    - pbs_fbserver(void);
173    - pbs_get_server_list(void);
174    - pbs_sigjobasync(int connect, char *job_id, char *signal, char *extend);
175    - pbs_alterjob_async(int connect, char *job_id, struct attrl *ATTRL, char *extend);
176    - pbs_checkpointjob(int connect, char *job_id, char *extend);
177    - log_ext(int,char *,char *,int);
178    - log_init(char *, char *);
179    - log_remove_old(char *,unsigned long);
180
181    Note: version 2.1.X is still supported
182
183  * autotools environment
184    - Added support for finding pbs-config (borrowed for mpiexec source)
185    - Detecting which version of torque is installed
186    - only support installations that have pbs-config installed.
187    - Removed old code for installations that do not a have pbs-config
188      installed.
189
190  * PBSQuery.py
191    - There was a bug in the new data interface (Resource_List for jobs)
192      Reported & Patch by: Mark Roberts ( mark at gingergeeks dot co dot uk)
193      Applied by with some minor changes: Bas van der Vlies
194
195    - The new data structure is the default. You can switch back to the old
196      one with:
197            p = PBSQuery()
198            p.old_data_structure()
199
200      Author: Bas van der Vlies
201
202   * examples/ha_server.py
203     An example how to use the High Availabilty torque functions
204     Author: Bas van der Vlies & Mark Roberts ( mark at gingergeeks dot co dot uk )
205
206=========== 3.6.0
207  * pbs, new generated pbs_wrap.c, pbs.py fixes a bug in pbs_runjob()
208    Reported by: Dennis Stam
209    Fixed by: Bas van der Vlies
210
211=========== 3.5.0
212  * PBSQuery
213        The class functions of node, job and queue support old and new data
214        structure.
215
216        Changed the behaviour of the new data stucture, We can use it as
217        dictionary and as class attribute, this is equivalent, eg:
218          - print node['np'] and print node.np
219
220        for a node we parse the 'status' line and split on '=' char, You now can
221        use these statements, eg
222          - print node.status.arch     (node['status'].arch or node['status']['arch'])
223          - print node.status.nsession
224
225        for a job we parse the 'Variable_List' line and split on '=' char, You now can
226        use the statements, eg:
227          - print job.Variable_List.PBS_O_WORKDIR
228          - print job.Variable_List.PBS_O_HOME
229
230        For more info see examples/new_interface.py
231
232    Author: Bas van der Vlies
233
234 * new_rack_pbsmon.py
235        Rewrite to new data structure and automatically determine how many nodes
236        and racks cluster has and skip printing of empty racks (default), use -w/--wide
237        for old behaviour.
238
239   Author: Bas van der Vlies
240
241
242=========== 3.2.0
243        - PBSQuery:
244          New data structure. In the old structure it is a dictionary
245          with a value and the value is a string. This is changed
246          that dictionary values are now of type list or dictionary depends
247          on the value of the keyword, eg for a node:
248            - np = 2:
249              * node['np'] = [ '2' ] 
250
251            - properties = cores2, mem4gb, parallel
252              * node['properties'] = [ 'cores2', 'mem4gb', 'parallel' ]
253
254            - status = arch=x86_64,sessions=22599,,size=70627864kb, ...
255              * node['status']['arch'] = [ 'x86_64' ]
256              * node['status']['sessions'] = [ '222599' ]
257              * ...
258
259         The data structure is activated by the function:
260          * new_data_structure()
261
262        In a future release it will be come the default.
263        example:
264                p = PBSQuery()
265                p.new_data_structure()
266
267                nodes = p.getnodes()
268                print nodes.np, nodes['np']
269
270        Author: Bas van der Vlies
271
272        - PBSQuery
273          For old and new data structure we now can use another syntax:
274           * node['np'] and node.np are equivalent
275
276          This new syntax works for all keywords.
277
278        Author: Bas van der Vlies
279
280        - Added iter object for job, node, queue and server objects, eg:
281          node = p.getnode('gb-r1n1')
282          print node.name
283          for attrib in node:
284                print '%\t%s = %s' %(attrib, node[attrib])
285        Author: Bas van der Vlies
286
287        - fixed an error in getnode, getqueue and getjob, return empty
288          dictionary if not found
289        Author: Bas van der Vlies
290       
291        - New build system for rpm packages, make -f Makefile.rpm
292          Author: Michel Jouvin <jouvin add lal dot in2p3 dot fr>
293
294        Applied: Bas van der Vlies
295
296=========== 3.0.1
297        - PBSQuery
298                * Removed a testing code line
299                Reported By: sam @ vpac dot org
300                Fixed by: Bas van der Vlies
301
302=========== 3.0.0
303        - PBSQuery
304                * Cleaned up some old lines of code which didn't do anything.
305                * Changed the returnvalue of getqueue, getnode and getjob. Now
306                  they return an instance of the object, instead of a dict.
307                  This is API change so we bumped the version to 3.0.0
308                * Now the getnode-function accepts the short name as well as
309                  the long name as an argument.
310
311                Author: Sil Westerveld <Sil.Westerveld@sara.nl>
312
313        - pbs.py, _pbs.so, PBSQuery.py
314                * Fixed a memory leak when used in daemon mode
315                Author: Bas van der Vlies <basv@sara.nl>
316
317=========== 2.9.8
318        - pbs_python.spec file patch added libdir and python defines so it
319          will build for RHEL5.1/CentOS 5.1
320          by        : Michael Sternberg <sternberg add anl dot gov>
321          applied by: Bas van der Vlies
322
323        - Updated the header files for pbs_python. So it can use some new
324          defines variables/defines, eg:
325                * ATTR_NODE_status
326                * ATTR_NODE_note
327                * ...
328          Added by: Dennis Stam <dennis.stam@sara.nl>, Bas van der Vlies
329
330        - Two new functions to log.h:
331                log_roll(int size);
332                log_size(void);
333          Added by: Bas van der Vlies
334
335        - We now have proper destructors (free memory) in SWIG for attrl and
336          attropl attributes.
337          Added by: Bas van der Vlies
338
339        - PBSQuery
340                * Added getnodes_with_property() function. This will get all nodes with
341                  a certain property, eg:
342                   - get all nodes with property 'cores8' and return all attributes:
343                      getnodes_with_property('cores8')
344               
345                   - get all nodes with property 'cores2' and only return the
346                     attributes 'state' and 'properties':
347                      attrl = [ 'state', 'properties' ]
348                      getnodes_with_property('cores2', attrl)
349               
350                     Only for torque version > 2.1.0
351             
352                Added by: Bas van der Vlies
353
354                * Added to class node the function
355                 - get_jobs(self, unique=None)
356                   Returns a list of the currently running job-id('s) on the node.
357                Added by: Sil Westerveld <Sil.Westerveld@sara.nl>
358
359                * Added get_server_name() function.
360                  This will return the PBS-server's name.
361                Added by: Sil Westerveld <Sil.Westerveld@sara.nl>
362
363                * Added to class _PBSobject the function
364                  - uniq(self, list)
365                    Filters out unique items of a list.
366                Added by: Sil Westerveld <Sil.Westerveld@sara.nl>
367
368                * Added to class job the function
369                  - get_nodes(self, unique=None)
370                    Returns a list of the nodes which run the job.
371                Added by: Sil Westerveld <Sil.Westerveld@sara.nl>
372
373=========== Version 2.9.4
374        -  Function pbs.pbs_geterrmsg does not exits any more is
375           replaced by pbs.error():
376                errno, text = pbs.error()
377                print errno, text
378           Reported by: pk at q-leap dot com
379           Fixed by   : Bas van der Vlies
380
381        - PBSQuery.py
382           * Added get_version() for server class, returns the version of
383             the batch server
384             Added by    : Bas van der Vlies
385
386           * Added get_nodes() for job class, returns a list of nodes on
387             which the job is run
388             Added by    : Bas van der Vlies
389
390           * PBSQuery.py rewrite make use of UserDict module.
391             Suggested by: Ramon Bastiaans
392             Fixed by    : Bas van der Vlies
393
394=========== Version 2.9.2
395        - The function pbs_rescquery() did not work. Fixed it
396        - The function avail(), did not work fixed it
397        - The functions pbs_rescreserve and pbs_rescrelease did not work
398          fixed it
399
400        - Fixed a bug in new_rack_pbsmon.py. Counting of serial
401          nodes was wrong and state down has topmost priority
402
403=========== Version 2.9.0
404        - Added support for new torque build system 2.1.0
405        - Added new_rack_pbsmon.py
406          Authors: Walter de Jong & Bas van der Vlies
407
408=========== Version 2.8.2
409
410Configure has a new option:
411        --with-pbsdir=PATH
412
413If set it will use PATH specified as argument to
414find the pbs/torque libraries. If unset it will use
415some useful defaults to find the libaries.
416Suggested by: Troy Baer
417
418PBSQuery.py:
419        - If we can not make a connection with the server
420          then raise an exception and let the programmer
421          decides what to do.
422
423pbs_python.spec:
424        - It can now handle mode bits
425          Author: Martin Pels from SARA
426       
427=========== Version 2.8.0
428
429setup.py:
430        - Added /usr/lib to PBS_LIB_DIR
431
432pbs.py:
433        - Added new function "pbs.error()". This function checks if
434          an error has occured with a pbs function, eg:
435            task_id = pbs.pbs_submit(c, attrl, "A.tsk", 'NULL', 'NULL')
436            error_number, err_txt = pbs.error()
437            if error_number:
438                print error_number, err_txt
439
440          prints the following message if script A.tsk does not exists:
441            15042 (qsub) cannot access script file
442
443pbs_python.spec:
444        - to make rpm packages from the source package, initial version
445        Author: Martin Pels from SARA
446
447README:
448        - Now requires python 2.1 and higher
449        - Explain how to build DEBIAN package
450
451=========== Version 2.7.10
452
453PBSQuery.py:
454        Fixed an error. You always got all resources from eg nodes
455        even when you supplied an attribute list where you only
456        requested the 'state' of the node.
457
458        Updated the inline python documentation
459
460=========== Version 2.7.9
461
462Fixed an error the previous version was in DEBUG mode. So
463you get an lot undesired output on the screen.
464
465debian package now also generated a package with the write
466version info.
467
468=========== Version 2.7.8
469
470Setup.py:
471        Can also handle ROCKS installations + the version
472        info is now the same as pbs.py
473        Thanks to: Roy Dragseth
474
475=========== Version 2.7.7
476
477PBSQuery module:
478  It can now be used in daemon programs. The pbs_server closes
479  the connection after an certain amount of time. This is fixed,
480  before every query there is an new connection and if the query
481  is finished the connection is closed.
482 
483=========== Version 2.7.6
484
485Now setup.py can also handle old Oscar Installations.
486Thanks To: Robin Humble
487
488Fixed an print statement fix in new_interface.py:
489Thanks To: Robin Humble
490
491Made some improvements for PBSQuery module:
492        - Updated the documentation
493        - Added the has_key() function to all
494          PBSObjects. So that the behaviour is similar to
495          an dictionary
496          Suggested By: Ramon Bastiaans
497       
498=========== Version 2.7.5
499 Added PBSQuery module.  This module requires pbs.py and it a simple module
500 for querying the pbs server. The documentation is in the module. Use for
501 example ipython to read it.
502
503 Written By: Roy Dragseth
504             Bas van der Vlies
505
506 eg: ( see also examples/new_interface.py)
507
508 from PBSQuery import PBSQuery
509 p = PBSQuery()
510
511 jobs = p.getjobs()
512 for name, job in jobs.items()
513        print job
514
515=========== Version 2.7.4
516 New versions fixes an bug in function pbs.pbs_statnode().
517 Reported by: Keith Poirier
518
519 Configure support:
520 Contributed By: Yaroslav Halchenko
521
522 Debian package support:
523 Contributed By: Yaroslav Halchenko
524
525 Added Support for Debuging the interface, You must edit pbs_wrap.c
526 and search for SARA_DEBUG:
527        #define SARA_DEBUG 1
528 Implemented by: Ramon Bastiaans
529
530 examples/pbsmon.py:
531        - Fixed an error when regex fails to determine node number,
532          no status was displayed
533        - It will now display and 'j' if the node is free for the batch
534          system and a job runs on the node (SMP-systems).
535 Changed by: Bas van der Vlies
536
537 examples/rack_pbsmon.py:
538        - An pbsmon that display node info per rack
539        - Edit the rack_pbsmon.py to adjust the values. It requires
540          that hostname contains rack and node id's,
541          eg: gb-r<number>n<number>
542 Contributed by: Walter de Jong
543
544=========== Version 2.7.3
545 The name SPBS is changed to the new name TORQUE (Tera-scale Open-source
546 Resource and QUEing manager).
547
548 The interface support OPENPBS and TORQUE.
549
550=========== Version 2.7.2
551 Some minor changes to include files. Now the Scalable PBS keywords
552 are also supported ( server attribute names: node_ping_rate and
553 node_check_rate). This does not interfere with the openpbs software.
554
555 pbsmon.py could not handle an one node cluster. Fixed it.
556
557=========== Version 2.7
558 Forgot to wrap the pbs_statfree() function. So we could not free
559 allocated memory from functions that return 'struct batch_status *'
560 like pbs_statjob().
561
562 pbsmon.py can now handle 2 and 3 digit hostnames. Patch supplied by
563 Daniel Olson
564
565=========== Version 2.6.1
566 Setup.py now checks if all openpbs libraries are installed to compile the
567 package.
568
569=========== Version 2.6
570
571 Fixed a bug in the pbs python module. Forgot to map the pbs_statjob()
572 function. This bug prevented to pass 'struct attrl' variables to this
573 function. Thanks to Evelyn Shiu for reporting this bug.
574
575
576=========== Version 2.5
577 Added the OpenPbs logging functions (log.h and liblog.a). There
578 is an example in the examples directory: logpbs.py
579
580 Fixed a bug in examples/resmom_info.py used the default port for
581 pbs_resmon if getservbyname fails.
582
583 Fixed a bug in resmom code. If the user has no permissions to query
584 the pbs_mom daemon a empty string is returned. Now we check for
585 the empty string. So we do not get a python exception.
586
587 Fixed a bug in setup.py forgot the exit statement when we did not
588 find the pbs libraries.
589
590=========== Version 2.3
591 Bugfix in the pbsnodes-a.py. Forgot to import the sys module.
592
593 Removed some obsolete code.
594
595 As for now we make use of the distutils module. So the package
596 is automatically compiled and installed in proper directories, usage:
597    python setup.py install
598 This requires that the package 'distutils' must be installed. As for
599 version 1.6 and higher it is in the distribution.
600
601 Added a function that prints the pbs python interface version. So we
602 can check if we have the right interface version:
603 import pbs
604 print pbs.version()
605 
606
607=========== Version 2.2
608
609In this release we can query the pbs_mom daemon with the aid of
610resource management functions:
611   pbs.openrm()
612   pbs.closerm()
613   pbs.downrm()
614   pbs.configrm()
615   pbs.addreq()
616   pbs.allreq()
617   pbs.flushreq()
618   pbs.activereq()
619   pbs.fullresp()
620   pbs.getreq()
621
622There is also a python function:
623   pbs.get_mom_values(id, [list]):
624     id   - connection id of pbs.openrm(node-name, port-number)
625     list - Is a optional paramter. A list of resource keywords
626
627   If 'list' is not supplied then this function will get the values
628   for the standard resource keywords, eg uname, loadave, .. + 'arch'
629   depended keywords. The arch feature is only implemented for linux,
630   but it can easily be ported to other oses.
631
632   if 'list' is supplied then we only fetch values for the keywords
633   that are in the list.
634
635   The function returns a dictonary. The keys are the resource keywords.
636
637   See 'resmom_info.py' for a example of this new feature.
638
639=========== Version 2.0
640
641The previous version was a simple interface above the PBS C API LIB.  This
642interface has gone a major changes. The most noticeable change is  that
643the functions accept/return Python lists instead of C-lists.  So you can now
644use standard Python syntax for manipulating Python lists. If you are familar
645with the Python syntax then you will appreciate this new interface.
646
647NOTE:
648  This interface is NOT compatible with the old one
649
650Here are some guidelines to convert your code to the new one:
651
652The constructors for the struct has gone a major change:
653  old code:
654    temp = pbs.new_attrl();
655    attrl_p = pbs.attrlPtr(temp)
656    attrl_p.name = 'state'
657
658  new_code:
659     attr_l = pbs.new_attrl(1)   // Creates a list of attrl structs length 1
660     attr_l[0].name = 'state'
661
662
663The pbs_stat functions returns Python lists instead of C-lists. There is
664NO next field anymore:
665  old_code:
666    temp = pbs.new_attrl();
667    attrl_p = pbs.attrlPtr(temp)
668    attrl_p.this = 'NULL'
669
670    nodes = pbs.pbs_statnode(con, "", attrl_p, "NULL")
671
672    while nodes.this != 'NULL':
673      print nodes.name
674
675      node_attrl = nodes.attribs
676      while node_attrl.this != 'NULL':
677        print '\t', node_attrl.name, '=', node_attrl.value
678        node_attrl = node_attrl.next
679
680    nodes = nodes.next
681
682
683  new_code:
684    nodes = pbs.pbs_statnode(con, "", "NULL", "NULL")
685    for node in nodes:
686      print node.name
687      for attrib in node.attribs:
688        print '\t', attrib.name, '=', attrib.value
689
690Another advantage is you can use the print statement to show the
691connect of attrl and attropl classes:
692   attr_l = pbs.new_attrl(2)
693   attr_l[0].name = 'bas'
694   attr_l[0].value = 'van der Vlies'
695
696   print  attr_l[0]
697   >> (bas,,van der Vlies)
698   
699I hope these examples illustrate the changes. If you specify a wrong type
700for a function then function wil raise a Python exception.
Note: See TracBrowser for help on using the repository browser.