Index: /tags/0.92/LICENSE
===================================================================
--- /tags/0.92/LICENSE	(revision 19)
+++ /tags/0.92/LICENSE	(revision 19)
@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
Index: /tags/0.92/Make.inc
===================================================================
--- /tags/0.92/Make.inc	(revision 19)
+++ /tags/0.92/Make.inc	(revision 19)
@@ -0,0 +1,47 @@
+# to be included in the Makefile
+# Version of the software
+STOPOS_VERSION=0.92
+
+# installation directory of the server on local system:
+CGI_INSTALL_DIR=$(HOME)/myroot/usr/local/www/cgi-bin
+
+# installation directory of the server on the remote system
+REMOTE_CGI_INSTALL_DIR=stopos.osd.surfsara.nl:/usr/local/stopos/cgi-bin
+
+# location of the client programs:
+CLIENT_INSTALL_DIR=$(HOME)/myroot/sara/sw/stopos-$(STOPOS_VERSION)
+
+# remote installation directory of the client programs:
+REMOTE_CLIENT_INSTALL_DIR=stopos.osd.surfsara.nl:/usr/local/stopos/bin
+
+# location of the modulefile
+MODULE_INSTALL_DIR=$(HOME)/mymodulefiles/64
+
+# name of the c++ compiler, and it's flags
+CXX=g++
+CXXFLAGS=-Wall -O2
+
+# Choose to generate one ore more of the following:
+#MAKE_FLAT=yes
+MAKE_GDBM=yes
+#MAKE_MYSQL=yes
+#MAKE_FILES=yes
+
+# Choose here the default the client will choose:
+#  this can be overwritten in run time by the environment
+#   variable STOPOS_PROT
+DB_DEFAULT=DEFAULT_GDBM
+#DB_DEFAULT=DEFAULT_FILES
+#DB_DEFAULT=DEFAULT_MYSQL
+#DB_DEFAULT=DEFAULT_FLAT
+
+# Define the default url of the server,
+# can be overwritten in runtime by environment variable
+# STOPOS_SERVER_URL
+#SERVER_URL=http://localhost:5050/cgi-bin/stoposserver/
+SERVER_URL=http://stopos.osd.surfsara.nl/cgi-bin/stoposserver
+
+# Define directory to store the non-mysql databases 
+# default is /tmp
+DB_DIR=/data/stopos
+
Index: /tags/0.92/Makefile
===================================================================
--- /tags/0.92/Makefile	(revision 19)
+++ /tags/0.92/Makefile	(revision 19)
@@ -0,0 +1,142 @@
+
+###
+#   Copyright 2013 Willem Vermin, SURFsara
+#
+#   Licensed under the Apache License, Version 2.0 (the "License");
+#   you may not use this file except in compliance with the License.
+#   You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
+###
+include Make.inc
+CGI_PROGRAMS       = stoposserver
+CLIENT_PROGRAMS    = stoposclient stoposzztostr
+CLIENT_PROGRAMS1   = stoposdump
+OTHER_PROGRAMS     = testpool 
+
+STOPOSSERVER_OBJS  := stoposserver.o stopos_pool.o stopos_key.o
+STOPOSSERVER_LIBS  := libwutils.a
+
+TESTPOOL_OBJS      := test_pool.o stopos_pool.o stopos_key.o clocks.o
+TESTPOOL_LIBS      := libwutils.a
+
+ifdef MAKE_GDBM
+  COMPILE_FLAGS     += -DMAKE_GDBM
+  STOPOSSERVER_OBJS +=  gdbm_pool.o
+  STOPOSSERVER_LIBS += -lgdbm
+  TESTPOOL_OBJS     +=  gdbm_pool.o
+  TESTPOOL_LIBS     += -lgdbm
+endif
+ifdef MAKE_FLAT
+  COMPILE_FLAGS     += -DMAKE_FLAT
+  STOPOSSERVER_OBJS +=  flatfile_pool.o
+  TESTPOOL_OBJS     +=  flatfile_pool.o
+endif
+ifdef MAKE_FILES
+  COMPILE_FLAGS     += -DMAKE_FILES
+  STOPOSSERVER_OBJS +=  files_pool.o
+  TESTPOOL_OBJS     +=  files_pool.o
+endif
+ifdef MAKE_MYSQL
+  COMPILE_FLAGS     += -DMAKE_MYSQL
+  STOPOSSERVER_OBJS +=  mysql_pool.o
+  STOPOSSERVER_LIBS += -lmysqlcppconn
+  TESTPOOL_OBJS     +=  mysql_pool.o
+  TESTPOOL_LIBS     += -lmysqlcppconn
+endif
+
+COMPILE_FLAGS += -D$(DB_DEFAULT)
+
+COMPILE_FLAGS += -DSERVER_URL=\"$(SERVER_URL)\"
+
+COMPILE_FLAGS += -DDB_DIR=\"$(DB_DIR)\"
+
+STOPOSCLIENT_OBJS  = stoposclient.o wrequest.o shellenv.o
+STOPOSCLIENT_LIBS  = -lcurl libwutils.a
+
+
+STOPOSZZTOSTR_OBJS = stoposzztostr.o
+STOPOSZZTOSTR_LIBS = libwutils.a -lz
+
+PROGRAMS           = $(CGI_PROGRAMS) $(CLIENT_PROGRAMS) $(OTHER_PROGRAMS)
+
+CGI_LIBS           = 
+CGI_INC            = 
+
+all: $(PROGRAMS)
+
+CGI_OBJS           = $(STOPOSSERVER_OBJS)
+CGI_SRCS           = $(CGI_OBJS:.o=.cpp)
+
+CLIENT_OBJS        = $(STOPOSCLIENT_OBJS)
+CLIENT_SRCS        = $(CLIENT_OBJS:.o=.cpp)
+
+OTHER_OBJS         = $(TESTPOOL_OBJS)
+OTHER_SRCS         = $(OTHER_OBJS:.o=.cpp)
+
+SRCS = $(sort $(CGI_SRCS) $(OTHER_SRCS) $(CLIENT_SRCS) wutils.cpp)
+
+stoposserver: $(STOPOSSERVER_OBJS) libwutils.a
+	$(CXX) -o $@ $(CXXFLAGS) $(STOPOSSERVER_OBJS) $(STOPOSSERVER_LIBS)
+
+stoposclient: $(STOPOSCLIENT_OBJS) libwutils.a
+	$(CXX) -o $@ $(CXXFLAGS) $(STOPOSCLIENT_OBJS) $(STOPOSCLIENT_LIBS)
+
+testpool:  $(TESTPOOL_OBJS) libwutils.a
+	$(CXX) -o $@ $(CXXFLAGS) $(TESTPOOL_OBJS) $(TESTPOOL_LIBS)
+
+stoposzztostr: $(STOPOSZZTOSTR_OBJS) libwutils.a
+	$(CXX) -o $@ $(CXXFLAGS) $(STOPOSZZTOSTR_OBJS) $(STOPOSZZTOSTR_LIBS)
+
+libwutils.a: wutils.o
+	ar -r $@ wutils.o
+
+stopos.1: stopos.man
+	sed 's"DEFAULTSERVERURL"$(SERVER_URL)"' stopos.man > $@
+
+%.o: 	%.cpp
+	$(CXX) $(CXXFLAGS) $(COMPILE_FLAGS) $(CGI_INC) -c -o $@ $<
+
+include .depend
+
+.depend dep:
+	$(CXX) -M $(SRCS) $(CGI_INC) > .depend
+
+clean:
+	rm -f *.o libwutils.a $(PROGRAMS) stopos.1 .depend
+
+realclean: clean
+
+install: install_cgi install_client install_module install_remote
+	
+install_cgi:	$(CGI_PROGRAMS)
+	mkdir -p $(CGI_INSTALL_DIR)
+	install $(CGI_PROGRAMS) $(CGI_INSTALL_DIR)
+
+install_client: $(CLIENT_PROGRAMS) $(CLIENT_PROGRAMS1) stopos.1
+	mkdir -p $(CLIENT_INSTALL_DIR)/bin
+	install $(CLIENT_PROGRAMS) $(CLIENT_PROGRAMS1) $(CLIENT_INSTALL_DIR)/bin
+	mkdir -p $(CLIENT_INSTALL_DIR)/man/man1
+	install stopos.1 -m 644 $(CLIENT_INSTALL_DIR)/man/man1
+	install sara-get-mem-size $(CLIENT_INSTALL_DIR)/bin
+	install sara-get-num-cores $(CLIENT_INSTALL_DIR)/bin
+	install sara-get-mem-size.1 -m 644 $(CLIENT_INSTALL_DIR)/man/man1
+	install sara-get-num-cores.1 -m 644 $(CLIENT_INSTALL_DIR)/man/man1
+
+install_module:
+	mkdir -p $(MODULE_INSTALL_DIR)/mystopos
+	install modulefile -m 644 $(MODULE_INSTALL_DIR)/mystopos/$(STOPOS_VERSION)
+
+install_remote: $(CLIENT_PROGRAMS) $(CGI_PROGRAMS)
+	chmod 755 $(CLIENT_PROGRAMS) $(CGI_PROGRAMS)
+	@echo "Now, manually transport"
+	@echo "$(CLIENT_PROGRAMS) to $(REMOTE_CLIENT_INSTALL_DIR)"
+	@echo "and"
+	@echo "$(CGI_PROGRAMS) to $(REMOTE_CGI_INSTALL_DIR)"
+
Index: /tags/0.92/aantekeningen.txt
===================================================================
--- /tags/0.92/aantekeningen.txt	(revision 19)
+++ /tags/0.92/aantekeningen.txt	(revision 19)
@@ -0,0 +1,30 @@
+key heeft nu twee velden: value en slot
+
+key::get_value()
+key::get_slot()
+
+key.str() levert "value.slot"
+
+stopos_key::stopos_key(const std::string &s)
+ {
+   split (v,s,'.');
+   value = v[0];
+   if (v.size() >1)
+   slot = v[1];
+ }
+
+ add_line roept get_freeslot() aan om een vrije locatie te bepalen
+  put_record moet vervolgens deze locatie gaan gebruiken.
+  pas list van vrije slots aan
+
+In flatfile:
+  put_record slaat record op op locatie slot, samen met
+  keyvalue. locatie uit key.get_slot
+
+  get_record haalt record van locatie key.getslot()
+
+  remove_record: kijkt of record begint met keyvalue.
+    ja: als al gemerkt als removed, error
+        remove record, pas link van vrije slots aan
+          en mark record als removed
+    nee: error
Index: /tags/0.92/cleanit
===================================================================
--- /tags/0.92/cleanit	(revision 19)
+++ /tags/0.92/cleanit	(revision 19)
@@ -0,0 +1,2 @@
+#!/bin/bash
+make clean
Index: /tags/0.92/clocks.cpp
===================================================================
--- /tags/0.92/clocks.cpp	(revision 19)
+++ /tags/0.92/clocks.cpp	(revision 19)
@@ -0,0 +1,71 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#include <sys/times.h>
+#include <sys/time.h>
+#include <time.h>
+#include <stdlib.h>
+#include "clocks.h"
+#include <unistd.h>
+double cputime() /* aix, xlf */
+{ struct tms b;
+  clock_t r;
+  times( &b);
+  r = b.tms_utime + b.tms_stime;
+  return ( (double) r/(double) sysconf(_SC_CLK_TCK));
+}
+double CPUTIME() /* cray  */
+{ 
+  return ( cputime());
+}
+double cputime_() /* g77, gcc */
+{ 
+  return ( cputime());
+}
+
+double wallclock() 
+{ 
+	struct timeval toot;
+	double r;
+
+	gettimeofday(&toot,0);
+	r=toot.tv_sec+0.000001*(double)toot.tv_usec;
+	return(r);
+}
+double WALLCLOCK()
+{
+  return (wallclock());
+}
+double wallclock_()
+{
+  return wallclock();
+}
+		
+void fortransleep(int *i)
+  {
+    sleep(*i);
+  }
+ 
+void FORTRANSLEEP(int *i)
+  {
+    sleep(*i);
+  }
+
+void fortransleep_(int *i)
+  {
+    sleep(*i);
+  }
+
Index: /tags/0.92/clocks.h
===================================================================
--- /tags/0.92/clocks.h	(revision 19)
+++ /tags/0.92/clocks.h	(revision 19)
@@ -0,0 +1,24 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#ifdef __cplusplus
+extern "C"  {
+#endif
+double wallclock(void);
+double cputime(void);
+#ifdef __cplusplus
+}
+#endif
Index: /tags/0.92/credits
===================================================================
--- /tags/0.92/credits	(revision 19)
+++ /tags/0.92/credits	(revision 19)
@@ -0,0 +1,17 @@
+
+CGIC License Terms
+------------------
+
+Basic License
+-------------
+
+CGIC, copyright 1996-2011 by Thomas Boutell and Boutell.Com, Inc. Permission 
+is granted to use CGIC in any application, commercial or noncommercial, at no 
+cost. HOWEVER, this copyright paragraph must appear on a "credits" page 
+accessible in the public online and offline documentation of the program. 
+Modified versions of the CGIC library should not be distributed without 
+the attachment of a clear statement regarding the author of the 
+modifications, and this notice may in no case be removed. 
+Modifications may also be submitted to the author for inclusion 
+in the main CGIC distribution.                                 
+
Index: /tags/0.92/doc/description.lyx
===================================================================
--- /tags/0.92/doc/description.lyx	(revision 19)
+++ /tags/0.92/doc/description.lyx	(revision 19)
@@ -0,0 +1,5316 @@
+#LyX 2.0 created this file. For more info see http://www.lyx.org/
+\lyxformat 413
+\begin_document
+\begin_header
+\textclass article
+\use_default_options true
+\maintain_unincluded_children false
+\language english
+\language_package default
+\inputencoding auto
+\fontencoding global
+\font_roman default
+\font_sans default
+\font_typewriter default
+\font_default_family default
+\use_non_tex_fonts false
+\font_sc false
+\font_osf false
+\font_sf_scale 100
+\font_tt_scale 100
+
+\graphics default
+\default_output_format default
+\output_sync 0
+\bibtex_command default
+\index_command default
+\paperfontsize 10
+\spacing single
+\use_hyperref false
+\papersize a4paper
+\use_geometry true
+\use_amsmath 1
+\use_esint 1
+\use_mhchem 1
+\use_mathdots 1
+\cite_engine basic
+\use_bibtopic false
+\use_indices false
+\paperorientation portrait
+\suppress_date false
+\use_refstyle 1
+\index Index
+\shortcut idx
+\color #008000
+\end_index
+\secnumdepth 3
+\tocdepth 3
+\paragraph_separation indent
+\paragraph_indentation 0bp
+\quotes_language english
+\papercolumns 1
+\papersides 1
+\paperpagestyle default
+\tracking_changes false
+\output_changes false
+\html_math_output 0
+\html_css_as_file 0
+\html_be_strict false
+\end_header
+
+\begin_body
+
+\begin_layout Title
+STOPOS
+\end_layout
+
+\begin_layout Author
+Willem Vermin, willem.vermin@surfsara.nl
+\begin_inset Newline newline
+\end_inset
+
+Bas van der Vlies, bas.vandervlies@surfsara.nl
+\end_layout
+
+\begin_layout Date
+2013-02-01
+\end_layout
+
+\begin_layout Abstract
+In this article we describe the internals and externals of the program stopos.
+ Stopos is primarily designed to facilitate the running of many tasks on
+ a cluster computer.
+ It is based on managing lines in a pool.
+ These lines will in general be used as command lines to define tasks.
+ Stopos is heavily inspired by ToPoS (https://grid.sara.nl/wiki/index.php/Using_the
+_Grid/ToPoS) and experiences with disparm (https://www.surfsara.nl/systems/lisa/so
+ftware/disparm).
+ This project was carried out as a 
+\begin_inset Quotes eld
+\end_inset
+
+seed-fund project
+\begin_inset Quotes erd
+\end_inset
+
+ at SURFsara.
+\end_layout
+
+\begin_layout Subsection*
+Introduction
+\end_layout
+
+\begin_layout Standard
+A typical use of a cluster computer, such as SURFsara's Lisa system 
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+https://www.surfsara.nl/systems/lisa
+\end_layout
+
+\end_inset
+
+, is running many tasks each task with different parameters.
+ A typical example is a parameter scan.
+ A typical job allocates a node, and runs a number of tasks in parallel.
+ The number of tasks should in general be equal to the amount of cores and/or
+ the amount of memory.
+ Sometimes, thousands of these jobs are necessary, and normally, the job
+ scripts are generated by a script.
+ In a homogeneous cluster (all nodes have the same amount of cores and the
+ same amount of memory) this is clumsy but doable, but when the cluster
+ is heterogeneous (such as the Lisa cluster), it becomes very hard to use
+ all types of nodes available, and adapt the number of processes to the
+ node where the job runs on.
+ To deal with this problems, stopos was created: a server that manages a
+ pool of lines.
+ These lines are defined by the user, and the server serves the lines to
+ jobs, such that each line is produced once.
+ Earlier successful solutions are ToPoS
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+https://grid.sara.nl/wiki/index.php/Using_the_Grid/ToPoS
+\end_layout
+
+\end_inset
+
+ and disparm
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+https://www.surfsara.nl/systems/lisa/software/disparm
+\end_layout
+
+\end_inset
+
+.
+ ToPoS is designed to be used on a server that can be accessed from the
+ Grid, while disparm is based on files in the user's home directory.
+ We think that ToPoS is not optimal for the Lisa system because of lack
+ of cluster-based authentication and the universal access.
+ The disparm approach is in these respects better, but in practice it appeared
+ that the performance under heavy usage the performance was too low, and
+ the reliability was not 100%: sometimes a line was produced more than once,
+ or lines were not produced at all.
+ Also, it is not possible to add lines dynamically: all lines have to be
+ inserted at the beginning.
+ We think that the approach followed in stopos is a well balanced solution,
+ particularly fit to be used in a cluster environment.
+\end_layout
+
+\begin_layout Subsection*
+Usage
+\end_layout
+
+\begin_layout Standard
+A typical usage of stopos is:
+\end_layout
+
+\begin_layout Itemize
+fill the pool with lines
+\end_layout
+
+\begin_layout Itemize
+start jobs, each job gets one or more lines from the pool, and remove the
+ lines that are not needed any more
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+Normally, a line is produced only once.
+ However, the user can request that a line is produced and used again, for
+ example in the case the previous task handling this line ran into a time
+ limit.
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Itemize
+maybe: add more lines to the pool
+\end_layout
+
+\begin_layout Standard
+Stopos is implemented as a shell function in the bash shell, or as an alias
+ in the (t)csh shell.
+ Stopos calls the program stoposclient and transforms it's output to environment
+ variables, the most important being STOPOS_RC and STOPOS_VALUE.
+\end_layout
+
+\begin_layout Standard
+The commands to create the pool:
+\begin_inset Newline newline
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+\begin_inset space ~
+\end_inset
+
+module load stopos 
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+# activate the stopos software
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+\begin_inset space ~
+\end_inset
+
+stopos create
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+# create the pool
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+\begin_inset space ~
+\end_inset
+
+stopos add parmfile
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+# add the lines in the file parmfile to the pool
+\begin_inset Newline newline
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+In a job script, one could use code like this:
+\begin_inset Newline newline
+\end_inset
+
+
+\begin_inset Newline newline
+\end_inset
+
+
+\family typewriter
+
+\begin_inset space ~
+\end_inset
+
+module load stopos
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+# activate the stopos software
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+\begin_inset space ~
+\end_inset
+
+stopos next 
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+# get a line from the pool
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+\begin_inset space ~
+\end_inset
+
+if [ $STOPOS_RC = OK] ; then # check the return code
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+run_my_prog $STOPOS_VALUE
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+# run the program using the line as parameter
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+\begin_inset space ~
+\end_inset
+
+fi
+\begin_inset Newline newline
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+A more elaborate job script, using all cores available on a node is available
+ in the Appendix, together with the man page of stopos.
+\end_layout
+
+\begin_layout Subsection*
+Tasks and communication
+\end_layout
+
+\begin_layout Standard
+Stopos is a shell function or an alias that calls stoposclient.
+ Stoposclient communicates with stoposserver via an http server (apache
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+http://httpd.apache.org/
+\end_layout
+
+\end_inset
+
+, thttpd
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+http://opensource.dyc.edu/sthttpd
+\end_layout
+
+\end_inset
+
+, ...).
+ The server sends commands to the server, and transforms the answers to
+ a format that are sourced by stopos to define environment variables.
+ Care is taken that communications between the client and the server are
+ short in length: at most a few kilobytes are sent between client and server.
+ 
+\end_layout
+
+\begin_layout Subsubsection*
+Tasks of the client
+\end_layout
+
+\begin_layout Standard
+The client parses it's command line, sends the appropriate command to the
+ server and receives the response.
+ This response is put in a form that can be 'eval-ed' so that environment
+ variables are set.
+\begin_inset Newline newline
+\end_inset
+
+
+\begin_inset Newline newline
+\end_inset
+
+The client recognizes the following commands on its command line:
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+-h,--h
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+: prints a short usage message
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+-v,--v
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+: prints the version number
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+create [ -p,--pool POOL ]
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ create a pool
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+status [ -p,--pool POOL ]
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ get status from a pool
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+purge [ -p,--pool POOL]
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ remove a pool
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+pools
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ what are my pools
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+add [ -p,--pool POOL] [ FILENAME ]:
+\family default
+ add a file to the pool
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+next [-m,--multi]
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ get next line from the pool
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+remove [ KEY ]
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ remove line
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+dump [ -p,--p POOL ]
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ dump a line from the pool
+\end_layout
+
+\begin_layout Standard
+For example:
+\begin_inset Newline newline
+\end_inset
+
+
+\begin_inset Newline newline
+\end_inset
+
+
+\family typewriter
+stopos create -p mypool1
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+stopos add -p mypool1 parmfile
+\begin_inset Newline newline
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+The -p (or --pool) flag defines the name of the pool file to be used (
+\begin_inset Quotes eld
+\end_inset
+
+parmfile
+\begin_inset Quotes erd
+\end_inset
+
+ in this example).
+ If no -p or --pool flag is present, the value of the pool is taken from
+ the environment variable STOPOS_POOL.
+ If that is not available, the value 
+\begin_inset Quotes eld
+\end_inset
+
+pool
+\begin_inset Quotes erd
+\end_inset
+
+ is used.
+\end_layout
+
+\begin_layout Standard
+The -m (or --multi) flag signifies that lines that are already produces,
+ may be produced again.
+\end_layout
+
+\begin_layout Standard
+The default for FILENAME is standard input.
+\end_layout
+
+\begin_layout Standard
+If no value is given for KEY with the 
+\begin_inset Quotes eld
+\end_inset
+
+remove
+\begin_inset Quotes erd
+\end_inset
+
+ command, the value of the environment variable STOPOS_KEY is used.
+ This value is set by the 
+\begin_inset Quotes eld
+\end_inset
+
+next
+\begin_inset Quotes erd
+\end_inset
+
+ command.
+\begin_inset Newline newline
+\end_inset
+
+
+\begin_inset Newline newline
+\end_inset
+
+The following environment variables are set by the command 
+\begin_inset Quotes eld
+\end_inset
+
+stopos
+\begin_inset Quotes erd
+\end_inset
+
+:
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+STOPOS_RC 
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ return code of the command.
+ The value 
+\begin_inset Quotes eld
+\end_inset
+
+OK
+\begin_inset Quotes erd
+\end_inset
+
+ means that things went well
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+STOPOS_VALUE
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+ :
+\family default
+ result of the 
+\begin_inset Quotes eld
+\end_inset
+
+next
+\begin_inset Quotes erd
+\end_inset
+
+ and 
+\begin_inset Quotes eld
+\end_inset
+
+pools
+\begin_inset Quotes erd
+\end_inset
+
+ command
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+STOPOS_COMMITTED :
+\family default
+ number of times the 
+\begin_inset Quotes eld
+\end_inset
+
+next
+\begin_inset Quotes erd
+\end_inset
+
+ line was produced
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+STOPOS_KEY 
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ set by the 
+\begin_inset Quotes eld
+\end_inset
+
+next
+\begin_inset Quotes erd
+\end_inset
+
+ command.
+ Used by the 
+\begin_inset Quotes eld
+\end_inset
+
+remove
+\begin_inset Quotes erd
+\end_inset
+
+ command
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+STOPOS_COUNT 
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ set by 
+\begin_inset Quotes eld
+\end_inset
+
+status
+\begin_inset Quotes erd
+\end_inset
+
+: number of lines added to the pool
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+STOPOS_PRESENT
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ set by 
+\begin_inset Quotes eld
+\end_inset
+
+status
+\begin_inset Quotes erd
+\end_inset
+
+: number of lines in the pool
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+STOPOS_PRESENT0
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ set by 
+\begin_inset Quotes eld
+\end_inset
+
+status
+\begin_inset Quotes erd
+\end_inset
+
+: number of lines never been committed
+\end_layout
+
+\begin_layout Standard
+Another task is creating a unique id for the user.
+ This id, combined with the login name of the user and the name of the pool,
+ will be used by the server to identify the pool to be used.
+ The id is stored in the HOME directory of the user, in the file $HOME/.stopos/rc.
+ If this file is missing, or inappropriate, the client will create this
+ file.
+\begin_inset Newline newline
+\end_inset
+
+
+\begin_inset Newline newline
+\end_inset
+
+All commands result in a single communication with the server, except the
+ 
+\begin_inset Quotes eld
+\end_inset
+
+add
+\begin_inset Quotes erd
+\end_inset
+
+ command: the client will send a number of 
+\begin_inset Quotes eld
+\end_inset
+
+add
+\begin_inset Quotes erd
+\end_inset
+
+ commands to the server, one for each line of the file.
+\begin_inset Newline newline
+\end_inset
+
+
+\begin_inset Newline newline
+\end_inset
+
+The 
+\begin_inset Quotes eld
+\end_inset
+
+dump
+\begin_inset Quotes erd
+\end_inset
+
+ command is kind of special.
+ It is created for the case a user wants to inspect the actual contents
+ of the pool.
+ The 
+\begin_inset Quotes eld
+\end_inset
+
+dump
+\begin_inset Quotes erd
+\end_inset
+
+ command will produce one line per call, and does not interfere with the
+ other commands.
+ (More specific: the 
+\begin_inset Quotes eld
+\end_inset
+
+next
+\begin_inset Quotes erd
+\end_inset
+
+ command and the 
+\begin_inset Quotes eld
+\end_inset
+
+dump
+\begin_inset Quotes erd
+\end_inset
+
+ command use different pointers) When the last line has been produced, the
+ next call will result in an error code (STOPOS_RC != OK).
+ Then the next call will start again at the first available record in the
+ pool.
+ 
+\end_layout
+
+\begin_layout Subsubsection*
+Messages from client to server
+\end_layout
+
+\begin_layout Standard
+A message from client to server consists of the following fields:
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+stopos
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ the string 
+\begin_inset Quotes eld
+\end_inset
+
+stopos
+\begin_inset Quotes erd
+\end_inset
+
+
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+prot
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ the protocol to be used (i.e.
+ which kind of database)
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+id
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ the identification of the user
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+pool
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ the name of the pool to be used
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+command
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ the command for the server
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+multi
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ the value of 
+\begin_inset Quotes eld
+\end_inset
+
+multi
+\begin_inset Quotes erd
+\end_inset
+
+: can a line be produced more than once?
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+value
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ a parameter
+\end_layout
+
+\begin_layout Standard
+All fields are coded in hexadecimal strings, embraced by a fixed header
+ and trailer.
+ This is to ensure that no problematic things (spaces, tabs, slashes, empty
+ strings etc.) have to be transmitted.
+ The fields are separated with the character 
+\begin_inset Quotes eld
+\end_inset
+
+-
+\begin_inset Quotes erd
+\end_inset
+
+.
+ Example: to send a message to the server located at 
+\begin_inset Quotes eld
+\end_inset
+
+http://www.stoposserver.surfsara.nl
+\begin_inset Quotes erd
+\end_inset
+
+ with:
+\end_layout
+
+\begin_layout Itemize
+protocol = 
+\begin_inset Quotes eld
+\end_inset
+
+gdbm
+\begin_inset Quotes erd
+\end_inset
+
+
+\end_layout
+
+\begin_layout Itemize
+user identification = 
+\begin_inset Quotes eld
+\end_inset
+
+willem.qR7w3XQp
+\begin_inset Quotes erd
+\end_inset
+
+
+\end_layout
+
+\begin_layout Itemize
+poolname = 
+\begin_inset Quotes eld
+\end_inset
+
+mypool
+\begin_inset Quotes erd
+\end_inset
+
+
+\end_layout
+
+\begin_layout Itemize
+command = 
+\begin_inset Quotes eld
+\end_inset
+
+add
+\begin_inset Quotes erd
+\end_inset
+
+
+\end_layout
+
+\begin_layout Itemize
+multi = 
+\begin_inset Quotes eld
+\end_inset
+
+no
+\begin_inset Quotes erd
+\end_inset
+
+
+\end_layout
+
+\begin_layout Itemize
+parameter = 
+\begin_inset Quotes eld
+\end_inset
+
+1 2 3
+\begin_inset Quotes erd
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+The client would create the following url:
+\begin_inset Newline newline
+\end_inset
+
+
+\begin_inset Newline newline
+\end_inset
+
+
+\family typewriter
+http://www.stoposserver.surfsara.nl?ZZZ73746f706f73YY
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset Quotes eld
+\end_inset
+
+stopos
+\begin_inset Quotes erd
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+-ZZZ6764626dYY
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+ 
+\begin_inset Quotes eld
+\end_inset
+
+gdbm
+\begin_inset Quotes erd
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+-ZZZ77696c6c656d2e7152377733585170YY
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+ 
+\begin_inset Quotes eld
+\end_inset
+
+willem.
+\family default
+qR7w3XQp
+\begin_inset Quotes erd
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+-ZZZ6d79706f6f6cYY
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+ 
+\begin_inset Quotes eld
+\end_inset
+
+mypool
+\begin_inset Quotes erd
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+-ZZZ616464YY
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+ 
+\begin_inset Quotes eld
+\end_inset
+
+add
+\begin_inset Quotes erd
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+-ZZZ6e6fYY
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+ 
+\begin_inset Quotes eld
+\end_inset
+
+no
+\begin_inset Quotes erd
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+-ZZZ3120322033YY
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+ 
+\begin_inset Quotes eld
+\end_inset
+
+1 2 3
+\begin_inset Quotes erd
+\end_inset
+
+
+\begin_inset Newline newline
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+The header is 
+\begin_inset Quotes eld
+\end_inset
+
+ZZZ
+\begin_inset Quotes erd
+\end_inset
+
+, the trailer is 
+\begin_inset Quotes eld
+\end_inset
+
+YY
+\begin_inset Quotes erd
+\end_inset
+
+.
+\begin_inset Newline newline
+\end_inset
+
+
+\begin_inset Newline newline
+\end_inset
+
+
+\end_layout
+
+\begin_layout Subsubsection*
+Tasks for the server
+\end_layout
+
+\begin_layout Standard
+The server parses the message from the client, executes the command and
+ sends the results back.
+ The commands the the server recognizes are:
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+create
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ create a pool
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+status
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ send the status of the pool
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+purge
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ remove the pool
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+add
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ add a line, taken from the 
+\begin_inset Quotes eld
+\end_inset
+
+value
+\begin_inset Quotes erd
+\end_inset
+
+ field of the message
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+next
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ send the next line together with the 
+\begin_inset Quotes eld
+\end_inset
+
+key
+\begin_inset Quotes erd
+\end_inset
+
+ and 
+\begin_inset Quotes eld
+\end_inset
+
+committed
+\begin_inset Quotes erd
+\end_inset
+
+.
+ Take care of 
+\begin_inset Quotes eld
+\end_inset
+
+multi
+\begin_inset Quotes erd
+\end_inset
+
+
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+remove
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ remove a line with key given in the 
+\begin_inset Quotes eld
+\end_inset
+
+value
+\begin_inset Quotes erd
+\end_inset
+
+ field
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+dump
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ return next line to be dumped, together with 
+\begin_inset Quotes eld
+\end_inset
+
+key
+\begin_inset Quotes erd
+\end_inset
+
+ and 
+\begin_inset Quotes eld
+\end_inset
+
+committed
+\begin_inset Quotes erd
+\end_inset
+
+
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+pools
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ return the names of the available pools for the user
+\end_layout
+
+\begin_layout Standard
+Not accidentally: these commands have the same names as the commands for
+ the client, but the context is different.
+\end_layout
+
+\begin_layout Subsubsection*
+Communication from server to client
+\end_layout
+
+\begin_layout Standard
+The server parses the message from the client and sends the result back,
+ using 
+\begin_inset Quotes eld
+\end_inset
+
+Content-type: text/plain
+\begin_inset Quotes erd
+\end_inset
+
+.
+ The server puts the following string in the output:
+\begin_inset Newline newline
+\end_inset
+
+
+\begin_inset Newline newline
+\end_inset
+
+
+\family typewriter
+STOPOS:
+\family default
+
+\begin_inset Newline newline
+\end_inset
+
+
+\begin_inset Newline newline
+\end_inset
+
+This string is immediately followed by the desired output, which consists
+ of the following fields:
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+return message
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ 
+\begin_inset Quotes eld
+\end_inset
+
+OK
+\begin_inset Quotes erd
+\end_inset
+
+ or some error message
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+key
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ key of the record just retrieved or empty
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+committed
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ number of times the record has been committed or empty
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+count
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ number of records added so far or empty
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+present
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ number of records present or empty
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+present0
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ number of records never been committed or empty
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+value
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+:
+\family default
+ return value or empty 
+\end_layout
+
+\begin_layout Itemize
+
+\family typewriter
+new line character
+\end_layout
+
+\begin_layout Standard
+The client will search for the string 
+\begin_inset Quotes eld
+\end_inset
+
+STOPOS:
+\begin_inset Quotes erd
+\end_inset
+
+ and use the string immediately following as the result.
+ The developer can, if she wishes, print extra information and have a look
+ at the total output of the server, as long as she does not use the string
+ 
+\begin_inset Quotes eld
+\end_inset
+
+STOPOS:
+\begin_inset Quotes erd
+\end_inset
+
+.
+ The fields are encoded, just like is done with the message from the client
+ to the server, only the separator here is 
+\begin_inset Quotes eld
+\end_inset
+
++
+\begin_inset Quotes erd
+\end_inset
+
+ for no special reason.
+ 
+\end_layout
+
+\begin_layout Subsection*
+Internals of the client
+\end_layout
+
+\begin_layout Standard
+The client parses the command line, using getopt(3).
+ It performs some basic checks on the parameters given, and sends an appropriate
+ command to the server, using libcurl(3).
+ The output from the server is parsed and transformed into statements that
+ can be executed in the shell.
+ These commands set environment variables, with names starting with 
+\begin_inset Quotes eld
+\end_inset
+
+STOPOS_
+\begin_inset Quotes erd
+\end_inset
+
+.
+\end_layout
+
+\begin_layout Subsection*
+Internals of the server
+\end_layout
+
+\begin_layout Standard
+The server parses the command sent from the client and an http-server (apache,
+ thttpd), executes it and sends the results back to the client.
+ Most of the interesting work is done in the class 
+\begin_inset Quotes eld
+\end_inset
+
+stopos_pool
+\begin_inset Quotes erd
+\end_inset
+
+.
+ This class, from which a database-specific class has to be derived, maintains
+ a double linked list of lines.
+ The addressing of the lines is based on keys that are computed from the
+ counter that is incremented each time a line is added.
+ The derived class has to deal with some basic I/O operations: creating,
+ opening, closing and removing of the database; reading, writing and removing
+ records.
+ In this code, four database implementations are provided: 
+\end_layout
+
+\begin_layout Itemize
+based on gdbm 
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+http://www.gnu.org.ua/software/gdbm/
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Itemize
+based om mysql 
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+http://www.mysql.com/
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Itemize
+based on a flat ascii file
+\end_layout
+
+\begin_layout Itemize
+based on files in a folder, each file containing one line
+\end_layout
+
+\begin_layout Standard
+The client determines which implementation the server has to use.
+ Storage of a record is based on the key, but, if desirable, the derived
+ class can provide a suitable slot for storing the record.
+ This feature is used in the 
+\begin_inset Quotes eld
+\end_inset
+
+flat file
+\begin_inset Quotes erd
+\end_inset
+
+ solution, in order to be able to re-use space occupied by removed records.
+ 
+\end_layout
+
+\begin_layout Subsubsection*
+Storage of lines
+\end_layout
+
+\begin_layout Standard
+Lines are stored in records, which also contain extra fields.
+ The records are hexadecimal coded, just like the communication from client
+ to server.
+ The separation character is here '/'.
+ Records are stored using unique keys, generated by the class 
+\begin_inset Quotes eld
+\end_inset
+
+stopos_pool
+\begin_inset Quotes erd
+\end_inset
+
+.
+ The derived class has no knowledge about the meaning of the records: it's
+ only task is to store, retrieve and delete them.
+\end_layout
+
+\begin_layout Subsection*
+Performance
+\end_layout
+
+\begin_layout Standard
+A program 
+\begin_inset Quotes eld
+\end_inset
+
+test_pool.cpp
+\begin_inset Quotes erd
+\end_inset
+
+ has been provided which runs all four implementations, without an http
+ server, but directly calling the functions from the class 
+\begin_inset Quotes eld
+\end_inset
+
+stopos_pool
+\begin_inset Quotes erd
+\end_inset
+
+ and reports how much time a implementation takes.
+ Furthermore, it does some consistency checks.
+ The pool consists of a few hundreds of lines.
+ The results are
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+The program was run on a Linux workstation, equipped with a 
+\begin_inset Quotes eld
+\end_inset
+
+normal
+\begin_inset Quotes erd
+\end_inset
+
+ (no SSD) disk 
+\end_layout
+
+\end_inset
+
+:
+\begin_inset Newline newline
+\end_inset
+
+
+\begin_inset Newline newline
+\end_inset
+
+
+\family typewriter
+database
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+time (sec)
+\begin_inset Newline newline
+\end_inset
+
+gdbm
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+3.31
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+flatfile 
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+0.19 
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+files 
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+0.73 
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+mysql 
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+4.7
+\family default
+1
+\begin_inset Newline newline
+\end_inset
+
+ 
+\begin_inset Newline newline
+\end_inset
+
+The winner is clearly the 
+\begin_inset Quotes eld
+\end_inset
+
+flat file
+\begin_inset Quotes erd
+\end_inset
+
+ implementation.
+ Disadvantage of this solution (and also of the 
+\begin_inset Quotes eld
+\end_inset
+
+files
+\begin_inset Quotes erd
+\end_inset
+
+ and 
+\begin_inset Quotes eld
+\end_inset
+
+mysql
+\begin_inset Quotes erd
+\end_inset
+
+ solution) is the amount of space that is used for the database: each line
+ takes 4 Kbyte.
+ The 
+\begin_inset Quotes eld
+\end_inset
+
+gdbm
+\begin_inset Quotes erd
+\end_inset
+
+ solution does not have this problem.
+ 
+\begin_inset Newline newline
+\end_inset
+
+Normally, the stopos software will be used using an http server, so it is
+ more relevant to have look at the timings in this case.
+ As an example, that man page of bash (5459 lines) was put in the pool,
+ and all lines were retrieved and removed:
+\begin_inset Newline newline
+\end_inset
+
+
+\begin_inset Newline newline
+\end_inset
+
+
+\family typewriter
+database 
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+time(sec)
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+gdbm 
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+187
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+flatfile 
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+182
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+files 
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+190
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+mysql 
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\begin_inset space ~
+\end_inset
+
+
+\family default
+309
+\begin_inset Newline newline
+\end_inset
+
+
+\begin_inset Newline newline
+\end_inset
+
+These timings show, that for performance reasons, there is not much difference
+ between 
+\begin_inset Quotes eld
+\end_inset
+
+gdbm
+\begin_inset Quotes erd
+\end_inset
+
+, 
+\begin_inset Quotes eld
+\end_inset
+
+flatfile
+\begin_inset Quotes erd
+\end_inset
+
+ and 
+\begin_inset Quotes eld
+\end_inset
+
+files
+\begin_inset Quotes erd
+\end_inset
+
+.
+ We choose the 
+\begin_inset Quotes eld
+\end_inset
+
+gdbm
+\begin_inset Quotes erd
+\end_inset
+
+ implementation, because of it's economy with disk space used.
+\end_layout
+
+\begin_layout Subsection*
+Installation
+\end_layout
+
+\begin_layout Standard
+The source tree contains a file 
+\begin_inset Quotes eld
+\end_inset
+
+Make.inc
+\begin_inset Quotes erd
+\end_inset
+
+, defining variables that determine the version of the program, the installation
+ directories and which database versions should be produced.
+ This file will be included in the Makefile.
+ A trivial script 
+\begin_inset Quotes eld
+\end_inset
+
+makeit
+\begin_inset Quotes erd
+\end_inset
+
+ sets the umask and calls make(1) to create and install the executables.
+ The script 
+\begin_inset Quotes eld
+\end_inset
+
+clean
+\begin_inset Quotes erd
+\end_inset
+
+ executes a 
+\begin_inset Quotes eld
+\end_inset
+
+make clean
+\begin_inset Quotes erd
+\end_inset
+
+ to get rid of compilation products.
+\end_layout
+
+\begin_layout Subsubsection*
+Extra tools
+\end_layout
+
+\begin_layout Standard
+The source tree also contains the scripts 
+\begin_inset Quotes eld
+\end_inset
+
+sara-get-num-cores
+\begin_inset Quotes erd
+\end_inset
+
+ and 
+\begin_inset Quotes eld
+\end_inset
+
+sara-get-mem-size
+\begin_inset Quotes erd
+\end_inset
+
+, along with their man pages.
+ Sara-get-num-cores prints the number of available cores, while sara-get-mem-siz
+e prints the amount of memory.
+ Both numbers can be used to determine how many processes should run in
+ parallel on a given node.
+ These scripts and man pages will be installed also by calling 
+\begin_inset Quotes eld
+\end_inset
+
+./makeit
+\begin_inset Quotes erd
+\end_inset
+
+.
+\end_layout
+
+\begin_layout Subsection*
+Example of a job using stopos
+\end_layout
+
+\begin_layout Standard
+\begin_inset listings
+inline false
+status open
+
+\begin_layout Plain Layout
+
+1  #PBS -lnodes=1 -lwalltime=3:00:00  
+\end_layout
+
+\begin_layout Plain Layout
+
+2  module load stopos  
+\end_layout
+
+\begin_layout Plain Layout
+
+3  ncores=`sara-get-num-cores`  
+\end_layout
+
+\begin_layout Plain Layout
+
+4  for ((i=1; i<=ncores; i++)) ; do  
+\end_layout
+
+\begin_layout Plain Layout
+
+5  (  
+\end_layout
+
+\begin_layout Plain Layout
+
+6    for ((j=1; j<=10; j++)) ; do  
+\end_layout
+
+\begin_layout Plain Layout
+
+7      stopos next -p pool1  
+\end_layout
+
+\begin_layout Plain Layout
+
+8        if [ "$STOPOS_RC" != "OK" ]; then  
+\end_layout
+
+\begin_layout Plain Layout
+
+9        break 
+\end_layout
+
+\begin_layout Plain Layout
+
+10      fi 
+\end_layout
+
+\begin_layout Plain Layout
+
+11      eval "scanner $STOPOS_VALUE" 
+\end_layout
+
+\begin_layout Plain Layout
+
+12      stopos remove -p pool1 
+\end_layout
+
+\begin_layout Plain Layout
+
+13    done 
+\end_layout
+
+\begin_layout Plain Layout
+
+14  ) & 
+\end_layout
+
+\begin_layout Plain Layout
+
+15  done 
+\end_layout
+
+\begin_layout Plain Layout
+
+16  wait
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Itemize
+1 Note: we do not specify how many cores a node should have.
+\end_layout
+
+\begin_layout Itemize
+2 Load the stopos module.
+\end_layout
+
+\begin_layout Itemize
+3 Determine the number of cores available on this node: Module stopos contains
+ 'sara-get-num-cores': this command prints the number of cores available
+ on the node.
+ The command 'sara-get-mem-size' prints the available amount of memory in
+ Mbytes.
+ We do not use it here, but this number could be interesting when running
+ memory intensive programs.
+\end_layout
+
+\begin_layout Itemize
+4 Loop over the number of cores.
+ The content of the loop is placed in the background (see 14)
+\end_layout
+
+\begin_layout Itemize
+6 Loop over the number of the desired number of repetitions.
+ 
+\end_layout
+
+\begin_layout Itemize
+7 Here we get the value of the next line, delivered in the environment variable
+ STOPOS_VALUE.
+ The environment variable STOPOS_RC has the value "OK" if everything went
+ ok.
+ STOPOS_KEY contains a value that can be used to remove the line from the
+ pool.
+ We use the pool 'pool1', created before.
+\end_layout
+
+\begin_layout Itemize
+8 ..
+ 10 Break out the loop if stopos fails (usually because there are no lines
+ left).
+ 
+\end_layout
+
+\begin_layout Itemize
+11 Use the parameters in $STOPOS_VALUE.
+ The 'eval' and quotes ensure that spaces and quotes in the parameters are
+ correctly interpreted.
+ 
+\end_layout
+
+\begin_layout Itemize
+12 Remove the line we just processed from the pool.
+ 
+\end_layout
+
+\begin_layout Itemize
+14 Notice the &: this takes care that ncores instantiations of scanner will
+ be started in parallel.
+ 
+\end_layout
+
+\begin_layout Itemize
+16 The 'wait' ensures that the job script will only finish after all background
+ processes are ended.
+\end_layout
+
+\begin_layout Itemize
+The job can be submitted a number of times with one 'qsub' command, for
+ example: 
+\family typewriter
+qsub -t 1-100 job
+\end_layout
+
+\begin_layout Subsection*
+Man page of stopos
+\end_layout
+
+\begin_layout Standard
+
+\family typewriter
+\begin_inset listings
+inline false
+status open
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+STOPOS(1)                                                            STOPOS(1)
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+NAME
+\end_layout
+
+\begin_layout Plain Layout
+
+       stopos - alias to call the program stoposclient and eval it's output
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+SYNOPSIS
+\end_layout
+
+\begin_layout Plain Layout
+
+       stopos  -h,--help
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       stopos  -v,--version
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       stopos  create    [ -p,--pool POOL ]
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       stopos  status    [ -p,--pool POOL]
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       stopos  purge     [ -p,--pool POOL ]
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       stopos  pools
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       stopos  add       [ -p,--pool POOL ] [ FILENAME ]
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       stopos  next      [ -p,--pool POOL ] [ -m,--multi ]
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       stopos  remove    [ -p,--pool POOL] [ KEY  ]
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       stopos  dump      [ -p,--pool POOL]
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       The -q,--quiet flag supresses most output.
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+DESCRIPTION
+\end_layout
+
+\begin_layout Plain Layout
+
+       Stopos  is an alias using the program stoposclient, an utility to
+ store
+\end_layout
+
+\begin_layout Plain Layout
+
+       and retrieve text lines in a pool.
+  In general, the text lines are used
+\end_layout
+
+\begin_layout Plain Layout
+
+       as command parameters, see Examples.
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+COMMANDS
+\end_layout
+
+\begin_layout Plain Layout
+
+       -h,--help
+\end_layout
+
+\begin_layout Plain Layout
+
+           Prints usage information.
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       -v,--version
+\end_layout
+
+\begin_layout Plain Layout
+
+           Prints version.
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       create [ -p,--pool POOL ] [ FILENAME ]
+\end_layout
+
+\begin_layout Plain Layout
+
+           Creates new pool.
+  A pool with the same name will be removed first.
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       status
+\end_layout
+
+\begin_layout Plain Layout
+
+           Prints to standard error       environment variable
+\end_layout
+
+\begin_layout Plain Layout
+
+           ---------------------------------------------------
+\end_layout
+
+\begin_layout Plain Layout
+
+           total number of lines added      STOPOS_COUNT
+\end_layout
+
+\begin_layout Plain Layout
+
+           number of lines present          STOPOS_PRESENT
+\end_layout
+
+\begin_layout Plain Layout
+
+           number of lines never committed  STOPOS_PRESENT0
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       purge [-p,--pool POOL ]
+\end_layout
+
+\begin_layout Plain Layout
+
+           Removes the pool.
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       pools [-p,--pool POOL ]
+\end_layout
+
+\begin_layout Plain Layout
+
+           Lists on standard error the pools available.
+ The environment
+\end_layout
+
+\begin_layout Plain Layout
+
+           variable STOPOS_VALUE is set accordingly.
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       add [ -p,--pool POOL ] [ FILENAME ]
+\end_layout
+
+\begin_layout Plain Layout
+
+           Adds lines from FILENAME, default from stdin.
+  The  environ‐
+\end_layout
+
+\begin_layout Plain Layout
+
+           ment variables STOPOS_RC and STOPS_KEY are set.
+\end_layout
+
+\begin_layout Plain Layout
+
+           NOTE: when stopos is reading from a pipe as in:
+\end_layout
+
+\begin_layout Plain Layout
+
+              cat parmfile | stopos add
+\end_layout
+
+\begin_layout Plain Layout
+
+           no environment variables are set.
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       next [ -p,--pool POOL ] [ -m,--multi ]
+\end_layout
+
+\begin_layout Plain Layout
+
+           Gets the next line from the pool.
+  The environment variables
+\end_layout
+
+\begin_layout Plain Layout
+
+           STOPOS_VALUE, STOPOS_COMMITTED and STOPOS_KEY are  set,  see
+\end_layout
+
+\begin_layout Plain Layout
+
+           ENVIRONMENT.
+   By default, the same line in the pool will be
+\end_layout
+
+\begin_layout Plain Layout
+
+           produced only once.
+ When all lines  are  commited,  STOPS_RC
+\end_layout
+
+\begin_layout Plain Layout
+
+           will  not  be  equal to OK.
+ Using the --multi flag, the same
+\end_layout
+
+\begin_layout Plain Layout
+
+           line can be produced more than  once,  if  necessary  stopos
+\end_layout
+
+\begin_layout Plain Layout
+
+           will  wrap  around.
+   This  can  be  useful for dealing with
+\end_layout
+
+\begin_layout Plain Layout
+
+           crashed jobs.
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       remove [-p,--pool POOL ] [ KEY ]
+\end_layout
+
+\begin_layout Plain Layout
+
+           Removes the line with key KEY as ready.
+  If not specified on
+\end_layout
+
+\begin_layout Plain Layout
+
+           the  commandline,  the  value  of  environment variable STO‐
+\end_layout
+
+\begin_layout Plain Layout
+
+           POS_KEY is used.
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       dump [-p,--pool POOL ]
+\end_layout
+
+\begin_layout Plain Layout
+
+           Reads the next available line, and puts it, preceded by it's
+\end_layout
+
+\begin_layout Plain Layout
+
+           key  and  number of commitments in environment variable STO‐
+\end_layout
+
+\begin_layout Plain Layout
+
+           POS_VALUE.
+ The first call will produce the first line.
+  When
+\end_layout
+
+\begin_layout Plain Layout
+
+           all  lines have been delivered, STOPOS_RC gets a value other
+\end_layout
+
+\begin_layout Plain Layout
+
+           than OK.
+ A subsequent call will  start  the  dump  with  the
+\end_layout
+
+\begin_layout Plain Layout
+
+           first line again.
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+OPTIONS
+\end_layout
+
+\begin_layout Plain Layout
+
+       -p,--pool POOL
+\end_layout
+
+\begin_layout Plain Layout
+
+           POOL is a unique name of the pool.
+  Default: pool.
+  The name
+\end_layout
+
+\begin_layout Plain Layout
+
+           of the pool can also be set using the  environment  variable
+\end_layout
+
+\begin_layout Plain Layout
+
+           STOPOS_POOL.
+ The command line flag has precedence.
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+ENVIRONMENT
+\end_layout
+
+\begin_layout Plain Layout
+
+       Stopos sets the following environment variables:
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       STOPOS_RC if the value is OK, than no errors were found
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       STOPOS_KEY  contains  the keyvalue of the line produced with the
+\end_layout
+
+\begin_layout Plain Layout
+
+       'next' command
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       STOPOS_COMMITTED contains the number of times the line has  been
+\end_layout
+
+\begin_layout Plain Layout
+
+       committed
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       STOPOS_VALUE contains the line produced by the 'next' command or
+\end_layout
+
+\begin_layout Plain Layout
+
+       the result of the 'pools' command
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       STOPOS_COUNT STOPOS_PRESENT  STOPOS_PRESENT0  see  the  'status'
+\end_layout
+
+\begin_layout Plain Layout
+
+       command above
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       Stopos uses the following environment variables:
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       STOPOS_POOL see the description of the --pool flag under OPTIONS
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       STOPOS_KEY see the 'remove' flag above
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+       STOPOS_SERVER_URL  the  url  stopos  uses  to access the server,
+\end_layout
+
+\begin_layout Plain Layout
+
+       default http://stopos.osd.surfsara.nl/cgi-bin/stoposserver
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+EXAMPLES
+\end_layout
+
+\begin_layout Plain Layout
+
+       Create a pool with the first 10 lines from the man page of sed:
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+            man sed | head > parmfile
+\end_layout
+
+\begin_layout Plain Layout
+
+            stopos create
+\end_layout
+
+\begin_layout Plain Layout
+
+            stopos add parmfile
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+           Get a line:
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+            stopos next
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+           The environment variable STOPOS_VALUE contains  now  one  of
+\end_layout
+
+\begin_layout Plain Layout
+
+           the first ten lines of the man page of sed.
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+           You can use this in a command like this:
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+            eval "myparser $STOPOS_VALUE"
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+           The line can be removed by:
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+            stopos remove
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+           Finally, the pool can be completely removed by:
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+            stopos purge
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+FILES
+\end_layout
+
+\begin_layout Plain Layout
+
+       $HOME/.stopos/id
+\end_layout
+
+\begin_layout Plain Layout
+
+       This  file  is  set  by  stopos  to  store  an unique id, which,
+\end_layout
+
+\begin_layout Plain Layout
+
+       together with the login name, will be  used  to  identify  which
+\end_layout
+
+\begin_layout Plain Layout
+
+       pools are yours.
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+SEE ALSO
+\end_layout
+
+\begin_layout Plain Layout
+
+       sara-get-mem-size(1),   sara-get-num-cores(1),  stoposclient(1),
+\end_layout
+
+\begin_layout Plain Layout
+
+       stoposdump(1)
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+AUTHORS
+\end_layout
+
+\begin_layout Plain Layout
+
+       Willem Vermin
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+BUGS
+\end_layout
+
+\begin_layout Plain Layout
+
+       No bugs known yet.
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\begin_layout Plain Layout
+
+                                                                     STOPOS(1)
+\end_layout
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\end_body
+\end_document
Index: /tags/0.92/files_pool.cpp
===================================================================
--- /tags/0.92/files_pool.cpp	(revision 19)
+++ /tags/0.92/files_pool.cpp	(revision 19)
@@ -0,0 +1,229 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#include "stopos_pool.h"
+#include "files_pool.h"
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <stdlib.h>
+#include "wutils.h"
+#include <string.h>
+#include <strings.h>
+#include <stdio.h>
+#include <algorithm>
+#include <fstream>
+#include "stopos.h"
+#include <string>
+#include <list>
+#include <unistd.h>
+#include <errno.h>
+
+// constructor
+files_pool::files_pool()
+{
+  this->whoami = "files";
+}
+
+// own version of set_db_name.
+void files_pool::set_db_name(std::string filename)
+{
+  this->sdb_name = filename;
+  this->db_name  = strdup(sdb_name.c_str());
+}
+
+void files_pool::create_fname(std::string &fname,const std::string &k)
+{
+  fname = this->sdb_name + "/" + k;
+}
+
+int files_pool::get_record(std::string &r, const std::string &k, const std::string &l)
+{
+  std::string fname;
+  this->create_fname(fname,k);
+  std::ifstream f;
+  f.open(fname.c_str(),std::fstream::in);
+  if (f.fail())
+    return FETCHERROR;
+
+  char b[this->reclen];
+
+  f.getline(b,this->reclen);
+  if (f.bad())
+    return FETCHERROR;
+
+  r = b;
+
+  f.close();
+
+  return 0;
+}
+
+int files_pool::put_record(const std::string &b, 
+                           const std::string &k, 
+			   const std::string &l)
+{
+  bool recordtoolong = 0;
+
+  std::string buf = b;
+
+  if (buf.size() >= reclen)
+    recordtoolong=1;
+
+  std::string fname;
+
+  this->create_fname(fname,k);
+
+  std::ofstream f;
+  f.open(fname.c_str(),std::fstream::out);
+  if (f.fail())
+    return STOREERROR;
+
+  buf.resize(reclen);
+
+  f.write(buf.c_str(),this->reclen);
+  if (f.bad())
+    return STOREERROR;
+
+  f.close();
+
+  if(recordtoolong)
+    return RECORDTOOLONG;
+
+  return 0;
+}
+
+int files_pool::remove_record(const std::string &k,const std::string &l)
+{
+  int rc;
+  std::string fname;
+  this->create_fname(fname,k);
+  rc = unlink(fname.c_str());
+  if (rc == -1)
+    return this->REMOVEERROR;
+
+  return 0;
+}
+
+int files_pool::create_db(const std::string &dbname)
+{
+  int rc;
+  rc = this->purge_db();
+  if (rc != 0)
+    return rc;
+
+  rc = mkdir_p(dbname,0755);
+  if (rc != 0)
+    return rc;
+
+  return 0;
+}
+
+int files_pool::purge_db(void)
+{
+  int rc;
+  if (this->db_name == 0)
+    return NOFILENAME;
+
+  std::list <std::string> v;
+
+  rc = get_dir_list(v,this->sdb_name);
+  for (std::list<std::string>::iterator it=v.begin(); it != v.end(); ++it)
+  {
+    // . and .. will be automaticalle skipped by unlink
+    std::string f = this->sdb_name + '/' + (*it);
+    unlink(f.c_str());
+  }
+
+  rc = rmdir(this->sdb_name.c_str());
+  if (rc != 0)
+    if (errno == ENOENT)
+      rc = 0;
+
+  return rc;
+
+#if 0
+  std::string command;
+  command = "rm -rf " + this->sdb_name;
+  rc = system(command.c_str());
+  if (rc != 0)
+    ;
+  return 0;
+#endif
+}
+
+int files_pool::open_db(void)
+{
+  if (this->db_name == 0)
+    return NOFILENAME;
+
+  if (this->db_open)
+    return 0;
+
+  // create lockfilename
+  this->lockfilename = this->sdb_name+"/0lockfile";
+
+  int rc = access(this->db_name,R_OK|W_OK);
+  if (rc == -1)
+    return OPENERROR;
+
+  struct stat status;
+  stat(this->db_name, &status );
+
+  if ( !(status.st_mode & S_IFDIR) )
+     return OPENERROR;
+
+  this->lockfile = open(this->lockfilename.c_str(),O_CREAT|O_RDWR,0600);
+  if (this->lockfile < 0)
+    return LOCKERROR;
+
+  struct flock lock;
+  lock.l_type = F_RDLCK|F_WRLCK;
+  lock.l_whence = SEEK_SET;
+  lock.l_start  = 0;
+  lock.l_len    = 0;
+
+  rc = fcntl(this->lockfile,F_SETLKW,&lock);
+  if (rc < 0)
+    return LOCKERROR;
+
+  this->db_open = 1;
+
+  return 0;
+}
+
+int files_pool::close_db(void)
+{
+  if (!this->db_open)
+    return 0;
+
+  struct flock lock;
+  lock.l_type = F_UNLCK;
+  lock.l_whence = SEEK_SET;
+  lock.l_start  = 0;
+  lock.l_len    = 0;
+
+  int rc = fcntl(this->lockfile,F_SETLKW,&lock);
+  if (rc < 0)
+    return UNLOCKERROR;
+
+  close(this->lockfile);
+
+  this->db_open = 0;
+
+  return 0;
+}
+
Index: /tags/0.92/files_pool.h
===================================================================
--- /tags/0.92/files_pool.h	(revision 19)
+++ /tags/0.92/files_pool.h	(revision 19)
@@ -0,0 +1,48 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#ifndef FILES_POOL_H
+#define FILES_POOL_H
+#include "stopos_pool.h"
+#include "stopos_key.h"
+#include "stopos.h"
+
+class files_pool: public stopos_pool
+{
+
+  private:
+
+    int get_record(std::string &r, const std::string &k, const std::string &l);
+    int put_record(const std::string &r, const std::string &k,const std::string &l);
+    int remove_record(const std::string &k, const std::string &l);
+    int lockfile;
+    std::string lockfilename;
+
+    void create_fname(std::string &fname,const std::string &k);
+
+    const static int NOTDIR = 102;
+
+    std::string key_to_slot(const std::string &k){return "";}
+
+  public:
+    files_pool();
+    void set_db_name(std::string name);
+    int create_db(const std::string &dbname);
+    int purge_db(void);
+    int open_db(void);
+    int close_db(void);
+};
+#endif
Index: /tags/0.92/flatfile_pool.cpp
===================================================================
--- /tags/0.92/flatfile_pool.cpp	(revision 19)
+++ /tags/0.92/flatfile_pool.cpp	(revision 19)
@@ -0,0 +1,317 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#include "stopos_pool.h"
+#include "flatfile_pool.h"
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <stdlib.h>
+#include "wutils.h"
+#include <string.h>
+#include <strings.h>
+#include <stdio.h>
+#include <algorithm>
+#include "stopos.h"
+
+//
+// this->status.extra is always pointing to
+// the first free slot
+//
+// records are stored thusly:
+// character to denote if this record exists or is deleted
+// dbsep
+// 
+//   for a not deleted record, next fields:
+//   key for this record
+//   dbsep
+//   1st parameter of put_record
+//
+//   for a deleted record:
+//   pointer to next deleted record
+//
+
+// constructor
+flatfile_pool::flatfile_pool()
+{
+  this->whoami = "flat";
+  this->statuskey.set_slot(this->statuskey.get_key());
+}
+
+std::string flatfile_pool::key_to_slot(const std::string &k)
+{
+
+  if (strtoslot(this->status.extra) == 0)
+    return k;
+
+  std::string slot = this->status.extra;
+  std::string r;
+  off_t p = strtoslot(slot)*this->reclen;
+  this->read_string(r, p);
+  this->status.extra = r.substr(2);
+
+  return slot;
+}
+
+off_t flatfile_pool::strtoslot(const std::string &s)
+{
+  return stopos_key::KeyToNumber(s);
+}
+
+std::string flatfile_pool::slottostr(off_t l)
+{
+  return stopos_key::NumberToKey(l);
+}
+
+int flatfile_pool::haskey(std::string &r, const std::string &k)
+{
+  size_t q1 = 1; 
+  // find second separator. The first one is on postion 1
+  size_t q2 = r.find(dbsep,2);
+
+  std::string key = r.substr(q1+1,q2-q1-1);
+  if (key != k)
+    return this->NOTFOUND;
+  r.erase(0,q2+1);
+  return 0;
+}
+
+int flatfile_pool::get_record(std::string &r, const std::string &k, const std::string &l)
+{
+  int rc;
+
+  off_t p = this->strtoslot(l)*reclen;
+
+  rc = this->read_string(r,p);
+  if (rc !=0 )
+    return rc;
+
+  if (r[0] == this->deleted)
+    return this->NOTFOUND;
+
+  // this record was not deleted, but it could be that
+  // the key does not match
+  //
+
+  rc = this->haskey(r,k);
+  if (rc != 0)
+    return rc;
+
+  return 0;
+}
+
+int flatfile_pool::put_record(const std::string &r, 
+                              const std::string &k,
+			      const std::string &l)
+{
+  std::string buf;
+
+  bool rectoolong = 0;
+
+  buf = buf + this->exists + dbsep + k + dbsep + r;
+
+  if (buf.size() >= this->reclen) // >= because we need place for
+                                  // end-of-string zero
+  {
+    rectoolong = 1;
+    buf.resize(reclen-1);
+  }
+
+  off_t p = this->strtoslot(l)*reclen;
+
+  int rc =  this->write_string(buf,p);
+  if (!rectoolong)
+    return rc;
+  return RECORDTOOLONG;
+}
+
+int flatfile_pool::remove_record(const std::string &k, const std::string &l)
+{
+  int rc;
+
+  off_t p = this->strtoslot(l)*reclen;
+
+  std::string r;
+  rc = this->read_string(r,p);
+  if (rc != 0)
+    return rc;
+
+  rc = haskey(r,k);
+  if (rc != 0)
+    return rc;
+
+  r = this->deleted;
+
+  if (status.extra == slottostr(0))
+    r = r + dbsep + slottostr(0);
+  else
+    r = r + dbsep + status.extra;
+  status.extra = l;
+
+  return this->write_string(r,p);
+}
+
+int flatfile_pool::read_string(std::string &r, off_t p)
+{
+
+  char b[this->reclen];
+  off_t rc;
+  rc = lseek(this->ff, p, SEEK_SET);
+  if (rc == -1)
+    return FETCHERROR;
+
+  ssize_t rc1;
+  rc1 = read(this->ff, b, this->reclen);
+  if (rc1 != this->reclen)
+    return FETCHERROR;
+
+  r = b;
+
+  return 0;
+}
+
+
+int flatfile_pool::write_string(const std::string &r, off_t p)
+{
+  int rc;
+  char b[this->reclen];
+
+  rc = lseek(this->ff,p,SEEK_SET);
+  if (rc == -1)
+    return STOREERROR;
+
+  bzero(b, this->reclen);
+  strncpy(b,r.c_str(),r.size());
+  ssize_t first = 0;
+  ssize_t togo  = this->reclen;
+  ssize_t written;
+
+  do
+  {
+    written = write(ff,&b[first],togo);
+    if (written == -1)
+    {
+      perror("write:");
+      return 2;
+    }
+    togo   -= written;
+    first  += written;
+  } while(togo !=0);
+
+  return 0;
+}
+
+
+int flatfile_pool::create_db(const std::string &dbname)
+{
+  int rc; 
+
+  rc = this->purge_db();
+  if (rc != 0)
+    return rc;
+
+  // The db_name can be composed of parts separated with '/'
+  // we try to create this path until the last '/'
+  
+  size_t p = dbname.find_last_of('/');
+  rc = mkdir_p(dbname.substr(0,p+1),0755);
+  if (rc != 0)
+    return CREATEERROR;
+
+  rc = this->open_db(O_CREAT|O_RDWR);
+  if (rc != 0)
+    return rc;
+  this->close_db();
+
+  this->status.extra = stopos_key::NumberToKey(0);
+
+  return 0;
+}
+
+int flatfile_pool::purge_db(void)
+{
+  if (this->db_name == 0)
+    return NOFILENAME;
+
+  this->close_db();
+
+  unlink(this->db_name);
+  return 0;
+}
+
+int flatfile_pool::open_db()
+{
+  int rc;
+  rc = this->open_db(O_RDWR);
+  return rc;
+}
+
+int flatfile_pool::open_db(int flags)
+{
+  int rc;
+
+  if (this->db_name == 0)
+    return NOFILENAME;
+
+  if (this->db_open)
+    return 0;
+
+  this->ff = open(this->db_name, flags,0644);
+  if (this->ff < 0)
+    return OPENERROR;
+
+  struct flock lock;
+  lock.l_type   = F_RDLCK|F_WRLCK;
+  lock.l_whence = SEEK_SET;
+  lock.l_start  = 0;
+  lock.l_len    = 0;
+
+  rc = fcntl(this->ff,F_SETLKW,&lock);
+  if (rc < 0)
+    return LOCKERROR;
+
+  this->db_open = 1;
+
+  return 0;
+}
+
+int flatfile_pool::close_db(void)
+{
+  if (!this->db_open)
+     return 0;
+
+  int rc;
+
+  struct flock lock;
+  lock.l_type   = F_UNLCK;
+  lock.l_whence = SEEK_SET;
+  lock.l_start  = 0;
+  lock.l_len    = 0;
+
+  rc = fcntl(this->ff,F_SETLKW,&lock);
+  if (rc < 0)
+    return UNLOCKERROR;
+
+  rc = close(this->ff);
+  if (rc == -1)
+    return CLOSEERROR;
+
+  this->db_open = 0;
+
+  return 0;
+}
+
Index: /tags/0.92/flatfile_pool.h
===================================================================
--- /tags/0.92/flatfile_pool.h	(revision 19)
+++ /tags/0.92/flatfile_pool.h	(revision 19)
@@ -0,0 +1,56 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#ifndef FLATFILE_POOL_H
+#define FLATFILE_POOL_H
+#include "stopos_pool.h"
+#include "stopos_key.h"
+#include "stopos.h"
+
+class flatstatusrecord: statusrecord
+{
+  longuint nextd;
+};
+
+class flatfile_pool: public stopos_pool
+{
+  private:
+    int ff;    // handle of file to be used
+
+    int get_record(std::string &r, const std::string &k, const std::string &l);
+    int put_record(const std::string &r, const std::string &k,const std::string &l);
+    int remove_record(const std::string &k, const std::string &l);
+    int write_string(const std::string &r,off_t p);
+    int read_string(std::string &r,off_t p);
+
+    const static char deleted = 'd';
+    const static char exists  = '_';
+
+    std::string key_to_slot(const std::string &k);
+    off_t strtoslot(const std::string &s);
+    std::string slottostr(off_t l);
+    int haskey(std::string &r, const std::string &k);
+    int open_db(int flags);
+
+  public:
+    flatfile_pool();
+    int create_db(const std::string &dbname);
+    int purge_db(void);
+    int open_db(void);
+    int close_db(void);
+};
+
+#endif
Index: /tags/0.92/gdbm_pool.cpp
===================================================================
--- /tags/0.92/gdbm_pool.cpp	(revision 19)
+++ /tags/0.92/gdbm_pool.cpp	(revision 19)
@@ -0,0 +1,229 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#include <iostream>
+#include <string>
+#include <sstream>
+#include <iomanip>
+#include "stopos_pool.h"
+#include "gdbm_pool.h"
+#include "gdbm.h"
+#include <string.h>
+#include <stdlib.h>
+#include "wutils.h"
+#include <algorithm>
+#include <vector>
+#include "stopos.h"
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <unistd.h>
+
+gdbm_pool::gdbm_pool()
+{
+  this->whoami = "gdbm";
+}
+
+gdbm_pool::~gdbm_pool()
+{
+}
+
+int gdbm_pool::create_db(const std::string &dbname)
+{
+  int rc;
+
+  rc = this->purge_db();
+  if (rc != 0)
+    return rc;
+
+  // The db_name can be composed of parts separated with '/'
+  // we try to create this path until the last '/'
+  
+  size_t p = dbname.find_last_of('/');
+  rc = mkdir_p(dbname.substr(0,p+1),0755);
+  if (rc != 0)
+    return CREATEERROR;
+
+  this->dbf = gdbm_open(this->db_name,0,GDBM_NEWDB|this->sync,0644,0);
+
+  if (this->dbf == 0)
+    return this->OPENERROR;
+  this->close_db();
+
+  return 0;
+}
+
+int gdbm_pool::purge_db(void)
+{
+  if (this->db_name == 0)
+    return NOFILENAME;
+
+  this->close_db();
+
+  unlink(db_name);
+
+  return 0;
+}
+
+
+int gdbm_pool::open_db(void)
+{
+
+  if (this->db_open)
+    return 0;
+
+  if (this->db_name == 0)
+    return this->NOFILENAME;
+
+  // not really clear how gdbm signals that a file is not present
+  // so we open de file using open, and close it in close_db
+  //
+  this->ff = open(db_name,O_RDWR);
+  if (this->ff < 0)
+    return OPENERROR;
+
+  // only 'w' is used, left the other part in for future
+  // reference
+
+  // gdbm does not proper locking, somehow.
+  // so we use fcntl to get a lock on the file
+
+  struct flock lock;
+  lock.l_type   = F_RDLCK|F_WRLCK;
+  lock.l_whence = SEEK_SET;
+  lock.l_start  = 0;
+  lock.l_len    = 0;
+
+  int rc;
+
+  rc = fcntl(this->ff,F_SETLKW,&lock);
+  if (rc < 0)
+    return LOCKERROR;
+
+  char rw='w';
+  switch (rw)
+  {
+    case 'r':
+      do
+      {
+        dbf = gdbm_open(this->db_name,0,GDBM_READER,0644,0);
+        if (dbf == 0)
+          usleep(1000);
+      }
+      while (dbf == 0);
+      break;
+    case 'w':
+      do
+      {
+        dbf = gdbm_open(this->db_name,0,GDBM_WRCREAT|this->sync,0644,0);
+        if (dbf == 0)
+          usleep(1000);
+      }
+      while (dbf == 0);
+    break;
+    default:
+      break;
+  }
+  this->db_open = 1;
+  return 0;
+}
+
+int gdbm_pool::close_db(void)
+{
+  if (!db_open) 
+    return 0;
+
+  gdbm_close(this->dbf);
+
+  int rc;
+
+  struct flock lock;
+  lock.l_type   = F_UNLCK;
+  lock.l_whence = SEEK_SET;
+  lock.l_start  = 0;
+  lock.l_len    = 0;
+
+  rc = fcntl(this->ff,F_SETLKW,&lock);
+  if (rc < 0)
+    return UNLOCKERROR;
+
+  rc = close(this->ff);
+  if (rc == -1)
+    return CLOSEERROR;
+
+  this->db_open = 0;
+
+  return 0;
+}
+
+int gdbm_pool::get_record(std::string &r, const std::string &k, const std::string &l)
+{
+  datum key,content;
+  key.dptr = str_to_charp(k);
+  key.dsize = strlen(key.dptr) + 1;
+  content=gdbm_fetch(this->dbf,key);
+  free(key.dptr);
+  if (content.dptr == 0)
+    return this->FETCHERROR;
+
+  r = content.dptr;
+
+  free(content.dptr);
+
+  return 0;
+}
+
+int gdbm_pool::put_record(const std::string &r, const std::string &k, const std::string &l)
+{
+
+  datum key, content;
+  key.dptr = str_to_charp(k);
+  key.dsize = strlen(key.dptr) + 1;
+
+  content.dptr  = str_to_charp(r); 
+  content.dsize = strlen(content.dptr)+ 1;
+  bool rectoolong = 0;
+  /*
+  if (content.dsize > this->reclen)
+  {
+    rectoolong = 1;
+    WTD(this->reclen<<" "<<content.dsize<<" "<<strlen(content.dptr));
+    content.dsize = this->reclen;
+    content.dptr[this->reclen-1] = '\0';
+    WTD(this->reclen<<" "<<content.dsize<<" "<<strlen(content.dptr));
+  }
+  */
+  int rc = gdbm_store(this->dbf, key, content, GDBM_REPLACE);
+  free(content.dptr);
+  free(key.dptr);
+  if (!rectoolong)
+    return rc;
+  return this->RECORDTOOLONG;
+}
+
+int gdbm_pool::remove_record(const std::string &key, const std::string &l)
+{
+  datum k;
+  int rc;
+  k.dptr = str_to_charp(key);
+  k.dsize = strlen(k.dptr) + 1;
+
+  rc = gdbm_delete(this->dbf,k);
+  if (rc !=0)
+    return NOTFOUND;
+  return 0;
+}
+
Index: /tags/0.92/gdbm_pool.h
===================================================================
--- /tags/0.92/gdbm_pool.h	(revision 19)
+++ /tags/0.92/gdbm_pool.h	(revision 19)
@@ -0,0 +1,56 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#ifndef GDBM_POOL_H
+#define GDBM_POOL_H
+#include <string>
+#include "gdbm.h"
+#include "stopos_key.h"
+#include "stopos_pool.h"
+#include "stopos.h"
+ 
+class gdbm_pool : public stopos_pool
+{
+  private:
+    GDBM_FILE dbf;          // handle for access database
+    //static const int sync = GDBM_SYNC;
+    static const int sync = 0;
+
+    int get_record(std::string &r,const std::string &k,const std::string &l);
+    int put_record(const std::string &r,const std::string &k,const std::string &l);
+    int remove_record(const std::string &k, const std::string &l);
+
+    std::string key_to_slot(const std::string &k){return "";}
+
+    int ff;
+
+  public:
+    // constructor ()
+    gdbm_pool();
+
+    // destructor
+
+    ~gdbm_pool();
+
+
+    int create_db(const std::string &dbname);
+    int purge_db(void);
+
+    int open_db(void);
+    int close_db(void);
+};
+
+#endif
Index: /tags/0.92/index.html
===================================================================
--- /tags/0.92/index.html	(revision 19)
+++ /tags/0.92/index.html	(revision 19)
@@ -0,0 +1,12 @@
+<h2>Hallo</h2>
+
+<p>
+<form method="post" action="cgi-bin/hop/noot/mies/wim/zus">
+  <input type="submit" value="doit" name="naam">
+</form>
+<form method="post" action="cgi-bin/cgitest">
+  <input type="submit" value="test" name="naam">
+</form>
+<form method="post" action="cgi-bin/prenv/aap/noot">
+  <input type="submit" value="prenv" name="naam">
+</form>
Index: /tags/0.92/makefunction
===================================================================
--- /tags/0.92/makefunction	(revision 19)
+++ /tags/0.92/makefunction	(revision 19)
@@ -0,0 +1,17 @@
+
+###
+#   Copyright 2013 Willem Vermin, SURFsara
+#
+#   Licensed under the Apache License, Version 2.0 (the "License");
+#   you may not use this file except in compliance with the License.
+#   You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
+###
+eval 'stopos() { a="`'$PWD'/stoposclient "$@"`"; r=$?; eval "$a"; return $r;}'
Index: /tags/0.92/makegdbmh
===================================================================
--- /tags/0.92/makegdbmh	(revision 19)
+++ /tags/0.92/makegdbmh	(revision 19)
@@ -0,0 +1,11 @@
+#!/bin/bash
+# gdbm.h gives the following warning:
+# warning: ‘gdbm_pool’ has a field ‘gdbm_pool::dbf’ whose type 
+# uses the anonymous namespace [enabled by default]
+# this is because the typedef of GDBM_FILE contains a nameless
+# struct.
+# this script adds a name to this struct
+sed '/typedef struct.*GDBM_FILE/s/struct/struct gdbmfilestruct/' /usr/include/gdbm.h |
+  sed 's/char \*dptr/void *dptr/' > gdbm.h
+  exit
+  | sed 's/P((char/P((void/' > gdbm.h
Index: /tags/0.92/makeit
===================================================================
--- /tags/0.92/makeit	(revision 19)
+++ /tags/0.92/makeit	(revision 19)
@@ -0,0 +1,6 @@
+#!/bin/bash
+umask 022
+
+make || exit
+make install || exit
+
Index: /tags/0.92/modulefile
===================================================================
--- /tags/0.92/modulefile	(revision 19)
+++ /tags/0.92/modulefile	(revision 19)
@@ -0,0 +1,28 @@
+#%Module
+##
+##        stopos 0.92
+##
+
+proc ModulesHelp { } {
+        puts stderr "Sets up paths you need to use the stopos software"
+}
+
+set root /home/willem/myroot/sara/sw/stopos-0.92
+
+if { [ module-info shelltype ] == "csh" } {
+  set-alias stopos  "eval \"`$root/bin/stoposclient $*`\""
+} else { 
+         if { [module-info mode] == "load" } {
+
+	    puts stdout "stopos() { a=\"`$root/bin/stoposclient \"$@\"`\"; r=\$?; eval \"\$a\"; return \$r;}; export -f stopos;"
+	 } else {
+	    puts stdout "unset stopos;"
+	 }
+}
+
+prepend-path PATH               $root/bin
+prepend-path MANPATH            $root/man
+
+setenv STOPOS_SERVER_URL        http://stopos.osd.surfsara.nl/cgi-bin/stoposserver
+
+setenv SARA_STOPOS_ROOT          $root
Index: /tags/0.92/mysql_pool.cpp
===================================================================
--- /tags/0.92/mysql_pool.cpp	(revision 19)
+++ /tags/0.92/mysql_pool.cpp	(revision 19)
@@ -0,0 +1,281 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#include <iostream>
+#include "wutils.h"
+#include "stopos.h"
+
+#include "mysql_connection.h"
+#include <cppconn/driver.h>
+#include <cppconn/exception.h>
+#include <cppconn/resultset.h>
+#include <cppconn/statement.h>
+#include <cppconn/prepared_statement.h>
+
+#include "stopos_pool.h"
+#include "mysql_pool.h"
+// construcor
+mysql_pool::mysql_pool()
+{
+  this->whoami = "mysql";
+  std::string schema = "stopos";
+  sql::Driver *driver;
+  driver = get_driver_instance();
+  //  connection, mysqlusername, mysqlpassword
+  this->connection = driver->connect("tcp://127.0.0.1:3307",schema,"x");
+  this->connection->setSchema(schema);
+  this->stmt = this->connection->createStatement();
+}
+
+mysql_pool::~mysql_pool(void)
+{
+  delete this->stmt;
+  delete this->connection;
+}
+
+int mysql_pool::create_db(const std::string &dbname)
+{
+  int rc;
+  rc = this->purge_db();
+  if (rc != 0)
+    return rc;
+
+  try
+  {
+    int rc;
+
+    rc = this->open_db();
+    if (rc != 0)
+      return rc;
+    this->close_db();
+
+    return 0;
+  }
+  catch (sql::SQLException &e)
+  {
+    std::cerr << "# ERR: SQLException in " << __FILE__;
+    std::cerr << "(" << __FUNCTION__ << ") on line "
+       << __LINE__ << std::endl;
+    std::cerr << "# ERR: " << e.what();
+    std::cerr << " (MySQL error code: " << e.getErrorCode();
+    std::cerr << ", SQLState: " << e.getSQLState() <<
+   " )" << std::endl;
+    return OPENERROR;
+  }
+}
+
+int mysql_pool::purge_db(void)
+{
+  if (this->db_name == 0)
+    return NOFILENAME;
+  try
+  {
+    std::string cmd = "DROP TABLE IF EXISTS " + this->sdb_name;
+    this->stmt->execute(cmd);
+
+    cmd = "CREATE TABLE " + this->sdb_name +
+	  "(pkey CHAR(20) NOT NULL,"
+	  " PRIMARY KEY (pkey),"
+	  " INDEX(pkey),"
+	  " data VARCHAR(" +
+	    NumberToString(this->reclen) +
+	  +") NOT NULL)";
+    this->stmt->execute(cmd);
+  }
+  catch (sql::SQLException &e)
+  {
+    std::cerr << "# ERR: SQLException in " << __FILE__;
+    std::cerr << "(" << __FUNCTION__ << ") on line "
+       << __LINE__ << std::endl;
+    std::cerr << "# ERR: " << e.what();
+    std::cerr << " (MySQL error code: " << e.getErrorCode();
+    std::cerr << ", SQLState: " << e.getSQLState() <<
+   " )" << std::endl;
+    return PURGEERROR;
+  }
+    return 0;
+}
+
+int mysql_pool::open_db(void)
+{
+  if (! this->db_name)
+    return OPENERROR;
+
+  if (this->db_open)
+    return 0;
+
+  try
+  {
+    sql::ResultSet *res;
+    std::string cmd;
+    cmd="select count(*) as count "
+	"from information_schema.tables "
+	"where table_schema='stopos' "
+	"and table_name = '"+this->sdb_name+"'";
+    res = this->stmt->executeQuery(cmd);
+    res->beforeFirst();
+    if (res->next())
+      if (res->getInt("count") == 0)
+	return OPENERROR;
+    delete res;
+
+    cmd = "lock tables " + this->sdb_name + " write";
+    this->stmt->execute(cmd);
+  }
+  catch (sql::SQLException &e)
+  {
+    std::cerr << "# ERR: SQLException in " << __FILE__;
+    std::cerr << "(" << __FUNCTION__ << ") on line "
+       << __LINE__ << std::endl;
+    std::cerr << "# ERR: " << e.what();
+    std::cerr << " (MySQL error code: " << e.getErrorCode();
+    std::cerr << ", SQLState: " << e.getSQLState() <<
+   " )" << std::endl;
+    return OPENERROR;
+  }
+
+  this->db_open = 1;
+
+  return 0;
+}
+
+int mysql_pool::close_db(void)
+{
+  if (!this->db_open)
+    return 0;
+
+  std::string cmd;
+  try
+  {
+    cmd = "unlock tables";
+    this->stmt->execute(cmd);
+  }
+  catch (sql::SQLException &e)
+  {
+    std::cerr << "# ERR: SQLException in " << __FILE__;
+    std::cerr << "(" << __FUNCTION__ << ") on line "
+       << __LINE__ << std::endl;
+    std::cerr << "# ERR: " << e.what();
+    std::cerr << " (MySQL error code: " << e.getErrorCode();
+    std::cerr << ", SQLState: " << e.getSQLState() <<
+   " )" << std::endl;
+    return OPENERROR;
+  }
+
+  this->db_open = 0;
+
+  return 0;
+}
+
+int mysql_pool::get_record(std::string &r, const std::string &k, const std::string &l)
+{
+  try
+  {
+    std::string cmd;
+    sql::ResultSet *res;
+    cmd = "select data from " + sdb_name +
+        " where pkey = '" + k + "'";
+    res = this->stmt->executeQuery(cmd);
+    res->beforeFirst();
+    if (res->next() == 0)
+      return NOTFOUND;
+    r = res->getString("data"); 
+    delete res;
+  }
+  catch (sql::SQLException &e)
+  {
+    std::cerr << "# ERR: SQLException in " << __FILE__;
+    std::cerr << "(" << __FUNCTION__ << ") on line "
+       << __LINE__ << std::endl;
+    std::cerr << "# ERR: " << e.what();
+    std::cerr << " (MySQL error code: " << e.getErrorCode();
+    std::cerr << ", SQLState: " << e.getSQLState() <<
+   " )" << std::endl;
+    return NOTFOUND;
+  }
+
+  return 0;
+}
+
+int mysql_pool::put_record(const std::string &r, 
+                           const std::string &k,
+			   const std::string &l)
+{
+  bool rectoolong=0;
+  try
+  {
+    std::string cmd;
+    std::string rr;
+    rr = r;
+
+    if (rr.size() >= reclen)
+    {
+      rectoolong = 1;
+      rr.resize(reclen-1);
+    }
+
+    cmd = "replace " + sdb_name +
+      " values('" + k + "','" +
+      rr + "')";
+
+    this->stmt->execute(cmd);
+  }
+  catch (sql::SQLException &e)
+  {
+    std::cerr << "# ERR: SQLException in " << __FILE__;
+    std::cerr << "(" << __FUNCTION__ << ") on line "
+       << __LINE__ << std::endl;
+    std::cerr << "# ERR: " << e.what();
+    std::cerr << " (MySQL error code: " << e.getErrorCode();
+    std::cerr << ", SQLState: " << e.getSQLState() <<
+   " )" << std::endl;
+    return STOREERROR;
+  }
+
+  if (rectoolong)
+    return RECORDTOOLONG;
+
+  return 0;
+}
+
+int mysql_pool::remove_record(const std::string &k, const std::string &l)
+{
+  try
+  {
+    std::string cmd;
+
+    cmd = "delete from " + sdb_name +
+          " where pkey ='" + k + "'";
+
+    this->stmt->execute(cmd);
+  }
+  catch (sql::SQLException &e)
+  {
+    std::cerr << "# ERR: SQLException in " << __FILE__;
+    std::cerr << "(" << __FUNCTION__ << ") on line "
+       << __LINE__ << std::endl;
+    std::cerr << "# ERR: " << e.what();
+    std::cerr << " (MySQL error code: " << e.getErrorCode();
+    std::cerr << ", SQLState: " << e.getSQLState() <<
+   " )" << std::endl;
+    return STOREERROR;
+  }
+
+  return 0;
+}
+
+
+
Index: /tags/0.92/mysql_pool.h
===================================================================
--- /tags/0.92/mysql_pool.h	(revision 19)
+++ /tags/0.92/mysql_pool.h	(revision 19)
@@ -0,0 +1,52 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#ifndef MYSQL_POOL_H
+#define MYSQL_POOL_H
+#include "stopos_pool.h"
+#include "stopos_key.h"
+#include "mysql_connection.h"
+#include "stopos.h"
+#include <cppconn/driver.h>
+#include <cppconn/exception.h>
+#include <cppconn/resultset.h>
+#include <cppconn/statement.h>
+#include <cppconn/prepared_statement.h>
+
+class mysql_pool: public stopos_pool
+{
+  private:
+
+    int get_record(std::string &r, const std::string &k, const std::string &l);
+    int put_record(const std::string &r, const std::string &k, const std::string &l);
+    int remove_record(const std::string &k, const std::string &l);
+
+    void create_fname(std::string &fname,const stopos_key &k);
+
+    std::string key_to_slot(const std::string &k){return "";}
+
+    sql::Connection *connection;
+    sql::Statement *stmt;
+
+  public:
+    mysql_pool();
+    ~mysql_pool();
+    int create_db(const std::string &dbname);
+    int purge_db(void);
+    int open_db(void);
+    int close_db(void);
+};
+#endif
Index: /tags/0.92/prenv.cpp
===================================================================
--- /tags/0.92/prenv.cpp	(revision 19)
+++ /tags/0.92/prenv.cpp	(revision 19)
@@ -0,0 +1,23 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#include <iostream>
+#include <stdlib.h>
+int main()
+{
+  int rc = system("/bin/echo 'Content-Type: text/plain';/bin/echo;/usr/bin/env");
+  return rc;
+}
Index: /tags/0.92/protocol
===================================================================
--- /tags/0.92/protocol	(revision 19)
+++ /tags/0.92/protocol	(revision 19)
@@ -0,0 +1,31 @@
+A message consists of:
+
+header
+data
+end-of-data-marker  a character with value zero
+
+header:
+stopos              literally: "stopos"
+TAB                 field separator: '/'
+protocolnumber      human readable string, eg "0001"
+TAB
+whoami              human readable string
+TAB
+command             human readable string, eg "next"
+TAB
+poolname            poolname, eg "mypool"
+TAB
+user                human readable string, eg "willem"
+TAB
+
+length of header may not exceed 256 bytes
+
+data:
+string not containing 0-bytes, example "-l 1 2 3" Max length = 4096
+
+example:
+stopos\t0001\tnext\tmypool\twillem\t-l 1 2 3\0
+
+None of above may containg 0-bytes, except for the end-of-data marker
+
+
Index: /tags/0.92/runtiming
===================================================================
--- /tags/0.92/runtiming	(revision 19)
+++ /tags/0.92/runtiming	(revision 19)
@@ -0,0 +1,19 @@
+#!/bin/bash
+doit()
+{
+  seq 100 | stopos add
+  while true ; do
+    stopos next
+    if [ "$STOPOS_RC" != OK ]; then
+      break
+    fi
+    stopos remove
+  done
+}
+module load stopos
+for p in gdbm flat files mysql ; do
+  export STOPOS_PROT=$p
+  echo  $STOPOS_PROT :
+  stopos create
+  time doit
+done
Index: /tags/0.92/sara-get-mem-size
===================================================================
--- /tags/0.92/sara-get-mem-size	(revision 19)
+++ /tags/0.92/sara-get-mem-size	(revision 19)
@@ -0,0 +1,3 @@
+#!/bin/bash
+cat /proc/meminfo | awk '
+  /MemTotal:/ { print int($2/1000000) ; exit }'
Index: /tags/0.92/sara-get-mem-size.1
===================================================================
--- /tags/0.92/sara-get-mem-size.1	(revision 19)
+++ /tags/0.92/sara-get-mem-size.1	(revision 19)
@@ -0,0 +1,17 @@
+.TH SARA-GET-MEM-SIZE 1 
+.SH NAME
+sara-get-mem-size \-  get the amount of memory on the node
+.SH SYNOPSIS
+.B sara-get-mem-size
+.SH DESCRIPTION
+.I Sara-get-mem-size
+Writes the amount of memory in Mbytes to standard output.
+
+.TP 8
+.SH "SEE ALSO"
+sara-get-num-cores(1), topos(1)
+.SH AUTHORS
+Willem Vermin
+.br
+.SH BUGS
+No bugs known yet.
Index: /tags/0.92/sara-get-num-cores
===================================================================
--- /tags/0.92/sara-get-num-cores	(revision 19)
+++ /tags/0.92/sara-get-num-cores	(revision 19)
@@ -0,0 +1,4 @@
+#!/bin/bash
+cat /proc/cpuinfo | awk '
+  /processor/ { p = $3 }
+  END { print p+1 }'
Index: /tags/0.92/sara-get-num-cores.1
===================================================================
--- /tags/0.92/sara-get-num-cores.1	(revision 19)
+++ /tags/0.92/sara-get-num-cores.1	(revision 19)
@@ -0,0 +1,17 @@
+.TH SARA-GET-NUM-CORES 1 
+.SH NAME
+sara-get-num-cores \-  get the number of cores on the node
+.SH SYNOPSIS
+.B sara-get-num-cores
+.SH DESCRIPTION
+.I Sara-get-num-cores
+Writes the number of cores in the node to standard output.
+
+.TP 8
+.SH "SEE ALSO"
+sara-get-mem-size(1), topos(1)
+.SH AUTHORS
+Willem Vermin
+.br
+.SH BUGS
+No bugs known yet.
Index: /tags/0.92/shellenv.cpp
===================================================================
--- /tags/0.92/shellenv.cpp	(revision 19)
+++ /tags/0.92/shellenv.cpp	(revision 19)
@@ -0,0 +1,55 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#include <string>
+#include "shellenv.h"
+//
+// Example:
+//   shellenv("aap noot mies", "LEESPLANK", SHELL_SH);
+// returns the string:
+//   LEESPLANK='aap noot mies';export LEESPLANK;
+//
+//  "'" characters in s are escaped
+//
+
+std::string shellenv(std::string s, const std::string name, int shelltype)
+{
+  std::string r = "'";
+  std::string::iterator it;
+  for ( it=s.begin() ; it < s.end(); it++ )
+  {
+    if ( *it == '\'')
+    {
+      r.push_back('\'');
+      r.push_back('\\');
+      r.push_back('\'');
+      r.push_back('\'');
+    }
+    else
+      r.push_back(*it);
+  }
+  r.push_back('\'');
+  switch (shelltype)
+  {
+    case SHELL_CSH:
+      r = "setenv " + name + " " + r + ";\n";
+      break;
+    case SHELL_SH: 
+      r = name + "=" + r + ";" + "export " + name + ";\n";
+      break;
+  }
+  return r;
+}
Index: /tags/0.92/shellenv.h
===================================================================
--- /tags/0.92/shellenv.h	(revision 19)
+++ /tags/0.92/shellenv.h	(revision 19)
@@ -0,0 +1,20 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+
+const int SHELL_SH = 0;
+const int SHELL_CSH = 1;
+std::string shellenv(const std::string s, const std::string name, int shelltype);
Index: /tags/0.92/showflatfiledb.c
===================================================================
--- /tags/0.92/showflatfiledb.c	(revision 19)
+++ /tags/0.92/showflatfiledb.c	(revision 19)
@@ -0,0 +1,35 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#include <stdio.h>
+int main()
+{
+  int c;
+  int s=1;
+  while ((c=getchar())!=EOF)
+  {
+    if (c == 0)
+    {
+      if (s) putchar('\n');
+      s=0;
+    }
+    else
+    {
+      putchar(c);
+      s=1;
+    }
+  }
+}
Index: /tags/0.92/split.cpp
===================================================================
--- /tags/0.92/split.cpp	(revision 19)
+++ /tags/0.92/split.cpp	(revision 19)
@@ -0,0 +1,44 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#include <string>
+#include <vector>
+// 
+// splits string s to vector v, using separator character sep
+// return value: v
+// It seems that it follows the python implementation of split()
+//
+std::vector <std::string> split(std::vector <std::string> &v, const std::string s, const char sep)
+{
+  v.clear();
+  for (size_t p=0, q=0; q != s.npos; p = q + 1)
+    v.push_back(s.substr(p,(q=s.find(sep,p))-p));
+  return v;
+}
+#if 0
+#include <iostream>
+int main()
+{
+  char sep = '/';
+  std::string s="/version/command/ /modifier/";
+  std::vector <std::string> v;
+
+  v=split(v,s,sep);
+  for (unsigned i=0; i<v.size(); i++)
+    std::cout << i << ":" << v[i] << ":" << std::endl;
+  return 0;
+}
+#endif
Index: /tags/0.92/startthttpd
===================================================================
--- /tags/0.92/startthttpd	(revision 19)
+++ /tags/0.92/startthttpd	(revision 19)
@@ -0,0 +1,19 @@
+#!/bin/bash
+###
+#   Copyright 2013 Willem Vermin, SURFsara
+#
+#   Licensed under the Apache License, Version 2.0 (the "License");
+#   you may not use this file except in compliance with the License.
+#   You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
+###
+root=$HOME/myroot/usr/local/www
+killall thttpd 2>/dev/null
+thttpd -p 5050 -d $root -l $root/log -c "/cgi-bin/*"
Index: /tags/0.92/stopos.h
===================================================================
--- /tags/0.92/stopos.h	(revision 19)
+++ /tags/0.92/stopos.h	(revision 19)
@@ -0,0 +1,33 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#ifndef STOPOS_H
+#define STOPOS_H
+#include "wtd.h"
+static const char dbsep           = '/';  // separator used in database
+static const char httpsep         = '-';  // separator used in http string from client to server
+static const char outsep          = '+';  // separator used in output from server to client
+static const char keysep          = '.';  // separator to separate stopos
+                                          // key from extra key
+const std::string program_name    = "stopos";
+const std::string program_version = "0.92";
+const std::string default_pool    = "pool";
+
+typedef unsigned long long longuint;
+
+const static size_t maxlinelength = 1024;
+
+#endif
Index: /tags/0.92/stopos.man
===================================================================
--- /tags/0.92/stopos.man	(revision 19)
+++ /tags/0.92/stopos.man	(revision 19)
@@ -0,0 +1,200 @@
+.\"
+.\" Copyright 2013 Willem Vermin, SURFsara
+.\"
+.\" Licensed under the Apache License, Version 2.0 (the "License");
+.\" you may not use this file except in compliance with the License.
+.\" You may obtain a copy of the License at
+.\"
+.\"     http://www.apache.org/licenses/LICENSE-2.0
+.\"
+.\" Unless required by applicable law or agreed to in writing, software
+.\" distributed under the License is distributed on an "AS IS" BASIS,
+.\" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.\" See the License for the specific language governing permissions and
+.\" limitations under the License.
+.\"
+
+.TH STOPOS 1 
+.SH NAME
+stopos \- alias to call the program stoposclient and eval it's output
+.SH SYNOPSIS
+.B stopos
+\fB\ -h,--help  \fP  
+.P 8
+.B stopos
+\fB\ -v,--version  \fP  
+.P 8
+.B stopos
+\fB\ create \fP   [\fB -p,--pool POOL\fP ]
+.P 8
+.B stopos
+\fB\ status \fP   [\fB -p,--pool POOL\fP]
+.P 8
+.B stopos
+\fB\ purge \fP    [\fB -p,--pool POOL\fP ]
+.P 8
+.B stopos
+\fB\ pools \fP
+.P 8
+.B stopos
+\fB\ add   \fP    [\fB -p,--pool POOL\fP ] [\fB FILENAME \fB]
+.P 8
+.B stopos
+\fB\ next  \fP    [\fB -p,--pool POOL\fP ] [\fB -m,--multi \fB]
+.P 8
+.B stopos
+\fB\ remove \fP   [\fB -p,--pool POOL\fP] [\fB KEY \fP ]
+.P 8
+.B stopos
+\fB\ dump \fP     [\fB -p,--pool POOL\fP]
+.P 8
+The -q,--quiet flag supresses most output.
+.P 8
+.SH DESCRIPTION
+Stopos is an alias using the program stoposclient, an utility to store
+and retrieve text lines in a pool.
+In general, the
+text lines are used as command parameters, see Examples.
+.SH COMMANDS
+.TP 4
+.B -h,--help
+Prints usage information.
+.TP 4
+.B -v,--version
+Prints version.
+.TP 4
+.B create [ -p,--pool POOL ] [ FILENAME ]
+Creates new pool.
+A pool with the same name will be removed first.
+.TP 4
+.B status
+Prints to standard error       environment variable 
+.br
+--------------------------------------------------- 
+.br
+total number of lines added      STOPOS_COUNT
+.br 
+number of lines present          STOPOS_PRESENT
+.br
+number of lines never committed  STOPOS_PRESENT0
+.TP 4
+.TP 4
+.B purge [-p,--pool POOL ]
+Removes the pool.
+.TP 4
+.B pools [-p,--pool POOL ]
+Lists on standard error the pools available. The environment 
+variable STOPOS_VALUE is set accordingly.
+.TP 4
+.B add [ -p,--pool POOL ] [ FILENAME ]
+Adds lines from FILENAME, default from stdin. 
+The maximum line length is 1024 characters.
+The environment variables STOPOS_RC and STOPS_KEY are set. 
+.br
+NOTE: when stopos is reading from a pipe as in: 
+.br
+   cat parmfile | stopos add
+.br
+no environment variables are set.
+.TP 4
+.B next [ -p,--pool POOL ] [ -m,--multi ]
+Gets the next line from the pool.
+The environment variables STOPOS_VALUE, STOPOS_COMMITTED and STOPOS_KEY are set,
+see ENVIRONMENT.
+By default, the same line in the pool will be
+produced only once. When all lines are commited, STOPS_RC will not
+be equal to OK. Using the --multi flag, the same line can be produced 
+more than once, if necessary stopos will wrap around.
+This can be useful for dealing with crashed jobs.
+.TP 4
+.B remove [-p,--pool POOL ] [ KEY ]
+Removes the line with key KEY. 
+If not specified on the commandline, the
+value of environment variable STOPOS_KEY is used.
+.TP 4
+.B dump  [-p,--pool POOL ]
+Reads the next available line, and puts it,
+preceded by it's key and number of commitments in environment variable
+STOPOS_VALUE. The first call will produce the first line. 
+When all lines have been delivered, STOPOS_RC
+gets a value other than OK. A subsequent call will start the dump
+with the first line again.
+.SH OPTIONS
+.TP 4
+.B -p,--pool POOL
+POOL is a unique name of the pool. 
+Default: pool.
+The name of the pool can also be set using the environment variable STOPOS_POOL. The command line flag has precedence.
+
+.SH ENVIRONMENT
+Stopos sets the following environment variables:
+.P
+.B STOPOS_RC
+if the value is OK, than no errors were found
+.P
+.B STOPOS_KEY
+contains the keyvalue of the line produced with the 'next' command, or
+the keyvalue of the line just added. The keyvalue (a short string) is 
+guaranteed to be unique between to 'create"s' or between 
+'create' and 'purge'
+.P
+.B STOPOS_COMMITTED
+contains the number of times the line has been committed 
+.P
+.B STOPOS_VALUE
+contains the line produced by the 'next' command or the result of the 'pools' command
+.P
+.B STOPOS_COUNT STOPOS_PRESENT STOPOS_PRESENT0
+see the 'status' command above
+.P
+Stopos uses the following environment variables:
+.P
+.B STOPOS_POOL 
+see the description of the --pool flag under OPTIONS
+.P
+.B STOPOS_KEY 
+see the 'remove' flag above
+.P
+.B STOPOS_SERVER_URL 
+the url stopos uses to access the server, default DEFAULTSERVERURL
+
+.SH EXAMPLES
+.TP 4
+
+Create a pool with the first 10 lines from the man page of sed:
+
+ man sed | head > parmfile
+ stopos create 
+ stopos add parmfile
+
+Get a line:
+
+ stopos next
+
+The environment variable STOPOS_VALUE contains now one of the
+first ten lines of the man page of sed.
+
+You can use this in a command like this:
+
+ eval "myparser $STOPOS_VALUE"
+
+The line can be removed by:
+
+ stopos remove
+
+Finally, the pool can be completely removed by:
+
+ stopos purge
+
+.SH FILES
+$HOME/.stopos/id
+.br
+This file is set by stopos to store an unique id, which, together 
+with the login name, will be used to identify which pools are yours.
+.SH "SEE ALSO"
+sara-get-mem-size(1), sara-get-num-cores(1), stoposclient(1), stoposdump(1)
+.SH AUTHORS
+Willem Vermin
+.br
+.SH BUGS
+No bugs known yet.
Index: /tags/0.92/stopos_key.cpp
===================================================================
--- /tags/0.92/stopos_key.cpp	(revision 19)
+++ /tags/0.92/stopos_key.cpp	(revision 19)
@@ -0,0 +1,93 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#include <string>
+#include <vector>
+#include "stopos.h"
+#include "wutils.h"
+#include "stopos_key.h"
+
+// constructor
+stopos_key::stopos_key()
+{
+  this->set(0,"");
+}
+
+stopos_key::stopos_key(const std::string & s)
+{
+  std::vector <std::string> v;
+  v = split(v,s,keysep);
+  this->value = v[0];
+  if (v.size()>1)
+    this->slot = v[1];
+}
+
+void stopos_key::impossible()
+{
+  this->value="%%%%";
+}
+
+void stopos_key::increment(void)
+{ // notused
+  int p = this->value.size();
+  while(1)
+  {
+    p--;
+    if (this->value[p] == lastchar)
+    {
+      this->value[p] = firstchar;
+      if (p == 0)
+      {
+	this->value = firstchar + this->value;
+	p = 1;
+      }
+    }
+    else
+    {
+      this->value[p] ++;
+      break;
+    }
+  }
+}
+
+std::string stopos_key::str(void) const
+{
+  if (this->slot == "")
+    return value;
+
+  return this->value+keysep+this->slot;
+}
+
+std::string stopos_key::get_key(void) const
+{
+  return this->value;
+}
+
+std::string stopos_key::get_slot(void) const
+{
+  return this->slot;
+}
+
+bool stopos_key::operator==(const stopos_key &k) const
+{
+  return k.value == this->value;
+}
+
+bool stopos_key::operator!=(const stopos_key &k) const
+{
+  return k.value != this->value;
+}
+
Index: /tags/0.92/stopos_key.h
===================================================================
--- /tags/0.92/stopos_key.h	(revision 19)
+++ /tags/0.92/stopos_key.h	(revision 19)
@@ -0,0 +1,97 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#ifndef STOPOS_KEY_H
+#define STOPOS_KEY_H
+#include <string>
+#include <iostream>
+#include "stopos.h"
+
+class stopos_key
+{
+  protected:
+    std::string value;
+    std::string slot;
+    static const char firstchar = 'a';
+    static const char lastchar  = 'z';
+    static const unsigned int base = lastchar - firstchar + 1;
+    template <typename T> void tostring1(std::string &v,const T n)
+    {
+      if (n == 0)
+        return;
+      unsigned int c = n + firstchar - base*(n/base);
+      tostring1(v,n/base);
+      v.push_back(c);
+    }
+    template <typename T> void tostring(std::string &v,const T n)
+    {
+      if (n == 0)
+	v = firstchar;
+      else
+	tostring1(v,n);
+    }
+  public:
+    stopos_key();
+    stopos_key(const std::string &s);
+    void increment(void);
+    std::string str(void) const;
+    std::string get_key(void) const;
+    std::string get_slot(void) const;
+    void set_key(const std::string &k){ this->value = k;}
+    void set_slot(const std::string &s){ this->slot = s;}
+    void impossible();
+    bool operator==(const stopos_key &k) const;
+    bool operator!=(const stopos_key &k) const;
+
+    // to assign an unsigned integer value to a key
+    template <typename T> void set(const T k,const std::string &slot)
+      {
+        this->value = "";
+        this->tostring(this->value,k);
+	this->slot = slot;
+        return;
+      }
+
+    // to get the unsigned integer value from a key
+    template <typename T> T get() const
+    {
+      T r = 0;
+
+      for (unsigned int i=0; i<this->value.size(); i++)
+        r = this->base*r + this->value[i]-this->firstchar;
+
+      return r;
+    }
+
+    static longuint KeyToNumber(const std::string s)
+    {
+      longuint r = 0;
+      stopos_key d;
+      for (unsigned int i=0; i<s.size(); i++)
+        r = base*r + s[i]-firstchar;
+
+      return r;
+    }
+
+    template <typename T> static std::string NumberToKey(const T n)
+    {
+      std::string s;
+      stopos_key k;
+      k.tostring(s,n);
+      return s;
+    }
+};
+#endif
Index: /tags/0.92/stopos_pool.cpp
===================================================================
--- /tags/0.92/stopos_pool.cpp	(revision 19)
+++ /tags/0.92/stopos_pool.cpp	(revision 19)
@@ -0,0 +1,671 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#include "stopos_pool.h"
+#include "stopos.h"
+#include "wutils.h"
+#include <iostream>
+#include <string.h>
+#include <stdlib.h>
+
+int stopos_pool::get_record(std::string &r, const std::string &k, const std::string &l)
+  // non-sense example implementation, you have to overwrite this in a derived class
+{
+  std::cerr << __FILE__<<":"<<__LINE__<<"stopos_pool::get_record must be overwritten in a derived class"<<std::endl;
+  return 0;
+}
+
+int stopos_pool::put_record(const std::string &r, const std::string &k,const std::string &l)
+  // non-sense example implementation, you have to overwrite this in a derived class
+{
+  std::cerr << __FILE__<<":"<<__LINE__<<"stopos_pool::put_record must be overwritten in a derived class"<<std::endl;
+  std::cout <<r<<std::endl;
+  return 0;
+  //
+  // record r must be stored such that get_record() retrieves it using k
+  //
+}
+
+int stopos_pool::remove_record(const std::string &k, const std::string &l)
+  // non-sense example implementation, you have to overwrite this in a derived class
+{
+  std::cerr << __FILE__<<":"<<__LINE__<<"stopos_pool::remove_record must be overwritten in a derived class"<<std::endl;
+  std::cout << "Please forget the existence of the record with the following key:"<<k<<std::endl;
+  return 0;
+}
+
+int stopos_pool::create_db(const std::string &db)
+  // non-sense example implementation, you have to overwrite this in a derived class
+{
+  std::cerr << __FILE__<<":"<<__LINE__<<"stopos_pool::create_db must be overwritten in a derived class"<<std::endl;
+  std::cout << "Please create file " << db<< std::endl;
+  return 0;
+}
+
+int stopos_pool::purge_db(void)
+  // non-sense example implementation, you have to overwrite this in a derived class
+{
+  std::cerr << __FILE__<<":"<<__LINE__<<"stopos_pool::purge_db must be overwritten in a derived class"<<std::endl;
+  std::cout << "Please purge file " << this->db_name<< std::endl;
+  return 0;
+}
+
+int stopos_pool::open_db(void)
+  // non-sense example implementation, you have to overwrite this in a derived class
+{
+  std::cerr << __FILE__<<":"<<__LINE__<<"stopos_pool::open_db must be overwritten in a derived class"<<std::endl;
+  std::cout << "Please open file " << this->db_name<<" and assign the result to an handle in your class" << std::endl;
+
+  this->db_open = 1;
+  return 0;
+}
+
+int stopos_pool::close_db(void)
+  // non-sense example implementation, you have to overwrite this in a derived class
+{
+  std::cerr << __FILE__<<":"<<__LINE__<<"stopos_pool::close_db must be overwritten in a derived class"<<std::endl;
+  std::cout << "Please close file " << this->db_name<<std::endl;
+
+  db_open = 0;
+  return 0;
+}
+
+// constructor:
+stopos_pool::stopos_pool()
+{
+  this->kvp         = 0;
+  this->statuskey   = stopos_key();  // full key
+  this->db_name     = 0;
+  this->db_open     = 0;
+}
+
+// destructor
+stopos_pool::~stopos_pool()
+{
+  if(db_name)
+    free (db_name);
+}
+
+int stopos_pool::create_db(void)
+{
+  int rc;
+  rc = this->create_db(this->sdb_name);
+  if(rc != 0)
+    return rc;
+
+  this->status.init();
+  rc = this->open_db();
+  if (rc != 0)
+    return rc;
+  rc =  this->put_status();
+  this->close_db();
+  return rc;
+}
+
+int stopos_pool::get_record(datarecord       &r,
+                            const stopos_key &k)
+{
+  std::string s;
+  int rc;
+  rc = this->get_record(s, k.get_key(),k.get_slot());
+  if (rc != 0)
+    return rc;
+  r = datarecord(s,dbsep);
+  return 0;
+}
+
+int stopos_pool::put_record(const datarecord &r,
+                            const stopos_key &k)
+{
+  std::string b;
+  int rc;
+
+  b = r.str(dbsep);
+
+  rc = this->put_record(b,k.get_key(),k.get_slot());
+
+  return rc;
+}
+
+int stopos_pool::remove_record(const stopos_key &k)
+{
+  return this->remove_record(k.get_key(),k.get_slot());
+}
+
+void stopos_pool::set_db_name(std::string filename)
+{
+  this->sdb_name = filename;
+  this->db_name  = strdup(filename.c_str());
+}
+
+std::string stopos_pool::get_db_name(void)
+{
+  return this->sdb_name;
+}
+
+int stopos_pool::put_status(void)
+{
+  std::string buf;
+  buf = this->status.str(dbsep);
+
+  return this->put_record(buf,this->statuskey.get_key(),this->statuskey.get_slot());
+}
+
+int stopos_pool::get_status(void)
+{
+  std::string buf;
+  int rc =this->get_record(buf,this->statuskey.get_key(),this->statuskey.get_slot());
+  if (rc != 0) 
+    return rc;
+
+  this->status = statusrecord(buf,dbsep);
+  return 0;
+}
+
+int stopos_pool::add_line(const std::string line, 
+                                std::string &key)
+{
+  if (!this->db_open)
+    return DBNOTOPEN;
+
+  datarecord d;
+  stopos_key dkey;
+  int rc;
+  rc = get_status();
+  if (rc != 0)
+    return rc;
+
+  //dkey is the key of the inserted record
+  //dkey = status.last; 
+  std::string slot = this->key_to_slot(stopos_key::NumberToKey(status.count +1));
+  dkey.set(status.count + 1,slot);
+
+  // if status.last != 0, modify 'next' of last record
+  if (status.last != this->statuskey)
+  {
+    datarecord l;
+    rc = this->get_record(l,this->status.last);
+    if (rc != 0)
+      return rc;
+    l.next = dkey;
+    rc = this->put_record(l,this->status.last);
+    if (rc !=0)
+      return rc;
+  }
+
+  // if status.next points to nothing
+  //   or
+  // if status.next points to a record that has been committed,
+  //   then
+  // status.next should point to the record just added
+
+  // modify status record
+
+  if (this->status.next == this->statuskey) // status.next points to nothing
+    this->status.next = dkey;
+  else  // look if status.next points to a record that has already been committed
+  {
+    datarecord r;
+    int rc;
+
+    rc = this->get_record(r,this->status.next);
+    if (rc != 0)
+      return FETCHERROR;
+
+    if (r.retrieved != 0)
+      this->status.next = dkey;
+  }
+
+  if (this->status.first == this->statuskey)
+    this->status.first = dkey;
+
+  d.prev            = status.last; 
+  this->status.last = dkey;
+
+  this->status.count++;
+  this->status.present++;
+  this->status.present0++;
+
+  rc = this->put_status();
+  if (rc != 0)
+    return this->STOREERROR;
+
+  d.value     = zstrtohex(line);
+  d.next      = this->statuskey;
+  d.retrieved = 0;
+
+  rc = put_record(d,dkey);
+
+  key = dkey.str();
+
+  return rc;
+}
+
+int stopos_pool::get_line(std::string      &line, 
+                          longuint         &retrieved, 
+			  std::string      &key)
+{
+  if (!this->db_open)
+    return DBNOTOPEN;
+
+  int rc;
+  rc = this->get_status();
+  if (rc != 0)
+    return FETCHERROR;
+
+  if (this->status.next == this->statuskey)
+  {
+    if (!this->kvp)
+      return  NOTFOUND;
+    // wrapping around ...
+    status.next = status.first;
+
+    if (this->status.next == this->statuskey)
+      return NOTFOUND;
+  }
+
+  datarecord r;
+  
+  rc = this->get_record(r,this->status.next);
+  if (rc != 0)
+    return FETCHERROR;
+
+  retrieved = r.retrieved;
+  if (!this->kvp)
+    if (r.retrieved !=0)
+      return NOTFOUND;
+  key = this->status.next.str();
+
+  line = zhextostr(r.value);
+
+  // adjust the 'retrieved' field of r, and write it
+
+  r.retrieved++;
+
+  rc = this->put_record(r,status.next);
+  if (rc != 0)
+    return STOREERROR;
+
+  // adjust the 'next' field of status. this must be equal
+  // to the 'next' field of r
+
+  this->status.next = r.next;
+
+  if (r.retrieved == 1)
+    this->status.present0--;
+
+  this->put_status();
+
+  return 0;
+}
+
+int stopos_pool::remove_line(const std::string &k)
+{
+  if (!this->db_open)
+    return DBNOTOPEN;
+
+  int rc;
+  datarecord r;
+  stopos_key key(k);
+
+  rc = this->get_record(r,key);
+  if (rc !=0)
+    return rc;
+
+  // remove this record from the linked lists.
+
+  rc=this->get_status();
+  if (rc != 0)
+    return rc;
+
+  //
+  //remove_line
+
+  // r = record to remove
+  // 
+  // situations we have to deal with:
+  // 
+  // r.next = 0  ( = key to status)
+  // r.next = data ( = key to data)
+  // 
+  // r.prev = 0
+  // r.prev = data
+  // 
+  // if (r.prev != 0)
+  //    get p(r.prev)
+  // 
+  // if (r.next != 0)
+  //    get n(r.next)
+  // 
+  // if (r.next = 0)    //r is last record
+  //   {
+  //      if r.prev = data
+  //      {
+  //          p.next = r.next     (=0)
+  //          if (key == s.next)
+  //             s.next = r.next  ( = 0)
+  //          if (key == s.last)     // must be true, r is last record
+  //             s.last = r.prev
+  //          if (key == s.first)
+  //             impossible, because there is a previous record
+  //      }
+  //      else  // r.prev = 0 // this is the only record
+  //      {
+  //          s.next = r.next ( =r.prev =0)
+  //          s.first = r.prev ( = r.next =0)
+  //          s.last  = r.prev ( = r.next = 0)
+  //      }
+  //   }
+  // else    // r has a follower
+  //  {
+  //      if r.prev = datarecord
+  //      {
+  //          p.next = r.next
+  //          n.prev = r.prev
+  //          if (key == s.next)
+  //             s.next = r.next
+  //          if (key == s.last)
+  //             impossible, there is a follower
+  //          if (key == s.first)
+  //             impossible, there is a predecessor
+  //      }
+  //      else  // r.prev = 0: r is the first record
+  //      {
+  //          n.prev = r.prev (=0)
+  //          if (key == s.next)
+  //             s.next = r.next
+  //          if (key == s.last)
+  //             impossible, there is a follower
+  //          if (key == s.first)  // must be
+  //             s.first = r.next
+  //      }
+  //  }
+
+  // remove record r
+  //
+
+  this->get_record(r,key); 
+  // 
+  // situations we have to deal with:
+  // 
+  // r.next = 0  ( = key to status)
+  // r.next = data ( = key to data)
+  // 
+  // r.prev = 0
+  // r.prev = data
+  // 
+
+  datarecord p,n;
+
+  if (r.prev != this->statuskey)
+    this->get_record(p,r.prev);
+  
+  if (r.next != this->statuskey)
+    this->get_record(n,r.next);
+//
+// and what to do if the next record to be dumped is
+// exactly this record?
+//
+// I think, status.nextd should become r.next,
+// whatever r.next is pointing to
+// However, if r.next is pointing to nothing,
+// we should give status.nextd an impossible value
+//
+  
+  if (this->status.nextd == key)
+  {
+    if (r.next == this->statuskey)
+      this->status.nextd.impossible();
+    else
+      this->status.nextd = r.next;
+  }
+
+  if (r.next == this->statuskey)  // r is last record
+  {
+    if (r.prev != this->statuskey)  // r is last record and has a predecessor
+      {
+	p.next = r.next;
+	this->put_record(p,r.prev);
+
+	if (key == status.next)
+	  status.next = r.next;
+
+	// if (key == status.last)  // always the case here
+	status.last = r.prev;
+
+	// if (key == status.first)
+	//   impossible, because there is a previous record
+      }
+    else  // r is the only record
+    {
+      status.next  = r.next;
+      status.first = r.prev;
+      status.last  = r.prev;
+    }
+  }
+  else   // r has a follower
+  {
+    if (r.prev != this->statuskey)  // r has a predecessor and a follower
+    {
+      p.next = r.next;
+      this->put_record(p,r.prev);
+
+      n.prev = r.prev;
+      this->put_record(n,r.next);
+
+      if (key == status.next)
+	status.next = r.next;
+      // if (key == status.last) 
+      //   impossible, r has a follower
+      // if (key == s.first)
+      //   impossible, r has a predecessor
+    }
+    else  // r is first record and has a follower
+    {
+      n.prev = r.prev;
+      this->put_record(n,r.next);
+
+      if (key == status.next)
+	status.next = r.next;
+
+      // if (key == status.last)
+      //   impossible, r has a follower
+      
+      // if (key == status.first)  // always true here
+      status.first = r.next;
+    }
+  }
+
+  // finally, remove the record
+
+
+  rc = this->remove_record(key);
+  if (rc != 0)
+    return rc;
+
+  // and rewrite status
+
+  this->status.present--;
+  if (r.retrieved == 0) 
+    this->status.present0--;
+
+  rc = this->put_status();
+  if (rc != 0)
+    return rc;
+
+  return 0;
+}
+
+bool stopos_pool::is_statusrecord(const std::string &k)
+{
+  return stopos_key::KeyToNumber(k) == 0;
+}
+
+void stopos_pool::set_kvp(bool v)
+{
+  this->kvp = v;
+}
+
+bool stopos_pool::get_kvp(void)
+{
+  return this->kvp;
+}
+
+int stopos_pool::get_counts(longuint &count, 
+	                    longuint &present, 
+	                    longuint &present0)
+{
+  int rc = get_status();
+  if (rc != 0) 
+    return rc;
+
+  count    = status.count;
+  present  = status.present;
+  present0 = status.present0;
+  return 0;
+}
+
+int stopos_pool::dump_line(std::string  &line, 
+                           longuint     &retrieved, 
+			   std::string  &key)
+{
+  if (!this->db_open)
+    return DBNOTOPEN;
+
+  int rc;
+  rc = this->get_status();
+  if (rc != 0)
+    return FETCHERROR;
+
+  //
+  // if status.nextd is pointing to nothing,
+  // we assume that one is starting the dump,
+  // and let nextd point to the first record
+  //
+
+  if (this->status.nextd == this->statuskey)
+    this->status.nextd = this->status.first;
+
+  // if nextd is still pointing to nothing,
+  // there is nothing
+
+  if (this->status.nextd == this->statuskey)
+    return NOTFOUND;
+
+  datarecord r;
+
+  rc = this->get_record(r,this->status.nextd);
+  if (rc != 0)
+  {
+    this->status.nextd = this->status.first;
+    this->put_status();
+    return NOTFOUND;
+  }
+
+  key                = this->status.nextd.str();
+
+  retrieved          = r.retrieved;
+
+  line               = zhextostr(r.value);
+
+  this->status.nextd = r.next;
+
+  if (this->status.nextd == this->statuskey) // we signal that we reached eoi
+    this->status.nextd.impossible();         // at the next call
+
+  this->put_status();
+
+  return 0;
+}
+ 
+datarecord::datarecord()
+{
+  this->retrieved = 0;
+}
+
+datarecord::datarecord(const std::string &s,
+                       char              sep)
+{
+  std::vector <std::string> v;
+  split(v,s,sep);
+
+  this->prev      = stopos_key(v[0]);
+  this->next      = stopos_key(v[1]);
+  this->retrieved = StringToNumber<longuint>(v[2]);
+  this->value     = v[3];
+}
+
+std::string datarecord::str(char sep) const
+{
+  std::string r;
+  r = this->prev.str()                + sep
+    + this->next.str()                + sep
+    + NumberToString(this->retrieved) + sep
+    + this->value
+    ;
+
+  return r;
+}
+
+statusrecord::statusrecord(void)
+{
+  this->init();
+}
+
+statusrecord::statusrecord(const std::string &s, 
+                           char              sep)
+{
+  std:: vector <std::string> v;
+
+  split(v,s,sep);
+  this->next     = stopos_key(v[0]);
+  this->first    = stopos_key(v[1]);
+  this->last     = stopos_key(v[2]);
+  this->nextd    = stopos_key(v[3]);
+  this->count    = StringToNumber<longuint>(v[4]);
+  this->present  = StringToNumber<longuint>(v[5]);
+  this->present0 = StringToNumber<longuint>(v[6]);
+  this->extra    = v[7];
+
+}
+void statusrecord::init(void)
+{
+  this->next     = stopos_key();
+  this->first    = stopos_key();
+  this->last     = stopos_key();
+  this->nextd    = stopos_key();
+  this->count    = 0;
+  this->present  = 0;
+  this->present0 = 0;
+  // note: extra is not initialized here, that is
+  // something for a derived class
+}
+
+std::string statusrecord::str(char sep)
+{
+  std::string r;
+  r = this->next.str()               + sep
+    + this->first.str()              + sep
+    + this->last.str()               + sep
+    + this->nextd.str()              + sep
+    + NumberToString(this->count)    + sep
+    + NumberToString(this->present)  + sep
+    + NumberToString(this->present0) + sep
+    + this->extra
+    ;
+  return r;
+}
+
Index: /tags/0.92/stopos_pool.h
===================================================================
--- /tags/0.92/stopos_pool.h	(revision 19)
+++ /tags/0.92/stopos_pool.h	(revision 19)
@@ -0,0 +1,216 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+//
+// stopos_pool is the base class responsible for the logic
+// behind adding, removing etc lines in the database.
+//
+// The access to the database itself is the task for the derived
+// class.
+//
+//  In the following: a key is unique, never will two records
+//  have the same slot in the same databse between two calls
+//  of create_db or betwen create_db and purge_db.
+//
+// The derived class must provide the following functions:
+//
+//  std::string key_to_slot(const std::string &k)
+//    given a key k, the function should return
+//    a slot, where a record with key k can be
+//    stored. If the database does not need a
+//    slot (mysql, gdbm), then return "".
+//    This slot will be used in subsequent put_record
+//    and get_record calls.
+//
+//  int get_record(std::string &r,
+//                         const std::string &k, 
+//                         const std::string &l)
+//    This function must, given the key k, and the slot l,
+//      return the record in r. Return value = 0 if 
+//      everything ok.
+//
+//
+//   int put_record(const std::string &r, 
+//	            const std::string &k, 
+//		    const std::string &l)
+//     This function shoud store record r, given key k
+//     and slot l. The slot may be ignored.
+//
+//   int remove_record(const std::string &k, 
+//	               const std::string &l)
+//     This function should remove the record given key k
+//       and slot l.
+//
+//
+//   int create_db(const std::string &dbname)
+//     This function should create the database. dbname is
+//     a string containing the filename, tablename or
+//     whatever. The caller (stoposserver) should know about the
+//     issues involved.
+//
+//  int purge_db(void)
+//    This function has to remove the database, the name
+//    is in this->db_name  (char *) and
+//          this->sdb_name  (std::string)
+//
+//  int open_db(void)
+//    This function has to open the database, such that 
+//    subsequent put_record, get_record, remove_record
+//    calls can function.
+//    See purge() above for the name of the database.
+//
+//  int close_db(void)
+//    This function is called when database operations
+//    are done, normally it should close the database.
+//    See purge() above for the name of the database.
+//
+// Derived classes have no knowledge of the structure of
+// the records, and can add extra information if so
+// desired, as long as a retrieved record is equal
+// to a stored record with the same key/slot combination.
+//
+// Also the "status record" is for usage of the base class only,
+// with one exception: the field "extra" i.e. this->status->extra
+// is not used by the base class and can be used by derived classes
+// at will.
+//
+
+#ifndef STOPOS_POOL_H
+#define STOPOS_POOL_H
+#include <string>
+#include "stopos.h"
+#include "stopos_key.h"
+#include <iostream>
+ 
+class datarecord
+{
+  public:
+    datarecord();
+    datarecord(const std::string &s,char sep);
+    stopos_key prev;
+    stopos_key next;
+    longuint retrieved;
+    std::string value;
+    std::string str(char dbsep) const;
+};
+
+class statusrecord
+{
+  public:
+  stopos_key next;      // points to first line for the 'next' command
+  stopos_key first;     // points to first record
+  stopos_key last;      // points to last record
+  stopos_key nextd;     // points to record to be dumped
+  longuint count;       // total number of records added
+  longuint present;     // total number of records now availabe
+  longuint present0;    // ditto, but uncommitted
+  std::string extra;    // string that can be used by derived class
+
+  statusrecord(void);
+  statusrecord(const std::string &s,char dbsep);
+  std::string str(char dbsep);
+  void init(void);
+};
+
+class stopos_pool
+{
+  protected:
+    char* db_name;        // name for database
+    std::string sdb_name; // idem
+    static const unsigned int reclen = 4096;
+    bool kvp; //true if Koot-Vermin paradigm is to be used
+    bool db_open;
+    
+    stopos_key statuskey;
+    stopos_key nextkey;
+    statusrecord status;
+
+    virtual std::string key_to_slot(const std::string &k) = 0;
+    virtual int create_db(const std::string &dbname)      = 0;
+    virtual int get_record(std::string &r,       
+	                   const std::string &k, 
+			   const std::string &l)          = 0;
+    virtual int put_record(const std::string &r, 
+	                   const std::string &k, 
+			   const std::string &l)          = 0;
+    virtual int remove_record(const std::string &k, 
+	                      const std::string &l)       = 0;
+
+    int put_status(void);
+    int get_status(void);
+
+    // the function add_line calls this to get a slot
+    // from the derived class.  If no slot is necessary,
+    // this function may return "".
+    // The subsequent put_record call will have the return
+    // value of this function as parameter.
+
+    int get_record(datarecord &r,const stopos_key &k);
+    int put_record(const datarecord &r,const stopos_key &k);
+    int remove_record(const stopos_key &k);
+
+    std::string whoami;
+
+  public:
+
+    stopos_pool();
+    ~stopos_pool();
+
+    virtual int purge_db(void)  = 0;
+    virtual int open_db(void)   = 0;
+    virtual int close_db(void)  = 0;
+
+
+    int create_db(void);
+
+    virtual void set_db_name(std::string filename);
+    virtual std::string get_db_name(void);
+
+    virtual int add_line(const std::string line,std::string &key);
+    virtual int get_line(std::string &line,longuint &retrieved,std::string &key);
+    virtual int remove_line(const std::string &key);
+
+    virtual void set_kvp(const bool v);
+    virtual bool get_kvp(void);
+
+    int get_counts(longuint &count, 
+		   longuint &present, 
+		   longuint &present0);
+
+    int dump_line(std::string  &line,
+	          longuint     &retrieved,
+		  std::string  &key);
+
+    bool is_statusrecord(const std::string &k);
+
+    const static int ERROR             =  1; 
+    const static int NOFILENAME        =  2;
+    const static int OPENERROR         =  3;
+    const static int STOREERROR        =  4;
+    const static int FETCHERROR        =  5;
+    const static int NOTFOUND          =  6;
+    const static int NOTIMPLEMENTED    =  7;
+    const static int REMOVEERROR       =  8;
+    const static int LOCKERROR         =  9;
+    const static int CLOSEERROR        = 10;
+    const static int UNLOCKERROR       = 11;
+    const static int DBNOTOPEN         = 12;
+    const static int CREATEERROR       = 13;
+    const static int PURGEERROR        = 14;
+    const static int RECORDTOOLONG     = 15;
+
+};
+#endif
Index: /tags/0.92/stoposclient.cpp
===================================================================
--- /tags/0.92/stoposclient.cpp	(revision 19)
+++ /tags/0.92/stoposclient.cpp	(revision 19)
@@ -0,0 +1,469 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#include <iostream>
+#include <vector>
+#include <stdlib.h>
+#include <getopt.h>
+#include <fstream>
+#include "wrequest.h"
+#include "shellenv.h"
+#include "stopos.h"
+
+struct returnvalue
+{
+  std::string msg;
+  std::string key;
+  std::string comm;
+  std::string count;
+  std::string present;
+  std::string present0;
+  std::string value;
+
+  void parse(std::string &s)
+  {
+    // we expect somewhere in the output of the server a line like
+    // STOPOS:/OK/123/beautiful
+    // OK        -> msg
+    // 123       -> key
+    // 1         -> comm
+    // beautiful -> value
+    //
+    // OK, 123 and beautiful are hex coded
+    this->msg       = "";
+    this->key       = "";
+    this->comm      = "0";
+    this->count     = "0";
+    this->present   = "0";
+    this->present0  = "0";
+    this->value     = "";
+
+    size_t p = s.find("STOPOS:");
+    if (p == s.npos)
+    {
+      this->msg="Error: stopos server not running.";
+      return;
+    }
+
+    size_t q = s.find('\n',p+1);
+    std::string l = s.substr(p,q-p);
+    std::vector <std::string> v;
+
+    split(v,l,outsep);
+    for (unsigned int i=1; i<v.size(); i++)
+    {
+      switch (i)
+      {
+	case 1: this->msg       = zhextostr(v[i]); break;
+	case 2: this->key       = zhextostr(v[i]); break;
+	case 3: this->comm      = zhextostr(v[i]); break;
+	case 4: this->count     = zhextostr(v[i]); break;
+	case 5: this->present   = zhextostr(v[i]); break;
+	case 6: this->present0  = zhextostr(v[i]); break;
+	case 7: this->value     = zhextostr(v[i]); break;
+      }
+      // special case for key: if key == "", we try to take
+      // it from the environment.
+      if (trim(this->key) == "")
+	this->key = envtostr("STOPOS_KEY");
+    }
+  }
+};
+
+void usage()
+{
+  std::cerr << "Usage:" << std::endl;
+
+  std::cerr << program_name << " -h,--help"                           << std::endl;
+  std::cerr << program_name << " -v,--version"                        << std::endl;
+  std::cerr << program_name << " create [-p,--pool POOL]"             << std::endl;
+  std::cerr << program_name << " purge  [-p,--pool POOL]"             << std::endl;
+  std::cerr << program_name << " pools"                               << std::endl;
+  std::cerr << program_name << " add    [-p,--pool POOL] [FILENAME]"  << std::endl;
+  std::cerr << program_name << " next   [-p,--pool POOL] [-m,--mult]" << std::endl;
+  std::cerr << program_name << " remove [-p,--pool POOL] [KEY]"       << std::endl;
+  std::cerr << program_name << " status [-p,--pool POOL]"             << std::endl;
+  std::cerr << program_name << " dump   [-p,--pool POOL]"             << std::endl;
+  std::cerr << " -q,--quiet added to any command will reduce the ouput" << std::endl;
+  std::cerr << "Defaults:"                                            << std::endl;
+  std::cerr << "  POOL     : pool"                                    << std::endl;
+  std::cerr << "  FILENAME : standard input"                          << std::endl;
+
+  std::cerr << "Environment:"<<std::endl;
+  std::cerr << "   Defined by user:"<<std::endl;
+  std::cerr << "     STOPOS_POOL    : name of pool"<<std::endl;
+  std::cerr << "     STOPOS_KEY     : key"<<std::endl;
+  std::cerr << "   Defined by "<<program_name<<":"<<std::endl;
+  std::cerr << "     STOPOS_RC      : return message of all commands. OK == good" << std::endl;
+  std::cerr << "     STOPOS_VALUE   : result of 'next' or 'pools'"<<std::endl;
+  std::cerr << "     STOPOS_KEY     : result of 'next'"<<std::endl;
+  std::cerr << "     STOPOS_COUNT   : result of 'status'"<<std::endl;
+  std::cerr << "     STOPOS_PRESENT : result of 'status'"<<std::endl;
+  std::cerr << "     STOPOS_PRESENT0: result of 'status'"<<std::endl;
+
+
+
+}
+
+int handle_add(wrequest &w,const std::string &fname,returnvalue &r, const bool quiet)
+{
+  size_t number_of_lines;
+
+  bool read_from_stdin = 0;
+  int rc;
+  std::ifstream inp;
+  std::istream *pinp;
+  if(fname == "")  // read from stdin
+  {
+    read_from_stdin = 1;
+    pinp = &std::cin;
+  }
+  else
+  {
+    inp.open(fname.c_str());
+    if (!inp.is_open())
+    {
+      r.msg = "ERROR: Cannot open '"+fname+"'";
+      return 1;
+    }
+
+    number_of_lines = count_lines(inp);
+    pinp = &inp;
+  }
+  std::string line;
+  size_t n = 0;
+  w.set_command("add");
+  std::string body,header;
+
+  progress_bar bar;
+
+  while (getline(*pinp,line))
+  {
+    if (! quiet)
+    {
+      bool c;
+      std::string s;
+      if (read_from_stdin)
+	s = bar.show(n,c);
+      else
+	s = bar.show(n,(double)n/number_of_lines,c);
+      if (c)
+	std::cerr << '\r' << s;
+    }
+    if (line.size() > maxlinelength)
+    {
+      r.msg =  "Error: line length exeeds "+ NumberToString(maxlinelength);
+      break;
+    }
+    w.set_value(line);
+
+    rc = w.send_message(body,header);
+
+    if (rc != 0)
+    {
+      r.msg = "ERROR: Cannot connect to server";
+      return 1;
+    }
+    r.parse(body);
+    if (r.msg != "OK")
+      break;
+    n++;
+  }
+  if (! quiet)
+  {
+    bar.clear();
+    if (read_from_stdin)
+      std::cerr << '\r' << bar.show(n) << std::endl;
+    else
+      std::cerr << '\r' << bar.show(n,1.0) << std::endl;
+
+    std::cerr << program_name << ": Number of lines added = "<<n<<std::endl;
+  }
+  if (r.msg != "OK")
+    return 1;
+  return 0;
+}
+
+int main(int argc, char*argv[])
+{
+  wrequest w;
+  
+  int rc;
+
+  rc = w.read_config();
+  if (rc != 0 || w.get_config().size() != w.config_size)
+  {
+    std::cerr << "No valid config file found, creating it ..." << std::endl;
+    rc = w.write_config();
+    rc = rc | w.read_config();
+    if (rc != 0)
+    {
+      std::cerr << "Cannot write config file, exiting." << std::endl;
+      return 1;
+    }
+  }
+
+  bool addflag=0;
+  std::string inputfilename;
+  std::vector <std::string> parms;
+
+  static struct option long_options[] =
+  {
+    {"help",     0, 0, 'h'},
+    {"pool",     1, 0, 'p'},
+    {"version",  0, 0, 'v'},
+    {"multi",    0, 0, 'm'},
+    {"quiet",    0, 0, 'q'},
+    {0,          0, 0, 0}
+  };
+
+  // determine name of pool, in case it is not set on commandline
+
+  std::string pool = default_pool;
+  std::string p = envtostr("STOPOS_POOL");
+
+  if (p != "")
+    pool = p;
+
+  if (trim(w.get_pool()) != "")
+    pool = w.get_pool();
+
+  bool quiet = 0;
+
+  while(1)
+  {
+    int c = getopt_long(argc,argv,"-hp:vmq",long_options,0);
+    if (c == -1)
+    {
+      if (optind < argc)  // unexpected extra argument
+      {
+	std::cerr << program_name << ": Unexpected argument:" << argv[optind] << std::endl;
+	return 1;
+      }
+      break;
+    }
+    if ( c == '?' || c == ':' )
+    {
+      std::cerr << program_name << ": Invalid parameter, try "<< program_name<<" --help " <<std::endl;
+      return 1;
+    }
+    switch(c)
+    {
+      case 'h':
+	usage();
+	return 0;
+	break;
+      case 'p':
+	pool = optarg;
+	break;
+      case 'f':
+	inputfilename = optarg;
+	break;
+      case 'v':
+	std::cerr << program_version << std::endl;
+	return 0;
+	break;
+      case 'm':
+	w.set_multi("yes");
+	break;
+      case 'q':
+	quiet = 1;
+	break;
+      case 1:
+	parms.push_back(optarg);
+    }
+  }
+
+  w.set_pool(pool);
+  std::string serverurl=envtostr("STOPOS_SERVER_URL");
+  if (serverurl == "")
+    serverurl = SERVER_URL;
+  w.set_serverurl(serverurl);
+#ifdef DEFAULT_FLAT
+  std::string prot = "flat";
+#endif
+#ifdef DEFAULT_GDBM
+  std::string prot = "gdbm";
+#endif
+#ifdef DEFAULT_FILES
+  std::string prot = "files";
+#endif
+#ifdef DEFAULT_MYSQL
+  std::string prot = "mysql";
+#endif
+  std::string protenv;
+  protenv = envtostr("STOPOS_PROT");
+  if (protenv.size() !=0)
+    prot = protenv;
+  w.set_protocol(prot);
+  w.set_whoami(envtostr("USER"));
+
+  rc = 0;
+  returnvalue r;
+
+  if (parms.size() == 0)
+  {
+    std::cerr << program_name << ": No command given." << std::endl;
+    return 1;
+  }
+
+  std::string command = parms[0];
+
+  // only the remove and add commands can accept an extra parameter
+  //
+
+  if (parms.size() >1)
+    if (command !="remove" && command != "add")
+    {
+      std::cerr << program_name << ": Extraneous parameter:'"<<parms[1]<<"'."<<std::endl;
+      return 1;
+    }
+
+  bool count_set     = 0;
+  bool value_set     = 0;
+  bool committed_set = 0;
+  bool present_set   = 0;
+  bool present0_set  = 0;
+  bool key_set       = 0;
+  bool rc_set        = 0;
+
+  if(command == "create")
+  {
+    rc_set = 1;
+    w.set_command("create");
+  }
+  else if(command == "purge")
+  {
+    rc_set = 1;
+    w.set_command("purge");
+  }
+  else if(command == "status")
+  {
+    rc_set       = 1;
+    present_set  = 1;
+    present0_set = 1;
+    count_set    = 1;
+    w.set_command("status");
+  }
+  else if(command == "next")
+  {
+    rc_set        = 1;
+    value_set     = 1;
+    key_set       = 1;
+    committed_set = 1;
+    w.set_command("next");
+  }
+  else if(command == "dump")
+  {
+    rc_set        = 1;
+    value_set     = 1;
+    key_set       = 1;
+    committed_set = 1;
+    w.set_command("dump");
+  }
+  else if(command == "pools")
+  {
+    rc_set    = 1;
+    value_set = 1;
+    w.set_command("pools");
+  }
+  else if(command == "remove")
+  {
+    rc_set    = 1;
+    w.set_command("remove");
+    if (parms.size() < 2)
+    {
+      std::string a=envtostr("STOPOS_KEY");
+      if (a == "")
+      {
+	std::cerr << program_name << ": No key found, exit.\n";
+	return 1;
+      }
+      w.set_value(a);
+    }
+    else
+      w.set_value(parms[1]);
+  }
+  else if(command == "add")
+  {
+    rc_set  = 1;
+    key_set = 1;
+    addflag = 1;
+  }
+  else
+  {
+    std::cerr << program_name << ": Invalid command: '"<<command<<"'"<<std::endl;
+    return 1;
+  }
+
+  if(addflag)
+  {
+    if (parms.size() > 1)
+      inputfilename = parms[1];
+    rc |= handle_add(w,inputfilename,r,quiet);
+  }
+  else
+  {
+    std::string body,header;
+
+    rc = w.send_message(body,header);
+
+    if (rc != 0)
+      r.msg = "ERROR: Cannot connect to server";
+    else
+      r.parse(body);
+  }
+
+  int shelltype;
+
+  char *shell = getenv("SHELL");
+  std::string sh;
+  if (shell == 0)
+    sh = "bash";
+  else
+    sh = shell;
+
+  if (sh.find("csh") != sh.npos)
+    shelltype=SHELL_CSH;
+  else
+    shelltype = SHELL_SH;
+
+  if(rc_set)        std::cout << shellenv(r.msg,      "STOPOS_RC",        shelltype);
+  if(key_set)       std::cout << shellenv(r.key,      "STOPOS_KEY",       shelltype);
+  if(committed_set) std::cout << shellenv(r.comm,     "STOPOS_COMMITTED", shelltype);
+  if(count_set)     std::cout << shellenv(r.count,    "STOPOS_COUNT",     shelltype);
+  if(present_set)   std::cout << shellenv(r.present,  "STOPOS_PRESENT",   shelltype);
+  if(present0_set)  std::cout << shellenv(r.present0, "STOPOS_PRESENT0",  shelltype);
+  if(value_set)     std::cout << shellenv(r.value,    "STOPOS_VALUE",     shelltype);
+
+  if (command == "status" && r.msg == "OK" && ! quiet)
+  {
+    std::cerr << "lines added:           " << r.count    <<std::endl;
+    std::cerr << "lines present:         " << r.present  <<std::endl;
+    std::cerr << "lines never committed: " << r.present0 << std::endl;
+  }
+  else if (command == "pools" && ! quiet)
+  {
+    if(r.value.size()>0) std::cerr << r.value << std::endl;
+  }
+  if (r.msg != "OK" && ! quiet)
+  {
+    std::cerr << program_name<<": "<<r.msg << std::endl;
+    rc = 1;
+  }
+
+  return rc;
+}
Index: /tags/0.92/stoposdump
===================================================================
--- /tags/0.92/stoposdump	(revision 19)
+++ /tags/0.92/stoposdump	(revision 19)
@@ -0,0 +1,45 @@
+#!/bin/bash
+###
+#   Copyright 2013 Willem Vermin, SURFsara
+#
+#   Licensed under the Apache License, Version 2.0 (the "License");
+#   you may not use this file except in compliance with the License.
+#   You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
+###
+usage()
+{
+  echo "Usage: $0 [-p pool]"
+}
+if [ "$1" = "-h" ]; then
+  usage
+  exit 0
+fi
+if [ "$1" = "-p" ]; then
+  shift
+  pool="$1"
+  shift
+fi
+if [ -n "$1" ]; then
+  usage
+  exit 1
+fi
+
+while true ; do
+  if [ -n "$pool" ] ; then
+    eval "`stoposclient -p "$pool" dump 2>/dev/null`"
+  else
+    eval "`stoposclient dump 2>/dev/null`"
+  fi
+  if [ "$STOPOS_RC" != "OK" ] ; then
+    break
+  fi
+  printf '%s\t%s\t%s\n' "$STOPOS_KEY" "$STOPOS_COMMITTED" "$STOPOS_VALUE"
+done
Index: /tags/0.92/stopospools
===================================================================
--- /tags/0.92/stopospools	(revision 19)
+++ /tags/0.92/stopospools	(revision 19)
@@ -0,0 +1,18 @@
+#!/bin/bash
+###
+#   Copyright 2013 Willem Vermin, SURFsara
+#
+#   Licensed under the Apache License, Version 2.0 (the "License");
+#   you may not use this file except in compliance with the License.
+#   You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
+###
+eval "`./stoposclient pools 2>/dev/null`"
+echo $STOPOS_VALUE
Index: /tags/0.92/stoposserver.cpp
===================================================================
--- /tags/0.92/stoposserver.cpp	(revision 19)
+++ /tags/0.92/stoposserver.cpp	(revision 19)
@@ -0,0 +1,441 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#include <iostream>
+#include <stdlib.h>
+#include <vector>
+#include <string>
+#include "stopos.h"
+#include "wutils.h"
+#include "stopos_pool.h"
+#ifdef MAKE_GDBM
+#include "gdbm_pool.h"
+#endif
+#ifdef MAKE_FLAT
+#include "flatfile_pool.h"
+#endif
+#ifdef MAKE_FILES
+#include "files_pool.h"
+#endif
+#ifdef MAKE_MYSQL
+#include "mysql_pool.h"
+#endif
+
+void output(const std::string &return_message, const std::string &key,
+             longuint committed, 
+	     longuint count,
+	     longuint present,
+	     longuint present0,
+	     const std::string &return_value)
+{
+  std::cout << "STOPOS:"                            << outsep <<
+               zstrtohex(return_message)            << outsep <<
+	       zstrtohex(key)                       << outsep <<
+	       zstrtohex(NumberToString(committed)) << outsep <<
+	       zstrtohex(NumberToString(count))     << outsep <<
+	       zstrtohex(NumberToString(present))   << outsep <<
+	       zstrtohex(NumberToString(present0))  << outsep <<
+	       zstrtohex(return_value)              << 
+	       std::endl;
+}
+
+void output(const std::string &msg)
+{
+  output(msg,"",0,0,0,0,"");
+}
+struct commandline
+{
+  std::string header;
+  std::string prot;
+  std::string id;
+  std::string pool;
+  std::string command;
+  std::string multi;
+  std::string value;
+  std::string full_dbname;
+  std::string return_value;
+  stopos_pool *handler;
+
+  int create_full_dbname(void)
+  {
+    if (this->prot == "gdbm" ||
+	this->prot == "flat" ||
+	this->prot == "files")
+    {
+      std::string db_dir = DB_DIR;
+      if (db_dir == "" ) db_dir = "/tmp";
+      db_dir += "/";
+
+      this->full_dbname = db_dir                +
+	                  zstrtohex(this->id)   +
+			  "/"                   +
+			  zstrtohex(this->pool) +
+			  "_"                   +
+			  this->prot;
+    }
+    else if (this->prot == "mysql")
+    {
+      this->full_dbname = zstrtohex(this->id) +
+	                  "_"                 +
+			  zstrtohex(this->pool);
+    }
+    else
+      return 1;
+
+    if (this->command != "pools")
+      this->handler->set_db_name(this->full_dbname);
+    return 0;
+  }
+
+  int create_handler()
+  {
+#ifdef MAKE_GDBM
+    if (this->prot == "gdbm")
+    {
+      this->handler = new gdbm_pool();
+    }
+#endif
+#ifdef MAKE_FLAT
+    else if (this->prot == "flat")
+    {
+      this->handler = new flatfile_pool();
+    }
+#endif
+#ifdef MAKE_FILES
+    else if (this->prot == "files")
+    {
+      this->handler = new files_pool();
+    }
+#endif
+#ifdef MAKE_MYSQL
+    else if (this->prot == "mysql")
+    {
+      this->handler = new mysql_pool();
+    }
+#endif
+    else
+    {
+      this->return_value = "ERROR: Unknown handler:" + this->prot;
+      return 1;
+    }
+    return 0;
+  }
+
+  int init()
+  {
+
+    int rc;
+    if (this->command != "pools")
+    {
+      rc = this->create_handler();
+      if (rc != 0) 
+	return rc;
+      if (this->multi == "yes")
+	this->handler->set_kvp(1);
+      else
+	this->handler->set_kvp(0);
+    }
+    rc = this->create_full_dbname();
+    return rc;
+  }
+
+  int create_db()
+  {
+    int rc = this->handler->create_db();
+    return rc;
+  }
+
+  int purge_db()
+  {
+    int rc = this->handler->purge_db();
+    return rc;
+  }
+
+  int add_line(std::string &key, std::string &r)
+  {
+    r = "";
+    int rc;
+    rc = this->handler->open_db();
+    if (rc != 0)
+      return rc;
+
+    rc = this->handler->add_line(this->value,key);
+    return rc;
+  }
+
+  int get_line(std::string &key, longuint &comm, std::string &r)
+  {
+    r = "";
+    int rc;
+    rc = this->handler->open_db();
+    if (rc != 0)
+      return rc;
+
+    rc = this->handler->get_line(this->return_value, comm, key);
+    return rc;
+  }
+
+  int dump_line(std::string &key, longuint &comm, std::string &r)
+  {
+    r = "";
+    int rc;
+    rc = this->handler->open_db();
+    if (rc != 0)
+      return rc;
+
+    rc = this->handler->dump_line(this->return_value, comm, key);
+    return rc;
+  }
+
+  int remove_line(std::string & rkey,std::string &key, std::string &r)
+  {
+    r = "";
+    int rc;
+    rc = this->handler->open_db();
+    if (rc != 0)
+      return rc;
+
+    rkey = key;
+    rc = this->handler->remove_line(key);
+    return rc;
+  }
+
+  int get_status(longuint &count, 
+                 longuint &present, 
+		 longuint &present0, 
+		 std::string &r)
+  {
+    r = "";
+    int rc;
+    rc = this->handler->open_db();
+    if (rc != 0)
+      return rc;
+
+    rc = this->handler->get_counts(count, present, present0);
+
+    return rc;
+  }
+};
+
+std::string sanitycheck(commandline &l)
+{
+  if (l.header != "stopos")
+    return "invalid header: '" + l.header + "'";
+
+  if (l.prot.find_first_of(' ') != l.prot.npos)
+    return "ERROR: No valid protocol given";
+
+  if(l.id.find_first_of(' ') != l.id.npos)
+    return "ERROR: No valid id given";
+
+  if(l.command != "pools")
+    if(l.pool.find_first_of(' ') != l.pool.npos)
+    {
+      return "ERROR: No valid pool given";
+    }
+  if(l.command.find_first_of(' ') != l.command.npos)
+  {
+    return "ERROR: No valid command given";
+  }
+  return "";
+}
+
+int parsecom(commandline &l,std::string &line)
+{
+  std::vector <std::string> v;
+  split(v,line,httpsep); 
+                       
+  for (unsigned int i =0; i<v.size(); i++)
+  {
+    switch(i)
+    {
+      case 0: l.header   = zhextostr(v[i]); break;
+      case 1: l.prot     = zhextostr(v[i]); break;
+      case 2: l.id       = zhextostr(v[i]); break;
+      case 3: l.pool     = zhextostr(v[i]); break;
+      case 4: l.command  = zhextostr(v[i]); break;
+      case 5: l.multi    = zhextostr(v[i]); break;
+      case 6: l.value    = zhextostr(v[i]); break;
+    }
+  }
+  return 0;
+}
+
+
+int executecom(commandline &l)
+{
+  int rc;
+  longuint committed = 0;
+  longuint count     = 0;
+  longuint present   = 0;
+  longuint present0  = 0;
+
+  std::string return_message;
+  std::string key;
+  rc = l.init();
+
+  if (rc != 0)
+  {
+    if (l.return_value.size() == 0)
+      output("ERROR: cannot initialize server software");
+    else
+      output(l.return_value);
+    return rc;
+  }
+
+  if (l.command == "create")
+  {
+    rc = l.create_db();
+    if (rc != 0)
+      return_message = "ERROR: cannot create pool "+l.pool;
+  }
+  else if (l.command == "purge")
+  {
+    rc = l.purge_db();
+  }
+  else if (l.command == "add")
+  {
+    rc = l.add_line(key,return_message);
+  }
+  else if (l.command == "next")
+  {
+    rc = l.get_line(key, committed,return_message);
+  }
+  else if (l.command == "dump")
+  {
+    rc = l.dump_line(key, committed,return_message);
+  }
+  else if (l.command == "remove")
+  {
+    rc = l.remove_line(key,l.value, return_message);
+  }
+  else if (l.command == "status")
+  {
+    rc = l.get_status(count,present,present0,return_message);
+  }
+  else if (l.command == "pools")
+  {
+    if      (l.prot == "mysql")
+    {
+      l.return_value = "No pools for mysql";
+      rc = 0;
+    }
+    else if (l.prot == "files" ||
+	     l.prot == "gdbm"  ||
+	     l.prot == "flat"  )
+    {
+      // extract the directory from full_dbname:
+      size_t p = l.full_dbname.find_last_of('/');
+      if (p == l.full_dbname.npos)
+	rc = 1;
+      else
+      {
+        std::string dir = l.full_dbname.substr(0,p+1);
+	std::list <std::string> v;
+	get_dir_list(v,dir);
+	v.sort();
+
+	l.return_value = "";
+	for (std::list<std::string>::iterator it=v.begin(); it != v.end(); ++it)
+	{
+	  std::string f = *it;
+	  // 
+	  // if suffix ( eg: _files ) is found
+	  // this could be a valid file name
+	  //
+
+	  size_t sufsize = l.prot.size()+1;
+	  if (sufsize > f.size())
+	    continue;
+	  p = f.size() - sufsize;
+	  if (f.substr(p) != "_"+l.prot)
+	    continue;
+#if 0
+	  p = f.rfind("_"+l.prot);
+	  if (p == f.npos)
+	    continue;
+#endif
+	  f = f.substr(0,p);
+	  //
+	  // strip directory part
+	  //
+	  p = f.find_last_of('/');
+	  f = f.substr(p+1);
+	  l.return_value += zhextostr(f);
+	  l.return_value.push_back(' ');
+	}
+	if (l.return_value.size() > 0)
+	  l.return_value.erase(l.return_value.size()-1);
+      }
+      rc = 0;
+    }
+  }
+  else rc = 1;
+  if (rc == 0)
+    return_message="OK";
+  else
+    if (return_message == "")
+    {
+      switch (rc)
+      {
+	case stopos_pool::ERROR          : return_message = "ERROR";                                         break;
+	case stopos_pool::NOFILENAME     : return_message = "ERROR: No file name given";                     break;
+	case stopos_pool::OPENERROR      : return_message = "ERROR: Cannot find pool '"      + l.pool + "'"; break;
+	case stopos_pool::STOREERROR     : return_message = "ERROR: Cannot write to pool '"  + l.pool + "'"; break;
+	case stopos_pool::FETCHERROR     : return_message = "ERROR: Cannot read from pool '" + l.pool + "'"; break;
+	case stopos_pool::NOTFOUND       : return_message = "WARNING: Line not found";                       break;
+	case stopos_pool::NOTIMPLEMENTED : return_message = "ERROR: Feature not implemented";                break;
+	case stopos_pool::REMOVEERROR    : return_message = "ERROR: Cannot remove line";                     break;
+	case stopos_pool::LOCKERROR      : return_message = "ERROR: Cannot get lock";                        break;
+	case stopos_pool::CLOSEERROR     : return_message = "ERROR: Cannot close pool '"     + l.pool + "'"; break;
+	case stopos_pool::UNLOCKERROR    : return_message = "ERROR: Cannot unlock pool '"    + l.pool + "'"; break;
+	case stopos_pool::DBNOTOPEN      : return_message = "ERROR: Pool is not opened '"    + l.pool + "'"; break;
+	case stopos_pool::CREATEERROR    : return_message = "ERROR: Cannot create pool '"    + l.pool + "'"; break;
+	case stopos_pool::PURGEERROR     : return_message = "ERROR: Cannot purge pool '"     + l.pool + "'"; break;
+	case stopos_pool::RECORDTOOLONG  : return_message = "ERROR: Record too long";                        break;
+	default                          : return_message = "ERROR";                                         break;
+
+      }
+    }
+  
+
+  output(return_message, key, committed, count,
+         present, present0, l.return_value);
+  return rc;
+}
+
+int main()
+{
+  int rc;
+  std::cout << "Content-Type: text/plain\n\n";
+  std::string line;
+  line = envtostr("QUERY_STRING");
+  if (line.size() >0)
+    if (line[0] == '/')
+      line.erase(0,1);
+
+  commandline l;
+  parsecom(l,line);
+  std::string msg = sanitycheck(l);
+  if (msg != "")
+  {
+    output(msg);
+    return 1;
+  }
+  rc = executecom(l);
+
+  return rc;
+}
Index: /tags/0.92/stoposzztostr.cpp
===================================================================
--- /tags/0.92/stoposzztostr.cpp	(revision 19)
+++ /tags/0.92/stoposzztostr.cpp	(revision 19)
@@ -0,0 +1,56 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#include <iostream>
+#include <string>
+#include "wutils.h"
+int main()
+{
+  size_t p,q;
+  size_t size_begin = ZBEGIN_.size();
+  size_t size_end   = ZEND_.size();
+  std::string line;
+
+  while(std::getline(std::cin,line))
+  {
+    p = 0;
+    q = 0;
+    while(1)
+    {
+      q = line.find(ZBEGIN_,p);
+      std::cout <<line.substr(p,q-p);
+      //std::cout <<__LINE__<<":"<<line.substr(p,q-p)<<std::endl;
+      if ( q == line.npos)
+	break;
+      else
+      {
+	p = q;
+	q = line.find(ZEND_,p+size_begin);
+	if (q == line.npos)
+	{
+	  std::cout <<line.substr(p);
+	  break;
+	}
+	else
+	{
+	  std::cout <<zhextostr(line.substr(p,q-p+size_end));
+	  p = q+size_end;
+	}
+      }
+    }
+    std::cout << std::endl;
+  }
+}
Index: /tags/0.92/test1
===================================================================
--- /tags/0.92/test1	(revision 19)
+++ /tags/0.92/test1	(revision 19)
@@ -0,0 +1,8 @@
+#!/bin/bash
+while true ; do
+  stopos next -m
+  echo "$STOPOS_COMMITTED $STOPOS_VALUE"
+  if [ "$STOPOS_RC" != OK ]; then 
+    exit
+  fi
+done
Index: /tags/0.92/test2
===================================================================
--- /tags/0.92/test2	(revision 19)
+++ /tags/0.92/test2	(revision 19)
@@ -0,0 +1,12 @@
+#!/bin/bash
+while true ; do
+  stopos next 
+  echo "$STOPOS_COMMITTED $STOPOS_VALUE"
+  if [ "$STOPOS_RC" != OK ]; then 
+    exit
+  fi
+  stopos remove
+  if [ "$STOPOS_RC" != OK ]; then 
+    exit
+  fi
+done
Index: /tags/0.92/test_pool.cpp
===================================================================
--- /tags/0.92/test_pool.cpp	(revision 19)
+++ /tags/0.92/test_pool.cpp	(revision 19)
@@ -0,0 +1,740 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#include <iostream>
+#include <fstream>
+#include <string>
+#include <vector>
+#include <stdlib.h>
+#include "wutils.h"
+#include "clocks.h"
+#ifdef MAKE_GDBM
+#include "gdbm_pool.h"
+#endif
+#ifdef MAKE_FLAT
+#include "flatfile_pool.h"
+#endif
+#ifdef MAKE_FILES
+#include "files_pool.h"
+#endif
+#ifdef MAKE_MYSQL
+#include "mysql_pool.h"
+#endif
+#include <unistd.h>
+
+#include "stopos_key.h"
+int testkeysetget()
+{
+  stopos_key k;
+  longuint n = k.get<longuint>();
+  if (n != 0)
+  {
+    std::cerr << "Keytesterror new key get: "<< n <<std::endl;
+    return 1;
+  }
+
+  n = 1234567890123456;
+  k.set(n,"");
+  longuint l = k.get<longuint>();
+  if (l != n)
+  {
+    std::cerr << "Keytesterror expect: " << n<< ":" << " got:" << l << std::endl;
+    return 1;
+  }
+  return 0;
+}
+ 
+
+int testpool(bool openclose,int which)
+{
+#define OPENCLOSE \
+  if (openclose) \
+  {\
+     p->close_db();\
+     rc = p->open_db();\
+     if(rc)\
+     {\
+        std::cerr << __FILE__<<":"<<__LINE__<<": cannot open db:" << rc << std::endl;\
+        return rc;\
+     }\
+   }
+
+  stopos_pool *p;
+  std::string dbname;
+  switch (which)
+  {
+    case 1: dbname = "/tmp/testdb_gdbm";
+#ifdef MAKE_GDBM
+	    p = new gdbm_pool();
+#endif
+	    break;
+    case 2: dbname = "/tmp/testdb_flat";
+#ifdef MAKE_FLAT
+	    p = new flatfile_pool();
+#endif
+	    break;
+    case 3: dbname = "/tmp/testdb_dir";
+#ifdef MAKE_FILES
+	    p = new files_pool();
+#endif
+	    break;
+    case 4: dbname = "testdb";
+#ifdef MAKE_MYSQL
+	    p = new mysql_pool();
+#endif
+	    break;
+    default: std::cerr << __LINE__ << "INVALID WHICH" << std::endl;
+	     return 6;
+  }
+
+  std::vector <std::string> lines;
+  std::vector <std::string> keys;
+
+  std::string linesfile = "/tmp/linesfile";
+  std::string command;
+  command="man bash 2>/dev/null | head -n 100  > "+linesfile;
+  int rc;
+  rc = system(command.c_str());
+  if (rc != 0)
+    std::cerr << "Iets niet helemaal goed met man ... \n";
+  std::ifstream infile;
+  infile.open(linesfile.c_str());
+
+  int ntest = 0;
+  while (infile)
+  {
+    ntest++;
+    std::string line;
+    getline(infile,line);
+    lines.push_back(NumberToString(ntest)+":"+line);
+  }
+
+  p->set_db_name(dbname);
+  rc = p->create_db();
+  if (rc !=0)
+  {
+    std::cerr << __FILE__<<":"<<__LINE__<<": cannot create db:" << rc << std::endl;
+    return rc;
+  }
+  if (which ==0)
+  {
+    std::cerr << "Ja NU!\n";
+    sleep (5);
+    std::cerr << "continuing \n";
+  }
+  rc = p->open_db();
+  if (rc !=0)
+  {
+    std::cerr << __FILE__<<":"<<__LINE__<<": cannot open db\n";
+    return 7;
+  }
+
+  for (int i=0; i<ntest; i++)
+  {
+    keys.resize(0);
+    int rc;
+    std::string key;
+    OPENCLOSE
+    rc = p->add_line(lines[i],key);
+    if (rc!=0) 
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":put_line '"<<lines[i]<<"' number: " << i << ":" << rc << std::endl;
+      return rc;
+    }
+    keys.push_back(key);
+  }
+  p->close_db();
+  rc = p->open_db();
+  if (rc !=0)
+  {
+    std::cerr << __FILE__<<":"<<__LINE__<<": cannot open db\n";
+    return rc;
+  }
+  p->remove_line("4");
+  p->remove_line("2");
+  p->remove_line("3");
+  p->remove_line("1");
+  std::string line;
+  longuint m;
+  std::string getkey;
+  p->get_line(line,m,getkey);
+
+  p->create_db();
+  rc = p->open_db();
+  if (rc !=0)
+  {
+    std::cerr << __FILE__<<":"<<__LINE__<<": cannot open db\n";
+    return rc;
+  }
+
+  keys.resize(0);
+  for (int i=0; i<ntest; i++)
+  {
+    int rc;
+    std::string key;
+    OPENCLOSE
+    rc = p->add_line(lines[i],key);
+    if (rc!=0) 
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":add_line '"<<lines[i]<<"' number: " << i << std::endl;
+      return rc;
+    }
+    keys.push_back(key);
+  }
+
+  // try to read all lines:
+  p->close_db();
+  delete p;
+  switch (which)
+  {
+#ifdef MAKE_GDBM
+    case 1: p = new gdbm_pool();
+	    break;
+#endif
+#ifdef MAKE_FLAT
+    case 2: p = new flatfile_pool();
+	    break;
+#endif
+#ifdef MAKE_FILES
+    case 3: p = new files_pool();
+	    break;
+#endif
+#ifdef MAKE_MYSQL
+    case 4: p = new mysql_pool();
+	    break;
+#endif
+    default: std::cerr << __LINE__ << "INVALID WHICH" << std::endl;
+	     return 6;
+  }
+  p->set_db_name(dbname);
+  rc = p->open_db();
+  if (rc !=0)
+  {
+    std::cerr << __FILE__<<":"<<__LINE__<<": cannot open db:"<<rc<<"\n";
+    return rc;
+  }
+
+  for (int i=0; i<ntest; i++)
+  {
+    std::string line,getkey;
+    int rc;
+    longuint m;
+    OPENCLOSE
+    rc = p->get_line(line,m,getkey);
+
+    if (rc!=0) 
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line:"<<i<<std::endl;
+      return rc;
+    }
+    if (m != 0)
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line expected: 0 got:" << m <<std::endl;
+      return 9;
+    }
+    if (line != lines[i])
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line expected: '"<<lines[i]<<"' got:'" << line <<"'"<<std::endl;
+      return 9;
+    }
+  }
+
+  // test if no further lines can be read
+  for (int i=0; i<10; i++)
+  {
+    std::string line,getkey;
+    int rc;
+    longuint m;
+    OPENCLOSE
+    rc = p->get_line(line,m,getkey);
+    if (rc == 0)
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line expected: not 0 got:" << rc <<std::endl;
+      return 9;
+    }
+  }
+  p->close_db();
+  rc = p->open_db();
+  if (rc !=0)
+  {
+    std::cerr << __FILE__<<":"<<__LINE__<<": cannot open db\n";
+    return rc;
+  }
+
+  for (int i=0; i<ntest; i++)
+  {
+    int rc;
+    OPENCLOSE
+    rc = p->remove_line(keys[i]);
+    if (rc != 0)
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":cannot remove key " << "'"<< keys[i]<<"'" <<std::endl;
+      return 9;
+    }
+  }
+  p->close_db();
+  rc = p->open_db();
+  if (rc !=0)
+  {
+    std::cerr << __FILE__<<":"<<__LINE__<<": cannot open db\n";
+    return rc;
+  }
+  for (int i=0; i<ntest; i++)
+  {
+    int rc;
+    OPENCLOSE
+    rc = p->remove_line(keys[i]);
+    if (rc == 0)
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":can remove removed key " << "'"<< keys[i]<<"'" <<std::endl;
+      return 9;
+    }
+  }
+  p->close_db();
+  p->open_db();
+  p->set_kvp(0);
+
+  // typical use: fill database with lines,
+  // read and remove
+
+  {
+    for (int i=0; i<ntest; i++)
+    {
+      int rc;
+      std::string addkey;
+      OPENCLOSE
+      rc = p->add_line(lines[i],addkey);
+      if (rc!=0) 
+      {
+	p->close_db();
+	std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":put_line '"<<lines[i]<<"' number: " << i << ":"<<rc<<std::endl;
+	return rc;
+      }
+    }
+    for (int i=0; i<ntest; i++)
+    {
+      std::string line,getkey;
+      int rc;
+      longuint m;
+      OPENCLOSE
+      rc = p->get_line(line,m,getkey);
+      if (rc!=0) 
+      {
+	p->close_db();
+	std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line\n";
+	return rc;
+      }
+      if (m != 0)
+      {
+	p->close_db();
+	std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line expected: 0 got:" << m <<std::endl;
+	return 9;
+      }
+      if (line != lines[i])
+      {
+	p->close_db();
+	std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line expected: '"<<lines[i]<<"' got:'" << line <<"'"<<std::endl;
+	return 9;
+      }
+      OPENCLOSE
+      p->remove_line(getkey);
+      if (rc !=0)
+      {
+	p->close_db();
+	std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":cannot remove key " << "'"<< keys[i]<<"'" <<std::endl;
+	return 9;
+      }
+    }
+    // test if no further lines can be read
+    for (int i=0; i<10; i++)
+    {
+      std::string line,getkey;
+      int rc;
+      longuint m;
+      OPENCLOSE
+      rc = p->get_line(line,m,getkey);
+      if (rc == 0)
+      {
+	p->close_db();
+	std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line expected: not 0 got:" << rc <<std::endl;
+	return 9;
+      }
+    }
+  }
+  p->close_db();
+  p->open_db();
+  p->set_kvp(1);
+  keys.resize(0);
+  // kvp test
+  for (int i=0; i<ntest; i++)
+  {
+    int rc;
+    std::string key;
+    OPENCLOSE
+    rc = p->add_line(lines[i],key);
+    if (rc!=0) 
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":put_line '"<<lines[i]<<"' number: " << i << std::endl;
+      return rc;
+    }
+    keys.push_back(key);
+  }
+  for (int i=0; i<ntest*3; i++)
+  {
+    std::string line,getkey;
+    int rc;
+    longuint m;
+    OPENCLOSE
+    rc = p->get_line(line,m,getkey);
+    if (rc!=0) 
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line\n";
+      return rc;
+    }
+    if (m != (longuint)i/ntest)
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line expected: "<<i/ntest<<" got:" << m <<std::endl;
+      return 9;
+    }
+    if (line != lines[i%ntest])
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line expected: '"<<lines[i]<<"' got:'" << line <<"'"<<std::endl;
+      return 9;
+    }
+  }
+
+  p->close_db();
+  p->create_db();
+  p->open_db();
+
+  for (int i=0; i<10; i++)
+  {
+    std::string line,getkey;
+    int rc;
+    longuint m;
+    OPENCLOSE
+    rc = p->get_line(line,m,getkey);
+    if (rc == 0)
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line expected: not 0 got:" << rc << " line: '" << line << "'" <<std::endl;
+      return 9;
+    }
+  }
+
+  {
+    // test remove lines
+    p->set_kvp(0);
+    keys.resize(0);
+    for (int i=0; i<ntest; i++)
+    {
+      int rc;
+      std::string key;
+      OPENCLOSE
+      rc = p->add_line(lines[i],key);
+      if (rc!=0) 
+      {
+	p->close_db();
+	std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":put_line '"<<lines[i]<<"' number: " << i << std::endl;
+	return rc;
+      }
+      keys.push_back(key);
+    }
+    //remove half of lines
+    for (int i=0; i<ntest; i+=2)
+    {
+      OPENCLOSE
+      int rc = p->remove_line(keys[i]);
+      if (rc !=0)
+      {
+	p->close_db();
+	std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":cannot remove key " << "'"<< keys[i]<<"'" <<std::endl;
+	return 9;
+      }
+    }
+    // try to get other half
+    for (int i=1; i<ntest; i+=2)
+    {
+      int rc;
+      longuint m;
+      std::string line,getkey;
+      OPENCLOSE
+      rc = p->get_line(line,m,getkey);
+      if (rc!=0) 
+      {
+	p->close_db();
+	std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line\n";
+	return rc;
+      }
+      if (m != 0)
+      {
+	p->close_db();
+	std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line expected: 0 got:" << m <<std::endl;
+	return 9;
+      }
+      if (line != lines[i])
+      {
+	p->close_db();
+	std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line expected: '"<<lines[i]<<"' got:'" << line <<"'"<<std::endl;
+	return 9;
+      }
+    }
+    // there should be no lines left, check:
+    int nrem = 0;
+    longuint m; 
+    std::string line,getkey;
+    while (p->get_line(line,m,getkey)==0)
+      nrem++;
+
+    if (nrem != 0)
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line expected: not 0 got:" << nrem << " line: '" << line << "'" <<std::endl;
+      return 9;
+    }
+  }
+  p->close_db();
+  p->create_db();
+  p->open_db();
+  keys.resize(0);
+
+  {
+    // test the following
+    // after an add_line, the next line to be retrieved should be
+    // that line, unless there were already other lines not 
+    // retrieved yet
+    std::string line,key,getkey;
+    int rc;
+    for (int i=0; i<ntest/2; i++)
+    {
+      OPENCLOSE
+      p->add_line(lines[i],key);
+      keys.push_back(key);
+    }
+    longuint m;
+    int l=2;
+    for (int i=0; i<l; i++)
+    {
+      OPENCLOSE
+      p->get_line(line,m,getkey);
+    }
+    // the next line to be retrieved should be lines[l]
+    // first, we add a new line
+      OPENCLOSE
+    p->add_line(lines[ntest/2],key);
+    keys.push_back(key);
+    //check now which line we get:
+      OPENCLOSE
+    p->get_line(line,m,getkey);
+    if (line != lines[l])
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line expected: '"<<lines[l]<<"' got:'" << line <<"'"<<std::endl;
+      return 9;
+    }
+    // now retrieve all lines and add a new one. Can we retrieve that:
+    while (p->get_line(line,m,getkey) == 0);
+      OPENCLOSE
+    p->add_line(lines[0],key);
+      OPENCLOSE
+    rc = p->get_line(line,m,getkey);
+    if (rc != 0)
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line\n";
+      return rc;
+    }
+    if (line != lines[0])
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line expected: '"<<lines[0]<<"' got:'" << line <<"'"<<std::endl;
+      return 9;
+    }
+  }
+
+  p->create_db();
+  p->open_db();
+  {
+    std::string key,line,getkey;
+    longuint m;
+    int rc;
+    // some delete and insert tests
+    // put only one line in the database, and remove it
+    OPENCLOSE
+    rc = p->add_line(lines[0],key);
+    if (rc != 0)
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":add_line\n";
+      return rc;
+    }
+
+    OPENCLOSE
+    rc = p->remove_line(key);
+    if (rc != 0)
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":remove_line\n";
+      return rc;
+    }
+    OPENCLOSE
+    rc = p->get_line(line,m,getkey);
+    if (rc == 0)
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line" <<
+	"expected !0, got:"<<rc <<"\n";
+      return rc;
+    }
+    // add two lines, remove last one, we expect that get_line
+    // gets the first
+    OPENCLOSE
+    p->add_line(lines[0],key);
+    OPENCLOSE
+    p->add_line(lines[1],key);
+    OPENCLOSE
+    rc = p->remove_line(key);
+    if (rc != 0)
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":remove_line\n";
+      return rc;
+    }
+    OPENCLOSE
+    rc = p->get_line(line,m,getkey);
+    if (rc != 0)
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line\n";
+      return rc;
+    }
+    if (line != lines[0])
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line expected: '"<<lines[0]<<"' got:'" << line <<"'"<<std::endl;
+      return 9;
+    }
+    // add two lines, remove first one, we expect that get_line
+    // gets the last
+    p->close_db();
+    p->create_db();
+    p->open_db();
+    std::string key0;
+    OPENCLOSE
+    p->add_line(lines[0],key0);
+    OPENCLOSE
+    p->add_line(lines[1],key);
+    rc = p->remove_line(key0);
+    if (rc != 0)
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":remove_line\n";
+      return rc;
+    }
+    OPENCLOSE
+    rc = p->get_line(line,m,getkey);
+    if (rc != 0)
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line\n";
+      return rc;
+    }
+    if (line != lines[1])
+    {
+      p->close_db();
+      std::cerr << "Error in "<<__FILE__<<":"<<__LINE__<<":get_line expected: '"<<lines[0]<<"' got:'" << line <<"'"<<std::endl;
+      return 9;
+    }
+
+    // without testing: add ntest lines
+    for (int i=0; i<ntest; i++)
+    {
+      p->add_line(lines[i],key0);
+    }
+
+  }
+
+  p->close_db();
+  return 0;
+}
+int main()
+{
+  double t0;
+  bool openclose = 1;
+  std::cerr << "running keytest .... ";
+  if (testkeysetget()==0)
+    std::cerr << "ok\n";
+  else
+  {
+    std::cerr << "NOT OK\n";
+  }
+
+#ifdef MAKE_GDBM
+  std::cerr << "running gdbm ....... ";
+  t0=wallclock();
+  if (testpool(openclose,1) == 0)
+    std::cerr << "ok " << wallclock()-t0 << std::endl;
+  else
+  {
+    std::cerr << "NOT OK\n";
+  }
+#endif
+#ifdef MAKE_FLAT
+  std::cerr << "running flatfile ... ";
+  t0=wallclock();
+  if (testpool(openclose,2) == 0)
+    std::cerr << "ok " << wallclock()-t0 << std::endl;
+  else
+  {
+    std::cerr << "NOT OK\n";
+  }
+#endif
+#ifdef MAKE_FILES
+  std::cerr << "running files ...... ";
+  t0=wallclock();
+  if (testpool(openclose,3) == 0)
+    std::cerr << "ok " << wallclock()-t0 << std::endl;
+  else
+  {
+    std::cerr << "NOT OK\n";
+  }
+#endif
+#ifdef MAKE_MYSQL
+  std::cerr << "running mysql ...... ";
+  t0=wallclock();
+  if (testpool(openclose,4) == 0)
+    std::cerr << "ok " << wallclock() -t0 << std::endl;
+  else
+  {
+    std::cerr << "NOT OK\n";
+  }
+#endif
+  std::cerr << "all is well that ends well" << std::endl;
+  return 0;
+}
+
Index: /tags/0.92/watchit
===================================================================
--- /tags/0.92/watchit	(revision 19)
+++ /tags/0.92/watchit	(revision 19)
@@ -0,0 +1,21 @@
+#!/bin/bash
+###
+#   Copyright 2013 Willem Vermin, SURFsara
+#
+#   Licensed under the Apache License, Version 2.0 (the "License");
+#   you may not use this file except in compliance with the License.
+#   You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
+###
+reset
+#exe=./stoposclient
+#watch -n 1 " (rm $exe; make dep >/dev/null &&  make install && timeout 3 $exe -p willemspool create 2>&1) "
+exe=./testpool
+watch -n 0.1 " (rm $exe; make dep >/dev/null &&  make install && timeout 10 $exe 2>&1) "
Index: /tags/0.92/wrequest.cpp
===================================================================
--- /tags/0.92/wrequest.cpp	(revision 19)
+++ /tags/0.92/wrequest.cpp	(revision 19)
@@ -0,0 +1,139 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#include <iostream>
+#include <string>
+#include "stopos.h"
+#include "wrequest.h"
+#include <stdlib.h>
+#include <curl/curl.h>
+#include <fstream>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <errno.h>
+
+//constructor
+wrequest::wrequest(void)
+{
+  this->set_configfilename();
+  curl_global_init(CURL_GLOBAL_ALL);
+  this->curl = curl_easy_init();
+  this->debugging = 0;
+}
+
+//destructor
+wrequest::~wrequest()
+{
+  curl_easy_cleanup(curl);
+  curl_global_cleanup();
+}
+
+void wrequest::set_configfilename(void)
+{
+  char *h = getenv("HOME");
+  std::string home;
+  if (h == 0)
+    home = "/";
+  else
+    home = h;
+  this->configdirname = home + "/.stopos";
+  this->configfilename = this->configdirname + "/id";
+}
+
+int wrequest::read_config(void)
+{
+  int rc = get_file_contents(this->config,this->configfilename);
+  if (rc == 0)
+  {
+    size_t p= this->config.find_first_of('\n');
+    if (p != this->config.npos)
+      this->config.resize(p);
+    return 0;
+  }
+  else
+    return 1;
+}
+
+int wrequest::write_config(void)
+{
+  std::ofstream f;
+  int rc;
+
+  rc = mkdir(this->configdirname.c_str(),0700);
+  if (rc < 0)
+    if (errno != EEXIST)
+      return 1;
+  f.open(this->configfilename.c_str(),std::ios::out);
+  if (f.fail())
+    return 1;
+  this->config = gen_random(this->config_size)+'\n';
+  f << this->config;
+  f.close();
+  rc  = chmod(this->configfilename.c_str(),0600);
+  if (rc < 0)
+    return 1;
+  return 0;
+}
+
+
+size_t write_data(char *p, size_t size, size_t nmemb, std::string *userdata)
+{
+  size_t l = size*nmemb;
+  for (size_t i=0; i<l; i++)
+    (*userdata).push_back(p[i]);
+  return l;
+}
+
+int wrequest::set_serverurl(const std::string &u)
+{
+  this->serverurl = u + '?';
+  curl_easy_setopt(this->curl,CURLOPT_HEADER,0L);
+  curl_easy_setopt(this->curl,CURLOPT_NOPROGRESS, 1L);
+  curl_easy_setopt(this->curl,CURLOPT_WRITEFUNCTION, write_data);
+
+  return 0;
+}
+
+void wrequest::compose_message(std::string &m)
+{
+  m="";
+  m += zstrtohex("stopos")                                    + httpsep;
+  m += zstrtohex(this->message.get_protocol()               ) + httpsep;
+  m += zstrtohex(this->message.get_whoami()+"."+this->config) + httpsep;
+  m += zstrtohex(this->message.get_pool()                   ) + httpsep;
+  m += zstrtohex(this->message.get_command()                ) + httpsep;
+  m += zstrtohex(this->message.get_multi()                  ) + httpsep;
+  m += zstrtohex(this->message.get_value()                  );
+}
+
+int wrequest::send_message(std::string &body, std::string &header)
+{
+
+  body.clear();
+  header.clear();
+  std::string m;
+  this->compose_message(m);
+  std::string x;
+  x = this->serverurl+m;
+  curl_easy_setopt(this->curl,CURLOPT_URL,x.c_str());
+  curl_easy_setopt(this->curl,CURLOPT_WRITEDATA, &body);
+  curl_easy_setopt(this->curl,CURLOPT_WRITEHEADER, &header);
+  int rc;
+  rc = curl_easy_perform(this->curl);
+
+  return rc;
+}
+
Index: /tags/0.92/wrequest.h
===================================================================
--- /tags/0.92/wrequest.h	(revision 19)
+++ /tags/0.92/wrequest.h	(revision 19)
@@ -0,0 +1,129 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#ifndef WREQUEST_H
+#define WREQUEST_H
+#include <string>
+#include <iostream>
+#include <curl/curl.h>
+#include "wutils.h"
+
+class messageclass
+{
+  private:
+    std::string protocol;
+    std::string whoami;
+    std::string pool;
+    std::string command;
+    std::string multi;
+    std::string value;
+
+  public:
+  //constructor
+    messageclass()
+    {
+      this->protocol = " ";
+      this->whoami   = " ";
+      this->pool     = " ";
+      this->command  = " ";
+      this->multi    = " ";
+      this->value    = " ";
+    }
+    void set_protocol(const std::string &p){ this->protocol = p;    }
+    std::string get_protocol(void)         { return this->protocol; }
+
+    void set_whoami(const std::string &u)  { this->whoami = u;      }
+    std::string  get_whoami(void)          { return this->whoami;   }
+
+    void set_pool(const std::string &p)    { this->pool = p;        }
+    std::string  get_pool(void)            { return this->pool;     }
+
+    void set_command(const std::string &c) { this->command = c;     }
+    std::string  get_command(void)         { return this->command;  }
+
+    void set_multi(const std::string &m)   { this->multi = m;       }
+    std::string  get_multi(void)           { return this->multi;    }
+
+    void set_value(const std::string &v)   { this->value = v;       }
+    std::string  get_value(void)           { return this->value;    }
+
+};
+
+class wrequest
+{
+  private: 
+    CURL *curl;
+
+    std::string configfilename;
+    std::string configdirname;
+    std::string config;
+    void set_configfilename(void);
+
+    std::string serveraddress;
+    std::string serverurl;
+    int port;
+
+    messageclass message;
+    void compose_message(std::string &m);
+    int get_answer(void);
+    std::string data;
+    bool debugging;
+
+  public:
+    //constructor
+    wrequest(void);
+
+    //destructor
+    ~wrequest(void);
+
+    static const unsigned int config_size = 8;
+
+    std::string get_config(void)             { return this->config; };
+
+    int set_serverurl(const std::string &u);
+    std::string get_serverurl(void)          { return this->serverurl; }
+    
+    void set_port(int);
+    int  get_port(void);
+
+    int read_config(void);
+    int write_config(void);
+
+    void set_protocol(const std::string &p){ this->message.set_protocol(p);       }
+    std::string get_protocol(void)         { return this->message.get_protocol(); }
+
+    void set_whoami(const std::string &u)  { this->message.set_whoami(u);         }
+    std::string  get_whoami(void)          { return this->message.get_whoami();   }
+
+    void set_pool(const std::string &p)    { this->message.set_pool(p);           }
+    std::string  get_pool(void)            { return this->message.get_pool();     }
+
+    void set_command(const std::string &c) { this->message.set_command(c);        }
+    std::string  get_command(void)         { return this->message.get_command();  }
+
+    void set_multi(const std::string &m)   { this->message.set_multi(m);          }
+    std::string  get_multi(void)           { return this->message.get_multi();    }
+
+    void set_value(const std::string &v)   { this->message.set_value(v);          }
+    std::string  get_value(void)           { return this->message.get_value();    }
+
+    void set_debug(void)                   { this->debugging = 1; }
+    void unset_debug(void)                 { this->debugging = 0; }
+
+    int send_message(std::string &body,std::string &header);
+  
+};
+#endif
Index: /tags/0.92/wtd.h
===================================================================
--- /tags/0.92/wtd.h	(revision 19)
+++ /tags/0.92/wtd.h	(revision 19)
@@ -0,0 +1,20 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#ifndef WTD_H
+#define WTD_H
+#define WTD(x) std::cerr<<"WTD:"<<__FILE__<<":"<<__LINE__<<":"<< x <<std::endl
+#endif
Index: /tags/0.92/wutils.cpp
===================================================================
--- /tags/0.92/wutils.cpp	(revision 19)
+++ /tags/0.92/wutils.cpp	(revision 19)
@@ -0,0 +1,370 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#include <string>
+#include <vector>
+#include <string.h>
+#include <iostream>
+#include <algorithm>
+#include <sys/time.h>
+#include "wutils.h"
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <errno.h>
+#include <stdlib.h>
+#include "wtd.h"
+
+// 
+// splits string s to vector v, using separator character sep
+// return value: v
+// It seems that it follows the python implementation of split()
+//
+std::vector <std::string> split(std::vector <std::string> &v, const std::string s, const char sep)
+{
+  v.clear();
+  for (size_t p=0, q=0; q != s.npos; p = q + 1)
+    v.push_back(s.substr(p,(q=s.find(sep,p))-p));
+  return v;
+}
+
+// converts string to char *
+// resulting string is created using malloc. Use free to free.
+char* str_to_charp(const std::string s)
+{
+  return strdup(s.c_str());
+}
+
+/*
+ * converts string to hex representation
+ */
+std::string strtohex(const std::string s)
+{
+  std::string a;
+  const std::string c=
+  "000102030405060708090a0b0c0d0e0f"
+  "101112131415161718191a1b1c1d1e1f"
+  "202122232425262728292a2b2c2d2e2f"
+  "303132333435363738393a3b3c3d3e3f"
+  "404142434445464748494a4b4c4d4e4f"
+  "505152535455565758595a5b5c5d5e5f"
+  "606162636465666768696a6b6c6d6e6f"
+  "707172737475767778797a7b7c7d7e7f"
+  "808182838485868788898a8b8c8d8e8f"
+  "909192939495969798999a9b9c9d9e9f"
+  "a0a1a2a3a4a5a6a7a8a9aaabacadaeaf"
+  "b0b1b2b3b4b5b6b7b8b9babbbcbdbebf"
+  "c0c1c2c3c4c5c6c7c8c9cacbcccdcecf"
+  "d0d1d2d3d4d5d6d7d8d9dadbdcdddedf"
+  "e0e1e2e3e4e5e6e7e8e9eaebecedeeef"
+  "f0f1f2f3f4f5f6f7f8f9fafbfcfdfeff";
+
+  a.resize(2*s.size());
+  int k = 0;
+  for (unsigned int i=0; i<s.size(); i++)
+  {
+    int p  = 2*(unsigned char)s[i];
+    a[k++] = c[p++];
+    a[k++] = c[p];
+  }
+
+  return a;
+}
+
+/*
+ * convert hex represented string to string
+ * unknown hex numbers result in '?'
+ */
+std::string hextostr(const std:: string s)
+{
+const char c[] = {
+  -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
+  -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
+  -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
+   0, 1, 2, 3, 4, 5, 6, 7, 8, 9,-1,-1,-1,-1,-1,-1,
+  -1,10,11,12,13,14,15,-1,-1,-1,-1,-1,-1,-1,-1,-1,
+  -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
+  -1,10,11,12,13,14,15,-1,-1,-1,-1,-1,-1,-1,-1,-1,
+  -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
+  -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
+  -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
+  -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
+  -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
+  -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
+  -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
+  -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
+  -1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1
+               };
+
+  std::string a;
+  unsigned int l = s.size();
+  a.resize((l+1)/2);
+
+  unsigned int k=0;
+  unsigned int l2 = (l/2)*2;
+  for (unsigned int i=0; i<l2; i+=2)
+  {
+    int p = c[(unsigned int)s[i]];
+    int q = c[(unsigned int)s[i+1]];
+    if (p >= 0 && q >= 0)
+      a[k++] = 16*p + q;
+    else
+      a[k++] = '?';
+  }
+
+  if (l2 < l)
+    a[k] = '?';
+
+  return a;
+
+}
+
+std::string zstrtohex(const std::string s)
+{
+  return ZBEGIN_+strtohex(s)+ZEND_;
+}
+
+std::string zhextostr(const std::string s)
+{
+  size_t lb = ZBEGIN_.size();
+  size_t le = ZEND_.size();
+  size_t ls = s.size();
+
+  if (ls < lb + le)
+    return s;
+
+  if (s.substr(0,lb) != ZBEGIN_)
+    return s;
+
+  if (s.substr(ls-le) != ZEND_)
+    return s;
+
+  return hextostr(s.substr(lb,ls-lb-le));
+}
+
+//generates random passwoord of length len
+std::string gen_random(const int len) {
+ timeval t1;
+ gettimeofday(&t1, NULL);
+ srand(t1.tv_usec * t1.tv_sec);
+ static const char alphanum[] =
+   "0123456789"
+   "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
+   "abcdefghijklmnopqrstuvwxyz";
+
+ std::string s;
+ s.resize(len);
+ for (int i = 0; i < len; ++i) 
+ {
+   s[i] = alphanum[std::rand() % (sizeof(alphanum) - 1)];
+ }
+ return s;
+}
+
+#include <fstream>
+#include <string>
+#include <cerrno>
+
+int get_file_contents(std::string &contents,const std::string &filename)
+{
+  std::ifstream in(filename.c_str(), std::ios::in | std::ios::binary);
+  if (in)
+  {
+    in.seekg(0, std::ios::end);
+    contents.resize(in.tellg());
+    in.seekg(0, std::ios::beg);
+    in.read(&contents[0], contents.size());
+    in.close();
+    return(0);
+  }
+  return -1;
+}
+
+// tries to mimic mkdir -p
+int mkdir_p(const std::string &d,mode_t mode)
+{
+  size_t p=0;
+
+  std::string s = d;
+  if (s[s.size()-1] != '/')
+    s += '/';
+
+  int rc;
+  std::string t;
+  while (( p = s.find_first_of('/',p)) != s.npos)
+  {
+    p++;
+    t = s.substr(0,p);
+    rc = mkdir (t.c_str(),mode);
+    if (rc < 0)
+    {
+      if (errno == EEXIST)
+      {
+	struct stat buf;
+	rc = lstat(t.c_str(),&buf);
+	if (rc < 0)
+	  break;
+	if (!S_ISDIR(buf.st_mode))
+	{
+	  rc = -1;
+	  break;
+	}
+      }
+      else
+	break;
+    }
+    if ( p == s.size())
+      break;
+  }
+  return rc;
+}
+
+std::string envtostr(const std::string &s)
+{
+  char *a = getenv(s.c_str());
+  if (a == 0)
+    return "";
+
+  return a;
+}
+
+int get_dir_list(std::list <std::string> &v,
+                 const std::string       &d)
+{
+  DIR *dir;
+  struct dirent *ent;
+  dir = opendir(d.c_str());
+  if (dir == 0)
+    return 1;
+
+  while ((ent = readdir (dir)) != 0) 
+    v.push_back(ent->d_name);
+  closedir (dir);
+  return 0;
+}
+
+size_t count_lines(std::ifstream &f) { 
+    int number_of_lines = 0;
+    std::string line;
+
+    std::streampos p = f.tellg();
+
+    while (std::getline(f, line))
+        ++number_of_lines;
+    f.clear();
+    f.seekg(p);
+    return number_of_lines;
+}
+
+#include <sys/times.h>
+#include <sys/time.h>
+#include <time.h>
+#include <stdlib.h>
+#include "clocks.h"
+#include <unistd.h>
+
+double progress_bar::wallc()
+{
+  struct timeval t;
+
+  gettimeofday(&t,0);
+  return t.tv_sec+0.000001*(double)t.tv_usec;
+}
+
+progress_bar::progress_bar()
+{
+  this->clear();
+  this->timer        = 0.3;
+  this->bar_length   = 60;
+}
+ 
+void progress_bar::clear()
+{
+  this->firsttime  = 1;
+  this->walltime   = wallc();
+}
+
+void progress_bar::set_timer(float t)
+{
+  this->timer = t;
+}
+
+std::string progress_bar::show(size_t num, double frac)
+{
+  bool c;
+  return this->show(num,frac,c);
+}
+
+std::string progress_bar::show(size_t num, double frac, bool &changed)
+{
+  changed = 0;
+  if (!this->firsttime)
+  {
+    if (this->wallc() - this->walltime < this->timer)
+      return this->pshow;
+  }
+
+  changed = this -> firsttime || this->pfrac != frac || this->pnum != num;
+
+  if (! changed)
+    return this->pshow;
+
+  this->firsttime = 0;
+
+  int f = 100*frac+0.5;
+
+  std::string b((size_t)(frac*this->bar_length),'x');
+  b.resize(this->bar_length,'-');
+
+  this->pshow = NumberToString(num) + ' ' + NumberToString(f) + "% ["
+                + b + ']';
+
+  this->pnum     = num;
+  this->pfrac    = frac;
+  this->walltime = this->wallc();
+
+  return this->pshow;
+  
+}
+
+std::string progress_bar::show(size_t num)
+{
+  bool c;
+  return this->show(num,c);
+}
+
+std::string progress_bar::show(size_t num, bool &changed)
+{
+  changed = 0;
+  if (!this->firsttime)
+  {
+    if (this->wallc() - this->walltime < this->timer)
+      return this->pshow;
+  }
+
+  changed = this -> firsttime || this->pnum != num;
+
+  if (! changed)
+    return this->pshow;
+
+  this->firsttime = 0;
+
+  this->pshow     = NumberToString(num);
+  this->pnum      = num;
+  this->walltime  = this->wallc();
+
+  return this->pshow;
+  
+}
Index: /tags/0.92/wutils.h
===================================================================
--- /tags/0.92/wutils.h	(revision 19)
+++ /tags/0.92/wutils.h	(revision 19)
@@ -0,0 +1,121 @@
+
+/*
+   Copyright 2013 Willem Vermin, SURFsara
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+ */
+#ifndef WUTILS_H
+#define WUTILS_H
+#include <vector>
+#include <string>
+#include <sstream>
+#include <iomanip>
+#include <sys/types.h>
+#include <list>
+#include <dirent.h>
+std::vector <std::string> split(std::vector <std::string> &v, const std::string s, const char sep);
+char* str_to_charp(const std::string s);
+
+template <typename T>
+  std::string NumberToString ( T Number )
+  {
+    std::ostringstream ss;
+    ss << Number;
+    return ss.str();
+  }
+template <typename T>
+  std::string NumberToString ( T Number,int w)
+  {
+    std::ostringstream ss;
+    ss <<std::setfill('0') << std::setw(w) << Number;
+    return ss.str();
+  }
+template <typename T>
+  T StringToNumber ( const std::string &Text )
+  {
+    std::istringstream ss(Text);
+    T result;
+    return ss >> result ? result : 0;
+  }
+
+std::string strtohex(const std::string s);
+std::string hextostr(const std::string s);
+const static std::string ZBEGIN_ = "ZZZ";
+const static std::string ZEND_   = "YY";
+
+std::string zstrtohex(const std::string s);
+std::string zhextostr(const std::string s);
+
+std::string gen_random(const int len);
+
+int get_file_contents(std::string &contents,const std::string &filename);
+
+inline std::string trim_right(
+  const std::string& s,
+  const std::string& delimiters = " \f\n\r\t\v" )
+{
+  size_t p = s.find_last_not_of(delimiters);
+  if (p == s.npos)
+    return "";
+  return s.substr( 0, p + 1 );
+}
+
+inline std::string trim_left(
+  const std::string& s,
+  const std::string& delimiters = " \f\n\r\t\v" )
+{
+  size_t p = s.find_first_not_of(delimiters);
+  if (p == s.npos)
+    return "";
+  return s.substr(p);
+}
+
+inline std::string trim(
+  const std::string& s,
+  const std::string& delimiters = " \f\n\r\t\v" )
+{
+  return trim_left( trim_right( s, delimiters ), delimiters );
+}
+
+int mkdir_p(const std::string &s,mode_t m);
+
+std::string envtostr(const std::string &s);
+
+int get_dir_list(std::list <std::string> &v,
+                 const std::string       &d);
+
+size_t count_lines(std::ifstream &f);
+
+class progress_bar
+{
+  private:
+    double timer;
+    double pfrac;
+    size_t pnum;
+    bool firsttime;
+    double walltime;
+    double wallc();
+    std::string pshow;
+    size_t bar_length;
+
+  public:
+    progress_bar();
+    void clear();
+    void set_timer(float t);
+    std::string show(size_t num, double frac,bool &changed);
+    std::string show(size_t num, double frac);
+    std::string show(size_t num, bool &changed);
+    std::string show(size_t num);
+
+};
+#endif
Index: /tags/0.92/zzdo
===================================================================
--- /tags/0.92/zzdo	(revision 19)
+++ /tags/0.92/zzdo	(revision 19)
@@ -0,0 +1,17 @@
+#!/bin/bash
+###
+#   Copyright 2013 Willem Vermin, SURFsara
+#
+#   Licensed under the Apache License, Version 2.0 (the "License");
+#   you may not use this file except in compliance with the License.
+#   You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#   Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
+###
+"$@" | ./stoposzztostr
