source: trunk/doc/description.lyx @ 20

Last change on this file since 20 was 7, checked in by willem, 11 years ago

willem

File size: 65.6 KB
Line 
1#LyX 2.0 created this file. For more info see http://www.lyx.org/
2\lyxformat 413
3\begin_document
4\begin_header
5\textclass article
6\use_default_options true
7\maintain_unincluded_children false
8\language english
9\language_package default
10\inputencoding auto
11\fontencoding global
12\font_roman default
13\font_sans default
14\font_typewriter default
15\font_default_family default
16\use_non_tex_fonts false
17\font_sc false
18\font_osf false
19\font_sf_scale 100
20\font_tt_scale 100
21
22\graphics default
23\default_output_format default
24\output_sync 0
25\bibtex_command default
26\index_command default
27\paperfontsize 10
28\spacing single
29\use_hyperref false
30\papersize a4paper
31\use_geometry true
32\use_amsmath 1
33\use_esint 1
34\use_mhchem 1
35\use_mathdots 1
36\cite_engine basic
37\use_bibtopic false
38\use_indices false
39\paperorientation portrait
40\suppress_date false
41\use_refstyle 1
42\index Index
43\shortcut idx
44\color #008000
45\end_index
46\secnumdepth 3
47\tocdepth 3
48\paragraph_separation indent
49\paragraph_indentation 0bp
50\quotes_language english
51\papercolumns 1
52\papersides 1
53\paperpagestyle default
54\tracking_changes false
55\output_changes false
56\html_math_output 0
57\html_css_as_file 0
58\html_be_strict false
59\end_header
60
61\begin_body
62
63\begin_layout Title
64STOPOS
65\end_layout
66
67\begin_layout Author
68Willem Vermin, willem.vermin@surfsara.nl
69\begin_inset Newline newline
70\end_inset
71
72Bas van der Vlies, bas.vandervlies@surfsara.nl
73\end_layout
74
75\begin_layout Date
762013-02-01
77\end_layout
78
79\begin_layout Abstract
80In this article we describe the internals and externals of the program stopos.
81 Stopos is primarily designed to facilitate the running of many tasks on
82 a cluster computer.
83 It is based on managing lines in a pool.
84 These lines will in general be used as command lines to define tasks.
85 Stopos is heavily inspired by ToPoS (https://grid.sara.nl/wiki/index.php/Using_the
86_Grid/ToPoS) and experiences with disparm (https://www.surfsara.nl/systems/lisa/so
87ftware/disparm).
88 This project was carried out as a
89\begin_inset Quotes eld
90\end_inset
91
92seed-fund project
93\begin_inset Quotes erd
94\end_inset
95
96 at SURFsara.
97\end_layout
98
99\begin_layout Subsection*
100Introduction
101\end_layout
102
103\begin_layout Standard
104A typical use of a cluster computer, such as SURFsara's Lisa system
105\begin_inset Foot
106status open
107
108\begin_layout Plain Layout
109https://www.surfsara.nl/systems/lisa
110\end_layout
111
112\end_inset
113
114, is running many tasks each task with different parameters.
115 A typical example is a parameter scan.
116 A typical job allocates a node, and runs a number of tasks in parallel.
117 The number of tasks should in general be equal to the amount of cores and/or
118 the amount of memory.
119 Sometimes, thousands of these jobs are necessary, and normally, the job
120 scripts are generated by a script.
121 In a homogeneous cluster (all nodes have the same amount of cores and the
122 same amount of memory) this is clumsy but doable, but when the cluster
123 is heterogeneous (such as the Lisa cluster), it becomes very hard to use
124 all types of nodes available, and adapt the number of processes to the
125 node where the job runs on.
126 To deal with this problems, stopos was created: a server that manages a
127 pool of lines.
128 These lines are defined by the user, and the server serves the lines to
129 jobs, such that each line is produced once.
130 Earlier successful solutions are ToPoS
131\begin_inset Foot
132status open
133
134\begin_layout Plain Layout
135https://grid.sara.nl/wiki/index.php/Using_the_Grid/ToPoS
136\end_layout
137
138\end_inset
139
140 and disparm
141\begin_inset Foot
142status open
143
144\begin_layout Plain Layout
145https://www.surfsara.nl/systems/lisa/software/disparm
146\end_layout
147
148\end_inset
149
150.
151 ToPoS is designed to be used on a server that can be accessed from the
152 Grid, while disparm is based on files in the user's home directory.
153 We think that ToPoS is not optimal for the Lisa system because of lack
154 of cluster-based authentication and the universal access.
155 The disparm approach is in these respects better, but in practice it appeared
156 that the performance under heavy usage the performance was too low, and
157 the reliability was not 100%: sometimes a line was produced more than once,
158 or lines were not produced at all.
159 Also, it is not possible to add lines dynamically: all lines have to be
160 inserted at the beginning.
161 We think that the approach followed in stopos is a well balanced solution,
162 particularly fit to be used in a cluster environment.
163\end_layout
164
165\begin_layout Subsection*
166Usage
167\end_layout
168
169\begin_layout Standard
170A typical usage of stopos is:
171\end_layout
172
173\begin_layout Itemize
174fill the pool with lines
175\end_layout
176
177\begin_layout Itemize
178start jobs, each job gets one or more lines from the pool, and remove the
179 lines that are not needed any more
180\begin_inset Foot
181status open
182
183\begin_layout Plain Layout
184Normally, a line is produced only once.
185 However, the user can request that a line is produced and used again, for
186 example in the case the previous task handling this line ran into a time
187 limit.
188\end_layout
189
190\end_inset
191
192
193\end_layout
194
195\begin_layout Itemize
196maybe: add more lines to the pool
197\end_layout
198
199\begin_layout Standard
200Stopos is implemented as a shell function in the bash shell, or as an alias
201 in the (t)csh shell.
202 Stopos calls the program stoposclient and transforms it's output to environment
203 variables, the most important being STOPOS_RC and STOPOS_VALUE.
204\end_layout
205
206\begin_layout Standard
207The commands to create the pool:
208\begin_inset Newline newline
209\end_inset
210
211
212\end_layout
213
214\begin_layout Standard
215
216\family typewriter
217\begin_inset space ~
218\end_inset
219
220module load stopos
221\begin_inset space ~
222\end_inset
223
224
225\begin_inset space ~
226\end_inset
227
228
229\begin_inset space ~
230\end_inset
231
232
233\begin_inset space ~
234\end_inset
235
236
237\begin_inset space ~
238\end_inset
239
240
241\begin_inset space ~
242\end_inset
243
244# activate the stopos software
245\end_layout
246
247\begin_layout Standard
248
249\family typewriter
250\begin_inset space ~
251\end_inset
252
253stopos create
254\begin_inset space ~
255\end_inset
256
257
258\begin_inset space ~
259\end_inset
260
261
262\begin_inset space ~
263\end_inset
264
265
266\begin_inset space ~
267\end_inset
268
269
270\begin_inset space ~
271\end_inset
272
273
274\begin_inset space ~
275\end_inset
276
277
278\begin_inset space ~
279\end_inset
280
281
282\begin_inset space ~
283\end_inset
284
285
286\begin_inset space ~
287\end_inset
288
289
290\begin_inset space ~
291\end_inset
292
293
294\begin_inset space ~
295\end_inset
296
297
298\begin_inset space ~
299\end_inset
300
301# create the pool
302\end_layout
303
304\begin_layout Standard
305
306\family typewriter
307\begin_inset space ~
308\end_inset
309
310stopos add parmfile
311\begin_inset space ~
312\end_inset
313
314
315\begin_inset space ~
316\end_inset
317
318
319\begin_inset space ~
320\end_inset
321
322
323\begin_inset space ~
324\end_inset
325
326
327\begin_inset space ~
328\end_inset
329
330
331\begin_inset space ~
332\end_inset
333
334# add the lines in the file parmfile to the pool
335\begin_inset Newline newline
336\end_inset
337
338
339\end_layout
340
341\begin_layout Standard
342In a job script, one could use code like this:
343\begin_inset Newline newline
344\end_inset
345
346
347\begin_inset Newline newline
348\end_inset
349
350
351\family typewriter
352
353\begin_inset space ~
354\end_inset
355
356module load stopos
357\begin_inset space ~
358\end_inset
359
360
361\begin_inset space ~
362\end_inset
363
364
365\begin_inset space ~
366\end_inset
367
368
369\begin_inset space ~
370\end_inset
371
372
373\begin_inset space ~
374\end_inset
375
376
377\begin_inset space ~
378\end_inset
379
380
381\begin_inset space ~
382\end_inset
383
384
385\begin_inset space ~
386\end_inset
387
388
389\begin_inset space ~
390\end_inset
391
392
393\begin_inset space ~
394\end_inset
395
396
397\begin_inset space ~
398\end_inset
399
400# activate the stopos software
401\end_layout
402
403\begin_layout Standard
404
405\family typewriter
406\begin_inset space ~
407\end_inset
408
409stopos next
410\begin_inset space ~
411\end_inset
412
413
414\begin_inset space ~
415\end_inset
416
417
418\begin_inset space ~
419\end_inset
420
421
422\begin_inset space ~
423\end_inset
424
425
426\begin_inset space ~
427\end_inset
428
429
430\begin_inset space ~
431\end_inset
432
433
434\begin_inset space ~
435\end_inset
436
437
438\begin_inset space ~
439\end_inset
440
441
442\begin_inset space ~
443\end_inset
444
445
446\begin_inset space ~
447\end_inset
448
449
450\begin_inset space ~
451\end_inset
452
453
454\begin_inset space ~
455\end_inset
456
457
458\begin_inset space ~
459\end_inset
460
461
462\begin_inset space ~
463\end_inset
464
465
466\begin_inset space ~
467\end_inset
468
469
470\begin_inset space ~
471\end_inset
472
473
474\begin_inset space ~
475\end_inset
476
477# get a line from the pool
478\end_layout
479
480\begin_layout Standard
481
482\family typewriter
483\begin_inset space ~
484\end_inset
485
486if [ $STOPOS_RC = OK] ; then # check the return code
487\end_layout
488
489\begin_layout Standard
490
491\family typewriter
492\begin_inset space ~
493\end_inset
494
495
496\begin_inset space ~
497\end_inset
498
499
500\begin_inset space ~
501\end_inset
502
503run_my_prog $STOPOS_VALUE
504\begin_inset space ~
505\end_inset
506
507
508\begin_inset space ~
509\end_inset
510
511# run the program using the line as parameter
512\end_layout
513
514\begin_layout Standard
515
516\family typewriter
517\begin_inset space ~
518\end_inset
519
520fi
521\begin_inset Newline newline
522\end_inset
523
524
525\end_layout
526
527\begin_layout Standard
528A more elaborate job script, using all cores available on a node is available
529 in the Appendix, together with the man page of stopos.
530\end_layout
531
532\begin_layout Subsection*
533Tasks and communication
534\end_layout
535
536\begin_layout Standard
537Stopos is a shell function or an alias that calls stoposclient.
538 Stoposclient communicates with stoposserver via an http server (apache
539\begin_inset Foot
540status open
541
542\begin_layout Plain Layout
543http://httpd.apache.org/
544\end_layout
545
546\end_inset
547
548, thttpd
549\begin_inset Foot
550status open
551
552\begin_layout Plain Layout
553http://opensource.dyc.edu/sthttpd
554\end_layout
555
556\end_inset
557
558, ...).
559 The server sends commands to the server, and transforms the answers to
560 a format that are sourced by stopos to define environment variables.
561 Care is taken that communications between the client and the server are
562 short in length: at most a few kilobytes are sent between client and server.
563 
564\end_layout
565
566\begin_layout Subsubsection*
567Tasks of the client
568\end_layout
569
570\begin_layout Standard
571The client parses it's command line, sends the appropriate command to the
572 server and receives the response.
573 This response is put in a form that can be 'eval-ed' so that environment
574 variables are set.
575\begin_inset Newline newline
576\end_inset
577
578
579\begin_inset Newline newline
580\end_inset
581
582The client recognizes the following commands on its command line:
583\end_layout
584
585\begin_layout Itemize
586
587\family typewriter
588-h,--h
589\begin_inset space ~
590\end_inset
591
592
593\begin_inset space ~
594\end_inset
595
596
597\begin_inset space ~
598\end_inset
599
600
601\begin_inset space ~
602\end_inset
603
604
605\begin_inset space ~
606\end_inset
607
608
609\begin_inset space ~
610\end_inset
611
612
613\begin_inset space ~
614\end_inset
615
616
617\begin_inset space ~
618\end_inset
619
620
621\begin_inset space ~
622\end_inset
623
624
625\begin_inset space ~
626\end_inset
627
628
629\begin_inset space ~
630\end_inset
631
632
633\begin_inset space ~
634\end_inset
635
636
637\begin_inset space ~
638\end_inset
639
640
641\begin_inset space ~
642\end_inset
643
644
645\begin_inset space ~
646\end_inset
647
648
649\begin_inset space ~
650\end_inset
651
652
653\begin_inset space ~
654\end_inset
655
656
657\begin_inset space ~
658\end_inset
659
660
661\begin_inset space ~
662\end_inset
663
664
665\begin_inset space ~
666\end_inset
667
668
669\begin_inset space ~
670\end_inset
671
672
673\begin_inset space ~
674\end_inset
675
676
677\begin_inset space ~
678\end_inset
679
680
681\begin_inset space ~
682\end_inset
683
684
685\begin_inset space ~
686\end_inset
687
688
689\begin_inset space ~
690\end_inset
691
692
693\begin_inset space ~
694\end_inset
695
696
697\begin_inset space ~
698\end_inset
699
700: prints a short usage message
701\end_layout
702
703\begin_layout Itemize
704
705\family typewriter
706-v,--v
707\begin_inset space ~
708\end_inset
709
710
711\begin_inset space ~
712\end_inset
713
714
715\begin_inset space ~
716\end_inset
717
718
719\begin_inset space ~
720\end_inset
721
722
723\begin_inset space ~
724\end_inset
725
726
727\begin_inset space ~
728\end_inset
729
730
731\begin_inset space ~
732\end_inset
733
734
735\begin_inset space ~
736\end_inset
737
738
739\begin_inset space ~
740\end_inset
741
742
743\begin_inset space ~
744\end_inset
745
746
747\begin_inset space ~
748\end_inset
749
750
751\begin_inset space ~
752\end_inset
753
754
755\begin_inset space ~
756\end_inset
757
758
759\begin_inset space ~
760\end_inset
761
762
763\begin_inset space ~
764\end_inset
765
766
767\begin_inset space ~
768\end_inset
769
770
771\begin_inset space ~
772\end_inset
773
774
775\begin_inset space ~
776\end_inset
777
778
779\begin_inset space ~
780\end_inset
781
782
783\begin_inset space ~
784\end_inset
785
786
787\begin_inset space ~
788\end_inset
789
790
791\begin_inset space ~
792\end_inset
793
794
795\begin_inset space ~
796\end_inset
797
798
799\begin_inset space ~
800\end_inset
801
802
803\begin_inset space ~
804\end_inset
805
806
807\begin_inset space ~
808\end_inset
809
810
811\begin_inset space ~
812\end_inset
813
814
815\begin_inset space ~
816\end_inset
817
818: prints the version number
819\end_layout
820
821\begin_layout Itemize
822
823\family typewriter
824create [ -p,--pool POOL ]
825\begin_inset space ~
826\end_inset
827
828
829\begin_inset space ~
830\end_inset
831
832
833\begin_inset space ~
834\end_inset
835
836
837\begin_inset space ~
838\end_inset
839
840
841\begin_inset space ~
842\end_inset
843
844
845\begin_inset space ~
846\end_inset
847
848
849\begin_inset space ~
850\end_inset
851
852
853\begin_inset space ~
854\end_inset
855
856
857\begin_inset space ~
858\end_inset
859
860:
861\family default
862 create a pool
863\end_layout
864
865\begin_layout Itemize
866
867\family typewriter
868status [ -p,--pool POOL ]
869\begin_inset space ~
870\end_inset
871
872
873\begin_inset space ~
874\end_inset
875
876
877\begin_inset space ~
878\end_inset
879
880
881\begin_inset space ~
882\end_inset
883
884
885\begin_inset space ~
886\end_inset
887
888
889\begin_inset space ~
890\end_inset
891
892
893\begin_inset space ~
894\end_inset
895
896
897\begin_inset space ~
898\end_inset
899
900
901\begin_inset space ~
902\end_inset
903
904:
905\family default
906 get status from a pool
907\end_layout
908
909\begin_layout Itemize
910
911\family typewriter
912purge [ -p,--pool POOL]
913\begin_inset space ~
914\end_inset
915
916
917\begin_inset space ~
918\end_inset
919
920
921\begin_inset space ~
922\end_inset
923
924
925\begin_inset space ~
926\end_inset
927
928
929\begin_inset space ~
930\end_inset
931
932
933\begin_inset space ~
934\end_inset
935
936
937\begin_inset space ~
938\end_inset
939
940
941\begin_inset space ~
942\end_inset
943
944
945\begin_inset space ~
946\end_inset
947
948
949\begin_inset space ~
950\end_inset
951
952
953\begin_inset space ~
954\end_inset
955
956:
957\family default
958 remove a pool
959\end_layout
960
961\begin_layout Itemize
962
963\family typewriter
964pools
965\begin_inset space ~
966\end_inset
967
968
969\begin_inset space ~
970\end_inset
971
972
973\begin_inset space ~
974\end_inset
975
976
977\begin_inset space ~
978\end_inset
979
980
981\begin_inset space ~
982\end_inset
983
984
985\begin_inset space ~
986\end_inset
987
988
989\begin_inset space ~
990\end_inset
991
992
993\begin_inset space ~
994\end_inset
995
996
997\begin_inset space ~
998\end_inset
999
1000
1001\begin_inset space ~
1002\end_inset
1003
1004
1005\begin_inset space ~
1006\end_inset
1007
1008
1009\begin_inset space ~
1010\end_inset
1011
1012
1013\begin_inset space ~
1014\end_inset
1015
1016
1017\begin_inset space ~
1018\end_inset
1019
1020
1021\begin_inset space ~
1022\end_inset
1023
1024
1025\begin_inset space ~
1026\end_inset
1027
1028
1029\begin_inset space ~
1030\end_inset
1031
1032
1033\begin_inset space ~
1034\end_inset
1035
1036
1037\begin_inset space ~
1038\end_inset
1039
1040
1041\begin_inset space ~
1042\end_inset
1043
1044
1045\begin_inset space ~
1046\end_inset
1047
1048
1049\begin_inset space ~
1050\end_inset
1051
1052
1053\begin_inset space ~
1054\end_inset
1055
1056
1057\begin_inset space ~
1058\end_inset
1059
1060
1061\begin_inset space ~
1062\end_inset
1063
1064
1065\begin_inset space ~
1066\end_inset
1067
1068
1069\begin_inset space ~
1070\end_inset
1071
1072
1073\begin_inset space ~
1074\end_inset
1075
1076
1077\begin_inset space ~
1078\end_inset
1079
1080:
1081\family default
1082 what are my pools
1083\end_layout
1084
1085\begin_layout Itemize
1086
1087\family typewriter
1088add [ -p,--pool POOL] [ FILENAME ]:
1089\family default
1090 add a file to the pool
1091\end_layout
1092
1093\begin_layout Itemize
1094
1095\family typewriter
1096next [-m,--multi]
1097\begin_inset space ~
1098\end_inset
1099
1100
1101\begin_inset space ~
1102\end_inset
1103
1104
1105\begin_inset space ~
1106\end_inset
1107
1108
1109\begin_inset space ~
1110\end_inset
1111
1112
1113\begin_inset space ~
1114\end_inset
1115
1116
1117\begin_inset space ~
1118\end_inset
1119
1120
1121\begin_inset space ~
1122\end_inset
1123
1124
1125\begin_inset space ~
1126\end_inset
1127
1128
1129\begin_inset space ~
1130\end_inset
1131
1132
1133\begin_inset space ~
1134\end_inset
1135
1136
1137\begin_inset space ~
1138\end_inset
1139
1140
1141\begin_inset space ~
1142\end_inset
1143
1144
1145\begin_inset space ~
1146\end_inset
1147
1148
1149\begin_inset space ~
1150\end_inset
1151
1152
1153\begin_inset space ~
1154\end_inset
1155
1156
1157\begin_inset space ~
1158\end_inset
1159
1160
1161\begin_inset space ~
1162\end_inset
1163
1164:
1165\family default
1166 get next line from the pool
1167\end_layout
1168
1169\begin_layout Itemize
1170
1171\family typewriter
1172remove [ KEY ]
1173\begin_inset space ~
1174\end_inset
1175
1176
1177\begin_inset space ~
1178\end_inset
1179
1180
1181\begin_inset space ~
1182\end_inset
1183
1184
1185\begin_inset space ~
1186\end_inset
1187
1188
1189\begin_inset space ~
1190\end_inset
1191
1192
1193\begin_inset space ~
1194\end_inset
1195
1196
1197\begin_inset space ~
1198\end_inset
1199
1200
1201\begin_inset space ~
1202\end_inset
1203
1204
1205\begin_inset space ~
1206\end_inset
1207
1208
1209\begin_inset space ~
1210\end_inset
1211
1212
1213\begin_inset space ~
1214\end_inset
1215
1216
1217\begin_inset space ~
1218\end_inset
1219
1220
1221\begin_inset space ~
1222\end_inset
1223
1224
1225\begin_inset space ~
1226\end_inset
1227
1228
1229\begin_inset space ~
1230\end_inset
1231
1232
1233\begin_inset space ~
1234\end_inset
1235
1236
1237\begin_inset space ~
1238\end_inset
1239
1240
1241\begin_inset space ~
1242\end_inset
1243
1244
1245\begin_inset space ~
1246\end_inset
1247
1248
1249\begin_inset space ~
1250\end_inset
1251
1252:
1253\family default
1254 remove line
1255\end_layout
1256
1257\begin_layout Itemize
1258
1259\family typewriter
1260dump [ -p,--p POOL ]
1261\begin_inset space ~
1262\end_inset
1263
1264
1265\begin_inset space ~
1266\end_inset
1267
1268
1269\begin_inset space ~
1270\end_inset
1271
1272
1273\begin_inset space ~
1274\end_inset
1275
1276
1277\begin_inset space ~
1278\end_inset
1279
1280
1281\begin_inset space ~
1282\end_inset
1283
1284
1285\begin_inset space ~
1286\end_inset
1287
1288
1289\begin_inset space ~
1290\end_inset
1291
1292
1293\begin_inset space ~
1294\end_inset
1295
1296
1297\begin_inset space ~
1298\end_inset
1299
1300
1301\begin_inset space ~
1302\end_inset
1303
1304
1305\begin_inset space ~
1306\end_inset
1307
1308
1309\begin_inset space ~
1310\end_inset
1311
1312
1313\begin_inset space ~
1314\end_inset
1315
1316:
1317\family default
1318 dump a line from the pool
1319\end_layout
1320
1321\begin_layout Standard
1322For example:
1323\begin_inset Newline newline
1324\end_inset
1325
1326
1327\begin_inset Newline newline
1328\end_inset
1329
1330
1331\family typewriter
1332stopos create -p mypool1
1333\end_layout
1334
1335\begin_layout Standard
1336
1337\family typewriter
1338stopos add -p mypool1 parmfile
1339\begin_inset Newline newline
1340\end_inset
1341
1342
1343\end_layout
1344
1345\begin_layout Standard
1346The -p (or --pool) flag defines the name of the pool file to be used (
1347\begin_inset Quotes eld
1348\end_inset
1349
1350parmfile
1351\begin_inset Quotes erd
1352\end_inset
1353
1354 in this example).
1355 If no -p or --pool flag is present, the value of the pool is taken from
1356 the environment variable STOPOS_POOL.
1357 If that is not available, the value
1358\begin_inset Quotes eld
1359\end_inset
1360
1361pool
1362\begin_inset Quotes erd
1363\end_inset
1364
1365 is used.
1366\end_layout
1367
1368\begin_layout Standard
1369The -m (or --multi) flag signifies that lines that are already produces,
1370 may be produced again.
1371\end_layout
1372
1373\begin_layout Standard
1374The default for FILENAME is standard input.
1375\end_layout
1376
1377\begin_layout Standard
1378If no value is given for KEY with the
1379\begin_inset Quotes eld
1380\end_inset
1381
1382remove
1383\begin_inset Quotes erd
1384\end_inset
1385
1386 command, the value of the environment variable STOPOS_KEY is used.
1387 This value is set by the
1388\begin_inset Quotes eld
1389\end_inset
1390
1391next
1392\begin_inset Quotes erd
1393\end_inset
1394
1395 command.
1396\begin_inset Newline newline
1397\end_inset
1398
1399
1400\begin_inset Newline newline
1401\end_inset
1402
1403The following environment variables are set by the command
1404\begin_inset Quotes eld
1405\end_inset
1406
1407stopos
1408\begin_inset Quotes erd
1409\end_inset
1410
1411:
1412\end_layout
1413
1414\begin_layout Itemize
1415
1416\family typewriter
1417STOPOS_RC
1418\begin_inset space ~
1419\end_inset
1420
1421
1422\begin_inset space ~
1423\end_inset
1424
1425
1426\begin_inset space ~
1427\end_inset
1428
1429
1430\begin_inset space ~
1431\end_inset
1432
1433
1434\begin_inset space ~
1435\end_inset
1436
1437
1438\begin_inset space ~
1439\end_inset
1440
1441
1442\begin_inset space ~
1443\end_inset
1444
1445:
1446\family default
1447 return code of the command.
1448 The value
1449\begin_inset Quotes eld
1450\end_inset
1451
1452OK
1453\begin_inset Quotes erd
1454\end_inset
1455
1456 means that things went well
1457\end_layout
1458
1459\begin_layout Itemize
1460
1461\family typewriter
1462STOPOS_VALUE
1463\begin_inset space ~
1464\end_inset
1465
1466
1467\begin_inset space ~
1468\end_inset
1469
1470
1471\begin_inset space ~
1472\end_inset
1473
1474
1475\begin_inset space ~
1476\end_inset
1477
1478 :
1479\family default
1480 result of the
1481\begin_inset Quotes eld
1482\end_inset
1483
1484next
1485\begin_inset Quotes erd
1486\end_inset
1487
1488 and
1489\begin_inset Quotes eld
1490\end_inset
1491
1492pools
1493\begin_inset Quotes erd
1494\end_inset
1495
1496 command
1497\end_layout
1498
1499\begin_layout Itemize
1500
1501\family typewriter
1502STOPOS_COMMITTED :
1503\family default
1504 number of times the
1505\begin_inset Quotes eld
1506\end_inset
1507
1508next
1509\begin_inset Quotes erd
1510\end_inset
1511
1512 line was produced
1513\end_layout
1514
1515\begin_layout Itemize
1516
1517\family typewriter
1518STOPOS_KEY
1519\begin_inset space ~
1520\end_inset
1521
1522
1523\begin_inset space ~
1524\end_inset
1525
1526
1527\begin_inset space ~
1528\end_inset
1529
1530
1531\begin_inset space ~
1532\end_inset
1533
1534
1535\begin_inset space ~
1536\end_inset
1537
1538
1539\begin_inset space ~
1540\end_inset
1541
1542:
1543\family default
1544 set by the
1545\begin_inset Quotes eld
1546\end_inset
1547
1548next
1549\begin_inset Quotes erd
1550\end_inset
1551
1552 command.
1553 Used by the
1554\begin_inset Quotes eld
1555\end_inset
1556
1557remove
1558\begin_inset Quotes erd
1559\end_inset
1560
1561 command
1562\end_layout
1563
1564\begin_layout Itemize
1565
1566\family typewriter
1567STOPOS_COUNT
1568\begin_inset space ~
1569\end_inset
1570
1571
1572\begin_inset space ~
1573\end_inset
1574
1575
1576\begin_inset space ~
1577\end_inset
1578
1579
1580\begin_inset space ~
1581\end_inset
1582
1583:
1584\family default
1585 set by
1586\begin_inset Quotes eld
1587\end_inset
1588
1589status
1590\begin_inset Quotes erd
1591\end_inset
1592
1593: number of lines added to the pool
1594\end_layout
1595
1596\begin_layout Itemize
1597
1598\family typewriter
1599STOPOS_PRESENT
1600\begin_inset space ~
1601\end_inset
1602
1603
1604\begin_inset space ~
1605\end_inset
1606
1607
1608\begin_inset space ~
1609\end_inset
1610
1611:
1612\family default
1613 set by
1614\begin_inset Quotes eld
1615\end_inset
1616
1617status
1618\begin_inset Quotes erd
1619\end_inset
1620
1621: number of lines in the pool
1622\end_layout
1623
1624\begin_layout Itemize
1625
1626\family typewriter
1627STOPOS_PRESENT0
1628\begin_inset space ~
1629\end_inset
1630
1631
1632\begin_inset space ~
1633\end_inset
1634
1635:
1636\family default
1637 set by
1638\begin_inset Quotes eld
1639\end_inset
1640
1641status
1642\begin_inset Quotes erd
1643\end_inset
1644
1645: number of lines never been committed
1646\end_layout
1647
1648\begin_layout Standard
1649Another task is creating a unique id for the user.
1650 This id, combined with the login name of the user and the name of the pool,
1651 will be used by the server to identify the pool to be used.
1652 The id is stored in the HOME directory of the user, in the file $HOME/.stopos/rc.
1653 If this file is missing, or inappropriate, the client will create this
1654 file.
1655\begin_inset Newline newline
1656\end_inset
1657
1658
1659\begin_inset Newline newline
1660\end_inset
1661
1662All commands result in a single communication with the server, except the
1663 
1664\begin_inset Quotes eld
1665\end_inset
1666
1667add
1668\begin_inset Quotes erd
1669\end_inset
1670
1671 command: the client will send a number of
1672\begin_inset Quotes eld
1673\end_inset
1674
1675add
1676\begin_inset Quotes erd
1677\end_inset
1678
1679 commands to the server, one for each line of the file.
1680\begin_inset Newline newline
1681\end_inset
1682
1683
1684\begin_inset Newline newline
1685\end_inset
1686
1687The
1688\begin_inset Quotes eld
1689\end_inset
1690
1691dump
1692\begin_inset Quotes erd
1693\end_inset
1694
1695 command is kind of special.
1696 It is created for the case a user wants to inspect the actual contents
1697 of the pool.
1698 The
1699\begin_inset Quotes eld
1700\end_inset
1701
1702dump
1703\begin_inset Quotes erd
1704\end_inset
1705
1706 command will produce one line per call, and does not interfere with the
1707 other commands.
1708 (More specific: the
1709\begin_inset Quotes eld
1710\end_inset
1711
1712next
1713\begin_inset Quotes erd
1714\end_inset
1715
1716 command and the
1717\begin_inset Quotes eld
1718\end_inset
1719
1720dump
1721\begin_inset Quotes erd
1722\end_inset
1723
1724 command use different pointers) When the last line has been produced, the
1725 next call will result in an error code (STOPOS_RC != OK).
1726 Then the next call will start again at the first available record in the
1727 pool.
1728 
1729\end_layout
1730
1731\begin_layout Subsubsection*
1732Messages from client to server
1733\end_layout
1734
1735\begin_layout Standard
1736A message from client to server consists of the following fields:
1737\end_layout
1738
1739\begin_layout Itemize
1740
1741\family typewriter
1742stopos
1743\begin_inset space ~
1744\end_inset
1745
1746
1747\begin_inset space ~
1748\end_inset
1749
1750
1751\begin_inset space ~
1752\end_inset
1753
1754:
1755\family default
1756 the string
1757\begin_inset Quotes eld
1758\end_inset
1759
1760stopos
1761\begin_inset Quotes erd
1762\end_inset
1763
1764
1765\end_layout
1766
1767\begin_layout Itemize
1768
1769\family typewriter
1770prot
1771\begin_inset space ~
1772\end_inset
1773
1774
1775\begin_inset space ~
1776\end_inset
1777
1778
1779\begin_inset space ~
1780\end_inset
1781
1782
1783\begin_inset space ~
1784\end_inset
1785
1786
1787\begin_inset space ~
1788\end_inset
1789
1790:
1791\family default
1792 the protocol to be used (i.e.
1793 which kind of database)
1794\end_layout
1795
1796\begin_layout Itemize
1797
1798\family typewriter
1799id
1800\begin_inset space ~
1801\end_inset
1802
1803
1804\begin_inset space ~
1805\end_inset
1806
1807
1808\begin_inset space ~
1809\end_inset
1810
1811
1812\begin_inset space ~
1813\end_inset
1814
1815
1816\begin_inset space ~
1817\end_inset
1818
1819
1820\begin_inset space ~
1821\end_inset
1822
1823
1824\begin_inset space ~
1825\end_inset
1826
1827:
1828\family default
1829 the identification of the user
1830\end_layout
1831
1832\begin_layout Itemize
1833
1834\family typewriter
1835pool
1836\begin_inset space ~
1837\end_inset
1838
1839
1840\begin_inset space ~
1841\end_inset
1842
1843
1844\begin_inset space ~
1845\end_inset
1846
1847
1848\begin_inset space ~
1849\end_inset
1850
1851
1852\begin_inset space ~
1853\end_inset
1854
1855:
1856\family default
1857 the name of the pool to be used
1858\end_layout
1859
1860\begin_layout Itemize
1861
1862\family typewriter
1863command
1864\begin_inset space ~
1865\end_inset
1866
1867
1868\begin_inset space ~
1869\end_inset
1870
1871:
1872\family default
1873 the command for the server
1874\end_layout
1875
1876\begin_layout Itemize
1877
1878\family typewriter
1879multi
1880\begin_inset space ~
1881\end_inset
1882
1883
1884\begin_inset space ~
1885\end_inset
1886
1887
1888\begin_inset space ~
1889\end_inset
1890
1891
1892\begin_inset space ~
1893\end_inset
1894
1895:
1896\family default
1897 the value of
1898\begin_inset Quotes eld
1899\end_inset
1900
1901multi
1902\begin_inset Quotes erd
1903\end_inset
1904
1905: can a line be produced more than once?
1906\end_layout
1907
1908\begin_layout Itemize
1909
1910\family typewriter
1911value
1912\begin_inset space ~
1913\end_inset
1914
1915
1916\begin_inset space ~
1917\end_inset
1918
1919
1920\begin_inset space ~
1921\end_inset
1922
1923
1924\begin_inset space ~
1925\end_inset
1926
1927:
1928\family default
1929 a parameter
1930\end_layout
1931
1932\begin_layout Standard
1933All fields are coded in hexadecimal strings, embraced by a fixed header
1934 and trailer.
1935 This is to ensure that no problematic things (spaces, tabs, slashes, empty
1936 strings etc.) have to be transmitted.
1937 The fields are separated with the character
1938\begin_inset Quotes eld
1939\end_inset
1940
1941-
1942\begin_inset Quotes erd
1943\end_inset
1944
1945.
1946 Example: to send a message to the server located at
1947\begin_inset Quotes eld
1948\end_inset
1949
1950http://www.stoposserver.surfsara.nl
1951\begin_inset Quotes erd
1952\end_inset
1953
1954 with:
1955\end_layout
1956
1957\begin_layout Itemize
1958protocol =
1959\begin_inset Quotes eld
1960\end_inset
1961
1962gdbm
1963\begin_inset Quotes erd
1964\end_inset
1965
1966
1967\end_layout
1968
1969\begin_layout Itemize
1970user identification =
1971\begin_inset Quotes eld
1972\end_inset
1973
1974willem.qR7w3XQp
1975\begin_inset Quotes erd
1976\end_inset
1977
1978
1979\end_layout
1980
1981\begin_layout Itemize
1982poolname =
1983\begin_inset Quotes eld
1984\end_inset
1985
1986mypool
1987\begin_inset Quotes erd
1988\end_inset
1989
1990
1991\end_layout
1992
1993\begin_layout Itemize
1994command =
1995\begin_inset Quotes eld
1996\end_inset
1997
1998add
1999\begin_inset Quotes erd
2000\end_inset
2001
2002
2003\end_layout
2004
2005\begin_layout Itemize
2006multi =
2007\begin_inset Quotes eld
2008\end_inset
2009
2010no
2011\begin_inset Quotes erd
2012\end_inset
2013
2014
2015\end_layout
2016
2017\begin_layout Itemize
2018parameter =
2019\begin_inset Quotes eld
2020\end_inset
2021
20221 2 3
2023\begin_inset Quotes erd
2024\end_inset
2025
2026
2027\end_layout
2028
2029\begin_layout Standard
2030The client would create the following url:
2031\begin_inset Newline newline
2032\end_inset
2033
2034
2035\begin_inset Newline newline
2036\end_inset
2037
2038
2039\family typewriter
2040http://www.stoposserver.surfsara.nl?ZZZ73746f706f73YY
2041\begin_inset space ~
2042\end_inset
2043
2044
2045\begin_inset space ~
2046\end_inset
2047
2048
2049\begin_inset space ~
2050\end_inset
2051
2052
2053\begin_inset Quotes eld
2054\end_inset
2055
2056stopos
2057\begin_inset Quotes erd
2058\end_inset
2059
2060
2061\end_layout
2062
2063\begin_layout Standard
2064
2065\family typewriter
2066\begin_inset space ~
2067\end_inset
2068
2069
2070\begin_inset space ~
2071\end_inset
2072
2073
2074\begin_inset space ~
2075\end_inset
2076
2077
2078\begin_inset space ~
2079\end_inset
2080
2081
2082\begin_inset space ~
2083\end_inset
2084
2085
2086\begin_inset space ~
2087\end_inset
2088
2089
2090\begin_inset space ~
2091\end_inset
2092
2093
2094\begin_inset space ~
2095\end_inset
2096
2097
2098\begin_inset space ~
2099\end_inset
2100
2101
2102\begin_inset space ~
2103\end_inset
2104
2105
2106\begin_inset space ~
2107\end_inset
2108
2109
2110\begin_inset space ~
2111\end_inset
2112
2113
2114\begin_inset space ~
2115\end_inset
2116
2117
2118\begin_inset space ~
2119\end_inset
2120
2121
2122\begin_inset space ~
2123\end_inset
2124
2125
2126\begin_inset space ~
2127\end_inset
2128
2129
2130\begin_inset space ~
2131\end_inset
2132
2133
2134\begin_inset space ~
2135\end_inset
2136
2137
2138\begin_inset space ~
2139\end_inset
2140
2141
2142\begin_inset space ~
2143\end_inset
2144
2145
2146\begin_inset space ~
2147\end_inset
2148
2149
2150\begin_inset space ~
2151\end_inset
2152
2153
2154\begin_inset space ~
2155\end_inset
2156
2157
2158\begin_inset space ~
2159\end_inset
2160
2161
2162\begin_inset space ~
2163\end_inset
2164
2165
2166\begin_inset space ~
2167\end_inset
2168
2169
2170\begin_inset space ~
2171\end_inset
2172
2173
2174\begin_inset space ~
2175\end_inset
2176
2177
2178\begin_inset space ~
2179\end_inset
2180
2181
2182\begin_inset space ~
2183\end_inset
2184
2185
2186\begin_inset space ~
2187\end_inset
2188
2189
2190\begin_inset space ~
2191\end_inset
2192
2193
2194\begin_inset space ~
2195\end_inset
2196
2197
2198\begin_inset space ~
2199\end_inset
2200
2201
2202\begin_inset space ~
2203\end_inset
2204
2205
2206\begin_inset space ~
2207\end_inset
2208
2209-ZZZ6764626dYY
2210\begin_inset space ~
2211\end_inset
2212
2213
2214\begin_inset space ~
2215\end_inset
2216
2217
2218\begin_inset space ~
2219\end_inset
2220
2221
2222\begin_inset space ~
2223\end_inset
2224
2225
2226\begin_inset space ~
2227\end_inset
2228
2229 
2230\begin_inset Quotes eld
2231\end_inset
2232
2233gdbm
2234\begin_inset Quotes erd
2235\end_inset
2236
2237
2238\end_layout
2239
2240\begin_layout Standard
2241
2242\family typewriter
2243\begin_inset space ~
2244\end_inset
2245
2246
2247\begin_inset space ~
2248\end_inset
2249
2250
2251\begin_inset space ~
2252\end_inset
2253
2254
2255\begin_inset space ~
2256\end_inset
2257
2258
2259\begin_inset space ~
2260\end_inset
2261
2262
2263\begin_inset space ~
2264\end_inset
2265
2266
2267\begin_inset space ~
2268\end_inset
2269
2270
2271\begin_inset space ~
2272\end_inset
2273
2274
2275\begin_inset space ~
2276\end_inset
2277
2278
2279\begin_inset space ~
2280\end_inset
2281
2282
2283\begin_inset space ~
2284\end_inset
2285
2286
2287\begin_inset space ~
2288\end_inset
2289
2290
2291\begin_inset space ~
2292\end_inset
2293
2294
2295\begin_inset space ~
2296\end_inset
2297
2298-ZZZ77696c6c656d2e7152377733585170YY
2299\begin_inset space ~
2300\end_inset
2301
2302
2303\begin_inset space ~
2304\end_inset
2305
2306
2307\begin_inset space ~
2308\end_inset
2309
2310
2311\begin_inset space ~
2312\end_inset
2313
2314
2315\begin_inset space ~
2316\end_inset
2317
2318 
2319\begin_inset Quotes eld
2320\end_inset
2321
2322willem.
2323\family default
2324qR7w3XQp
2325\begin_inset Quotes erd
2326\end_inset
2327
2328
2329\end_layout
2330
2331\begin_layout Standard
2332
2333\family typewriter
2334\begin_inset space ~
2335\end_inset
2336
2337
2338\begin_inset space ~
2339\end_inset
2340
2341
2342\begin_inset space ~
2343\end_inset
2344
2345
2346\begin_inset space ~
2347\end_inset
2348
2349
2350\begin_inset space ~
2351\end_inset
2352
2353
2354\begin_inset space ~
2355\end_inset
2356
2357
2358\begin_inset space ~
2359\end_inset
2360
2361
2362\begin_inset space ~
2363\end_inset
2364
2365
2366\begin_inset space ~
2367\end_inset
2368
2369
2370\begin_inset space ~
2371\end_inset
2372
2373
2374\begin_inset space ~
2375\end_inset
2376
2377
2378\begin_inset space ~
2379\end_inset
2380
2381
2382\begin_inset space ~
2383\end_inset
2384
2385
2386\begin_inset space ~
2387\end_inset
2388
2389
2390\begin_inset space ~
2391\end_inset
2392
2393
2394\begin_inset space ~
2395\end_inset
2396
2397
2398\begin_inset space ~
2399\end_inset
2400
2401
2402\begin_inset space ~
2403\end_inset
2404
2405
2406\begin_inset space ~
2407\end_inset
2408
2409
2410\begin_inset space ~
2411\end_inset
2412
2413
2414\begin_inset space ~
2415\end_inset
2416
2417
2418\begin_inset space ~
2419\end_inset
2420
2421
2422\begin_inset space ~
2423\end_inset
2424
2425
2426\begin_inset space ~
2427\end_inset
2428
2429
2430\begin_inset space ~
2431\end_inset
2432
2433
2434\begin_inset space ~
2435\end_inset
2436
2437
2438\begin_inset space ~
2439\end_inset
2440
2441
2442\begin_inset space ~
2443\end_inset
2444
2445
2446\begin_inset space ~
2447\end_inset
2448
2449
2450\begin_inset space ~
2451\end_inset
2452
2453
2454\begin_inset space ~
2455\end_inset
2456
2457
2458\begin_inset space ~
2459\end_inset
2460
2461-ZZZ6d79706f6f6cYY
2462\begin_inset space ~
2463\end_inset
2464
2465
2466\begin_inset space ~
2467\end_inset
2468
2469
2470\begin_inset space ~
2471\end_inset
2472
2473
2474\begin_inset space ~
2475\end_inset
2476
2477
2478\begin_inset space ~
2479\end_inset
2480
2481 
2482\begin_inset Quotes eld
2483\end_inset
2484
2485mypool
2486\begin_inset Quotes erd
2487\end_inset
2488
2489
2490\end_layout
2491
2492\begin_layout Standard
2493
2494\family typewriter
2495\begin_inset space ~
2496\end_inset
2497
2498
2499\begin_inset space ~
2500\end_inset
2501
2502
2503\begin_inset space ~
2504\end_inset
2505
2506
2507\begin_inset space ~
2508\end_inset
2509
2510
2511\begin_inset space ~
2512\end_inset
2513
2514
2515\begin_inset space ~
2516\end_inset
2517
2518
2519\begin_inset space ~
2520\end_inset
2521
2522
2523\begin_inset space ~
2524\end_inset
2525
2526
2527\begin_inset space ~
2528\end_inset
2529
2530
2531\begin_inset space ~
2532\end_inset
2533
2534
2535\begin_inset space ~
2536\end_inset
2537
2538
2539\begin_inset space ~
2540\end_inset
2541
2542
2543\begin_inset space ~
2544\end_inset
2545
2546
2547\begin_inset space ~
2548\end_inset
2549
2550
2551\begin_inset space ~
2552\end_inset
2553
2554
2555\begin_inset space ~
2556\end_inset
2557
2558
2559\begin_inset space ~
2560\end_inset
2561
2562
2563\begin_inset space ~
2564\end_inset
2565
2566
2567\begin_inset space ~
2568\end_inset
2569
2570
2571\begin_inset space ~
2572\end_inset
2573
2574
2575\begin_inset space ~
2576\end_inset
2577
2578
2579\begin_inset space ~
2580\end_inset
2581
2582
2583\begin_inset space ~
2584\end_inset
2585
2586
2587\begin_inset space ~
2588\end_inset
2589
2590
2591\begin_inset space ~
2592\end_inset
2593
2594
2595\begin_inset space ~
2596\end_inset
2597
2598
2599\begin_inset space ~
2600\end_inset
2601
2602
2603\begin_inset space ~
2604\end_inset
2605
2606
2607\begin_inset space ~
2608\end_inset
2609
2610
2611\begin_inset space ~
2612\end_inset
2613
2614
2615\begin_inset space ~
2616\end_inset
2617
2618
2619\begin_inset space ~
2620\end_inset
2621
2622
2623\begin_inset space ~
2624\end_inset
2625
2626
2627\begin_inset space ~
2628\end_inset
2629
2630
2631\begin_inset space ~
2632\end_inset
2633
2634
2635\begin_inset space ~
2636\end_inset
2637
2638
2639\begin_inset space ~
2640\end_inset
2641
2642
2643\begin_inset space ~
2644\end_inset
2645
2646-ZZZ616464YY
2647\begin_inset space ~
2648\end_inset
2649
2650
2651\begin_inset space ~
2652\end_inset
2653
2654
2655\begin_inset space ~
2656\end_inset
2657
2658
2659\begin_inset space ~
2660\end_inset
2661
2662
2663\begin_inset space ~
2664\end_inset
2665
2666 
2667\begin_inset Quotes eld
2668\end_inset
2669
2670add
2671\begin_inset Quotes erd
2672\end_inset
2673
2674
2675\end_layout
2676
2677\begin_layout Standard
2678
2679\family typewriter
2680\begin_inset space ~
2681\end_inset
2682
2683
2684\begin_inset space ~
2685\end_inset
2686
2687
2688\begin_inset space ~
2689\end_inset
2690
2691
2692\begin_inset space ~
2693\end_inset
2694
2695
2696\begin_inset space ~
2697\end_inset
2698
2699
2700\begin_inset space ~
2701\end_inset
2702
2703
2704\begin_inset space ~
2705\end_inset
2706
2707
2708\begin_inset space ~
2709\end_inset
2710
2711
2712\begin_inset space ~
2713\end_inset
2714
2715
2716\begin_inset space ~
2717\end_inset
2718
2719
2720\begin_inset space ~
2721\end_inset
2722
2723
2724\begin_inset space ~
2725\end_inset
2726
2727
2728\begin_inset space ~
2729\end_inset
2730
2731
2732\begin_inset space ~
2733\end_inset
2734
2735
2736\begin_inset space ~
2737\end_inset
2738
2739
2740\begin_inset space ~
2741\end_inset
2742
2743
2744\begin_inset space ~
2745\end_inset
2746
2747
2748\begin_inset space ~
2749\end_inset
2750
2751
2752\begin_inset space ~
2753\end_inset
2754
2755
2756\begin_inset space ~
2757\end_inset
2758
2759
2760\begin_inset space ~
2761\end_inset
2762
2763
2764\begin_inset space ~
2765\end_inset
2766
2767
2768\begin_inset space ~
2769\end_inset
2770
2771
2772\begin_inset space ~
2773\end_inset
2774
2775
2776\begin_inset space ~
2777\end_inset
2778
2779
2780\begin_inset space ~
2781\end_inset
2782
2783
2784\begin_inset space ~
2785\end_inset
2786
2787
2788\begin_inset space ~
2789\end_inset
2790
2791
2792\begin_inset space ~
2793\end_inset
2794
2795
2796\begin_inset space ~
2797\end_inset
2798
2799
2800\begin_inset space ~
2801\end_inset
2802
2803
2804\begin_inset space ~
2805\end_inset
2806
2807
2808\begin_inset space ~
2809\end_inset
2810
2811
2812\begin_inset space ~
2813\end_inset
2814
2815
2816\begin_inset space ~
2817\end_inset
2818
2819
2820\begin_inset space ~
2821\end_inset
2822
2823
2824\begin_inset space ~
2825\end_inset
2826
2827
2828\begin_inset space ~
2829\end_inset
2830
2831
2832\begin_inset space ~
2833\end_inset
2834
2835
2836\begin_inset space ~
2837\end_inset
2838
2839-ZZZ6e6fYY
2840\begin_inset space ~
2841\end_inset
2842
2843
2844\begin_inset space ~
2845\end_inset
2846
2847
2848\begin_inset space ~
2849\end_inset
2850
2851
2852\begin_inset space ~
2853\end_inset
2854
2855
2856\begin_inset space ~
2857\end_inset
2858
2859 
2860\begin_inset Quotes eld
2861\end_inset
2862
2863no
2864\begin_inset Quotes erd
2865\end_inset
2866
2867
2868\end_layout
2869
2870\begin_layout Standard
2871
2872\family typewriter
2873\begin_inset space ~
2874\end_inset
2875
2876
2877\begin_inset space ~
2878\end_inset
2879
2880
2881\begin_inset space ~
2882\end_inset
2883
2884
2885\begin_inset space ~
2886\end_inset
2887
2888
2889\begin_inset space ~
2890\end_inset
2891
2892
2893\begin_inset space ~
2894\end_inset
2895
2896
2897\begin_inset space ~
2898\end_inset
2899
2900
2901\begin_inset space ~
2902\end_inset
2903
2904
2905\begin_inset space ~
2906\end_inset
2907
2908
2909\begin_inset space ~
2910\end_inset
2911
2912
2913\begin_inset space ~
2914\end_inset
2915
2916
2917\begin_inset space ~
2918\end_inset
2919
2920
2921\begin_inset space ~
2922\end_inset
2923
2924
2925\begin_inset space ~
2926\end_inset
2927
2928
2929\begin_inset space ~
2930\end_inset
2931
2932
2933\begin_inset space ~
2934\end_inset
2935
2936
2937\begin_inset space ~
2938\end_inset
2939
2940
2941\begin_inset space ~
2942\end_inset
2943
2944
2945\begin_inset space ~
2946\end_inset
2947
2948
2949\begin_inset space ~
2950\end_inset
2951
2952
2953\begin_inset space ~
2954\end_inset
2955
2956
2957\begin_inset space ~
2958\end_inset
2959
2960
2961\begin_inset space ~
2962\end_inset
2963
2964
2965\begin_inset space ~
2966\end_inset
2967
2968
2969\begin_inset space ~
2970\end_inset
2971
2972
2973\begin_inset space ~
2974\end_inset
2975
2976
2977\begin_inset space ~
2978\end_inset
2979
2980
2981\begin_inset space ~
2982\end_inset
2983
2984
2985\begin_inset space ~
2986\end_inset
2987
2988
2989\begin_inset space ~
2990\end_inset
2991
2992
2993\begin_inset space ~
2994\end_inset
2995
2996
2997\begin_inset space ~
2998\end_inset
2999
3000
3001\begin_inset space ~
3002\end_inset
3003
3004
3005\begin_inset space ~
3006\end_inset
3007
3008-ZZZ3120322033YY
3009\begin_inset space ~
3010\end_inset
3011
3012
3013\begin_inset space ~
3014\end_inset
3015
3016
3017\begin_inset space ~
3018\end_inset
3019
3020
3021\begin_inset space ~
3022\end_inset
3023
3024
3025\begin_inset space ~
3026\end_inset
3027
3028 
3029\begin_inset Quotes eld
3030\end_inset
3031
30321 2 3
3033\begin_inset Quotes erd
3034\end_inset
3035
3036
3037\begin_inset Newline newline
3038\end_inset
3039
3040
3041\end_layout
3042
3043\begin_layout Standard
3044The header is
3045\begin_inset Quotes eld
3046\end_inset
3047
3048ZZZ
3049\begin_inset Quotes erd
3050\end_inset
3051
3052, the trailer is
3053\begin_inset Quotes eld
3054\end_inset
3055
3056YY
3057\begin_inset Quotes erd
3058\end_inset
3059
3060.
3061\begin_inset Newline newline
3062\end_inset
3063
3064
3065\begin_inset Newline newline
3066\end_inset
3067
3068
3069\end_layout
3070
3071\begin_layout Subsubsection*
3072Tasks for the server
3073\end_layout
3074
3075\begin_layout Standard
3076The server parses the message from the client, executes the command and
3077 sends the results back.
3078 The commands the the server recognizes are:
3079\end_layout
3080
3081\begin_layout Itemize
3082
3083\family typewriter
3084create
3085\begin_inset space ~
3086\end_inset
3087
3088
3089\begin_inset space ~
3090\end_inset
3091
3092
3093\begin_inset space ~
3094\end_inset
3095
3096:
3097\family default
3098 create a pool
3099\end_layout
3100
3101\begin_layout Itemize
3102
3103\family typewriter
3104status
3105\begin_inset space ~
3106\end_inset
3107
3108
3109\begin_inset space ~
3110\end_inset
3111
3112
3113\begin_inset space ~
3114\end_inset
3115
3116:
3117\family default
3118 send the status of the pool
3119\end_layout
3120
3121\begin_layout Itemize
3122
3123\family typewriter
3124purge
3125\begin_inset space ~
3126\end_inset
3127
3128
3129\begin_inset space ~
3130\end_inset
3131
3132
3133\begin_inset space ~
3134\end_inset
3135
3136
3137\begin_inset space ~
3138\end_inset
3139
3140:
3141\family default
3142 remove the pool
3143\end_layout
3144
3145\begin_layout Itemize
3146
3147\family typewriter
3148add
3149\begin_inset space ~
3150\end_inset
3151
3152
3153\begin_inset space ~
3154\end_inset
3155
3156
3157\begin_inset space ~
3158\end_inset
3159
3160
3161\begin_inset space ~
3162\end_inset
3163
3164
3165\begin_inset space ~
3166\end_inset
3167
3168
3169\begin_inset space ~
3170\end_inset
3171
3172:
3173\family default
3174 add a line, taken from the
3175\begin_inset Quotes eld
3176\end_inset
3177
3178value
3179\begin_inset Quotes erd
3180\end_inset
3181
3182 field of the message
3183\end_layout
3184
3185\begin_layout Itemize
3186
3187\family typewriter
3188next
3189\begin_inset space ~
3190\end_inset
3191
3192
3193\begin_inset space ~
3194\end_inset
3195
3196
3197\begin_inset space ~
3198\end_inset
3199
3200
3201\begin_inset space ~
3202\end_inset
3203
3204
3205\begin_inset space ~
3206\end_inset
3207
3208:
3209\family default
3210 send the next line together with the
3211\begin_inset Quotes eld
3212\end_inset
3213
3214key
3215\begin_inset Quotes erd
3216\end_inset
3217
3218 and
3219\begin_inset Quotes eld
3220\end_inset
3221
3222committed
3223\begin_inset Quotes erd
3224\end_inset
3225
3226.
3227 Take care of
3228\begin_inset Quotes eld
3229\end_inset
3230
3231multi
3232\begin_inset Quotes erd
3233\end_inset
3234
3235
3236\end_layout
3237
3238\begin_layout Itemize
3239
3240\family typewriter
3241remove
3242\begin_inset space ~
3243\end_inset
3244
3245
3246\begin_inset space ~
3247\end_inset
3248
3249
3250\begin_inset space ~
3251\end_inset
3252
3253:
3254\family default
3255 remove a line with key given in the
3256\begin_inset Quotes eld
3257\end_inset
3258
3259value
3260\begin_inset Quotes erd
3261\end_inset
3262
3263 field
3264\end_layout
3265
3266\begin_layout Itemize
3267
3268\family typewriter
3269dump
3270\begin_inset space ~
3271\end_inset
3272
3273
3274\begin_inset space ~
3275\end_inset
3276
3277
3278\begin_inset space ~
3279\end_inset
3280
3281
3282\begin_inset space ~
3283\end_inset
3284
3285
3286\begin_inset space ~
3287\end_inset
3288
3289:
3290\family default
3291 return next line to be dumped, together with
3292\begin_inset Quotes eld
3293\end_inset
3294
3295key
3296\begin_inset Quotes erd
3297\end_inset
3298
3299 and
3300\begin_inset Quotes eld
3301\end_inset
3302
3303committed
3304\begin_inset Quotes erd
3305\end_inset
3306
3307
3308\end_layout
3309
3310\begin_layout Itemize
3311
3312\family typewriter
3313pools
3314\begin_inset space ~
3315\end_inset
3316
3317
3318\begin_inset space ~
3319\end_inset
3320
3321
3322\begin_inset space ~
3323\end_inset
3324
3325
3326\begin_inset space ~
3327\end_inset
3328
3329:
3330\family default
3331 return the names of the available pools for the user
3332\end_layout
3333
3334\begin_layout Standard
3335Not accidentally: these commands have the same names as the commands for
3336 the client, but the context is different.
3337\end_layout
3338
3339\begin_layout Subsubsection*
3340Communication from server to client
3341\end_layout
3342
3343\begin_layout Standard
3344The server parses the message from the client and sends the result back,
3345 using
3346\begin_inset Quotes eld
3347\end_inset
3348
3349Content-type: text/plain
3350\begin_inset Quotes erd
3351\end_inset
3352
3353.
3354 The server puts the following string in the output:
3355\begin_inset Newline newline
3356\end_inset
3357
3358
3359\begin_inset Newline newline
3360\end_inset
3361
3362
3363\family typewriter
3364STOPOS:
3365\family default
3366
3367\begin_inset Newline newline
3368\end_inset
3369
3370
3371\begin_inset Newline newline
3372\end_inset
3373
3374This string is immediately followed by the desired output, which consists
3375 of the following fields:
3376\end_layout
3377
3378\begin_layout Itemize
3379
3380\family typewriter
3381return message
3382\begin_inset space ~
3383\end_inset
3384
3385:
3386\family default
3387 
3388\begin_inset Quotes eld
3389\end_inset
3390
3391OK
3392\begin_inset Quotes erd
3393\end_inset
3394
3395 or some error message
3396\end_layout
3397
3398\begin_layout Itemize
3399
3400\family typewriter
3401key
3402\begin_inset space ~
3403\end_inset
3404
3405
3406\begin_inset space ~
3407\end_inset
3408
3409
3410\begin_inset space ~
3411\end_inset
3412
3413
3414\begin_inset space ~
3415\end_inset
3416
3417
3418\begin_inset space ~
3419\end_inset
3420
3421
3422\begin_inset space ~
3423\end_inset
3424
3425
3426\begin_inset space ~
3427\end_inset
3428
3429
3430\begin_inset space ~
3431\end_inset
3432
3433
3434\begin_inset space ~
3435\end_inset
3436
3437
3438\begin_inset space ~
3439\end_inset
3440
3441
3442\begin_inset space ~
3443\end_inset
3444
3445
3446\begin_inset space ~
3447\end_inset
3448
3449:
3450\family default
3451 key of the record just retrieved or empty
3452\end_layout
3453
3454\begin_layout Itemize
3455
3456\family typewriter
3457committed
3458\begin_inset space ~
3459\end_inset
3460
3461
3462\begin_inset space ~
3463\end_inset
3464
3465
3466\begin_inset space ~
3467\end_inset
3468
3469
3470\begin_inset space ~
3471\end_inset
3472
3473
3474\begin_inset space ~
3475\end_inset
3476
3477
3478\begin_inset space ~
3479\end_inset
3480
3481:
3482\family default
3483 number of times the record has been committed or empty
3484\end_layout
3485
3486\begin_layout Itemize
3487
3488\family typewriter
3489count
3490\begin_inset space ~
3491\end_inset
3492
3493
3494\begin_inset space ~
3495\end_inset
3496
3497
3498\begin_inset space ~
3499\end_inset
3500
3501
3502\begin_inset space ~
3503\end_inset
3504
3505
3506\begin_inset space ~
3507\end_inset
3508
3509
3510\begin_inset space ~
3511\end_inset
3512
3513
3514\begin_inset space ~
3515\end_inset
3516
3517
3518\begin_inset space ~
3519\end_inset
3520
3521
3522\begin_inset space ~
3523\end_inset
3524
3525
3526\begin_inset space ~
3527\end_inset
3528
3529:
3530\family default
3531 number of records added so far or empty
3532\end_layout
3533
3534\begin_layout Itemize
3535
3536\family typewriter
3537present
3538\begin_inset space ~
3539\end_inset
3540
3541
3542\begin_inset space ~
3543\end_inset
3544
3545
3546\begin_inset space ~
3547\end_inset
3548
3549
3550\begin_inset space ~
3551\end_inset
3552
3553
3554\begin_inset space ~
3555\end_inset
3556
3557
3558\begin_inset space ~
3559\end_inset
3560
3561
3562\begin_inset space ~
3563\end_inset
3564
3565
3566\begin_inset space ~
3567\end_inset
3568
3569:
3570\family default
3571 number of records present or empty
3572\end_layout
3573
3574\begin_layout Itemize
3575
3576\family typewriter
3577present0
3578\begin_inset space ~
3579\end_inset
3580
3581
3582\begin_inset space ~
3583\end_inset
3584
3585
3586\begin_inset space ~
3587\end_inset
3588
3589
3590\begin_inset space ~
3591\end_inset
3592
3593
3594\begin_inset space ~
3595\end_inset
3596
3597
3598\begin_inset space ~
3599\end_inset
3600
3601
3602\begin_inset space ~
3603\end_inset
3604
3605:
3606\family default
3607 number of records never been committed or empty
3608\end_layout
3609
3610\begin_layout Itemize
3611
3612\family typewriter
3613value
3614\begin_inset space ~
3615\end_inset
3616
3617
3618\begin_inset space ~
3619\end_inset
3620
3621
3622\begin_inset space ~
3623\end_inset
3624
3625
3626\begin_inset space ~
3627\end_inset
3628
3629
3630\begin_inset space ~
3631\end_inset
3632
3633
3634\begin_inset space ~
3635\end_inset
3636
3637
3638\begin_inset space ~
3639\end_inset
3640
3641
3642\begin_inset space ~
3643\end_inset
3644
3645
3646\begin_inset space ~
3647\end_inset
3648
3649
3650\begin_inset space ~
3651\end_inset
3652
3653:
3654\family default
3655 return value or empty
3656\end_layout
3657
3658\begin_layout Itemize
3659
3660\family typewriter
3661new line character
3662\end_layout
3663
3664\begin_layout Standard
3665The client will search for the string
3666\begin_inset Quotes eld
3667\end_inset
3668
3669STOPOS:
3670\begin_inset Quotes erd
3671\end_inset
3672
3673 and use the string immediately following as the result.
3674 The developer can, if she wishes, print extra information and have a look
3675 at the total output of the server, as long as she does not use the string
3676 
3677\begin_inset Quotes eld
3678\end_inset
3679
3680STOPOS:
3681\begin_inset Quotes erd
3682\end_inset
3683
3684.
3685 The fields are encoded, just like is done with the message from the client
3686 to the server, only the separator here is
3687\begin_inset Quotes eld
3688\end_inset
3689
3690+
3691\begin_inset Quotes erd
3692\end_inset
3693
3694 for no special reason.
3695 
3696\end_layout
3697
3698\begin_layout Subsection*
3699Internals of the client
3700\end_layout
3701
3702\begin_layout Standard
3703The client parses the command line, using getopt(3).
3704 It performs some basic checks on the parameters given, and sends an appropriate
3705 command to the server, using libcurl(3).
3706 The output from the server is parsed and transformed into statements that
3707 can be executed in the shell.
3708 These commands set environment variables, with names starting with
3709\begin_inset Quotes eld
3710\end_inset
3711
3712STOPOS_
3713\begin_inset Quotes erd
3714\end_inset
3715
3716.
3717\end_layout
3718
3719\begin_layout Subsection*
3720Internals of the server
3721\end_layout
3722
3723\begin_layout Standard
3724The server parses the command sent from the client and an http-server (apache,
3725 thttpd), executes it and sends the results back to the client.
3726 Most of the interesting work is done in the class
3727\begin_inset Quotes eld
3728\end_inset
3729
3730stopos_pool
3731\begin_inset Quotes erd
3732\end_inset
3733
3734.
3735 This class, from which a database-specific class has to be derived, maintains
3736 a double linked list of lines.
3737 The addressing of the lines is based on keys that are computed from the
3738 counter that is incremented each time a line is added.
3739 The derived class has to deal with some basic I/O operations: creating,
3740 opening, closing and removing of the database; reading, writing and removing
3741 records.
3742 In this code, four database implementations are provided:
3743\end_layout
3744
3745\begin_layout Itemize
3746based on gdbm
3747\begin_inset Foot
3748status open
3749
3750\begin_layout Plain Layout
3751http://www.gnu.org.ua/software/gdbm/
3752\end_layout
3753
3754\end_inset
3755
3756
3757\end_layout
3758
3759\begin_layout Itemize
3760based om mysql
3761\begin_inset Foot
3762status open
3763
3764\begin_layout Plain Layout
3765http://www.mysql.com/
3766\end_layout
3767
3768\end_inset
3769
3770
3771\end_layout
3772
3773\begin_layout Itemize
3774based on a flat ascii file
3775\end_layout
3776
3777\begin_layout Itemize
3778based on files in a folder, each file containing one line
3779\end_layout
3780
3781\begin_layout Standard
3782The client determines which implementation the server has to use.
3783 Storage of a record is based on the key, but, if desirable, the derived
3784 class can provide a suitable slot for storing the record.
3785 This feature is used in the
3786\begin_inset Quotes eld
3787\end_inset
3788
3789flat file
3790\begin_inset Quotes erd
3791\end_inset
3792
3793 solution, in order to be able to re-use space occupied by removed records.
3794 
3795\end_layout
3796
3797\begin_layout Subsubsection*
3798Storage of lines
3799\end_layout
3800
3801\begin_layout Standard
3802Lines are stored in records, which also contain extra fields.
3803 The records are hexadecimal coded, just like the communication from client
3804 to server.
3805 The separation character is here '/'.
3806 Records are stored using unique keys, generated by the class
3807\begin_inset Quotes eld
3808\end_inset
3809
3810stopos_pool
3811\begin_inset Quotes erd
3812\end_inset
3813
3814.
3815 The derived class has no knowledge about the meaning of the records: it's
3816 only task is to store, retrieve and delete them.
3817\end_layout
3818
3819\begin_layout Subsection*
3820Performance
3821\end_layout
3822
3823\begin_layout Standard
3824A program
3825\begin_inset Quotes eld
3826\end_inset
3827
3828test_pool.cpp
3829\begin_inset Quotes erd
3830\end_inset
3831
3832 has been provided which runs all four implementations, without an http
3833 server, but directly calling the functions from the class
3834\begin_inset Quotes eld
3835\end_inset
3836
3837stopos_pool
3838\begin_inset Quotes erd
3839\end_inset
3840
3841 and reports how much time a implementation takes.
3842 Furthermore, it does some consistency checks.
3843 The pool consists of a few hundreds of lines.
3844 The results are
3845\begin_inset Foot
3846status open
3847
3848\begin_layout Plain Layout
3849The program was run on a Linux workstation, equipped with a
3850\begin_inset Quotes eld
3851\end_inset
3852
3853normal
3854\begin_inset Quotes erd
3855\end_inset
3856
3857 (no SSD) disk
3858\end_layout
3859
3860\end_inset
3861
3862:
3863\begin_inset Newline newline
3864\end_inset
3865
3866
3867\begin_inset Newline newline
3868\end_inset
3869
3870
3871\family typewriter
3872database
3873\begin_inset space ~
3874\end_inset
3875
3876
3877\begin_inset space ~
3878\end_inset
3879
3880
3881\begin_inset space ~
3882\end_inset
3883
3884time (sec)
3885\begin_inset Newline newline
3886\end_inset
3887
3888gdbm
3889\begin_inset space ~
3890\end_inset
3891
3892
3893\begin_inset space ~
3894\end_inset
3895
3896
3897\begin_inset space ~
3898\end_inset
3899
3900
3901\begin_inset space ~
3902\end_inset
3903
3904
3905\begin_inset space ~
3906\end_inset
3907
3908
3909\begin_inset space ~
3910\end_inset
3911
3912
3913\begin_inset space ~
3914\end_inset
3915
3916
3917\begin_inset space ~
3918\end_inset
3919
39203.31
3921\end_layout
3922
3923\begin_layout Standard
3924
3925\family typewriter
3926flatfile
3927\begin_inset space ~
3928\end_inset
3929
3930
3931\begin_inset space ~
3932\end_inset
3933
3934
3935\begin_inset space ~
3936\end_inset
3937
39380.19
3939\end_layout
3940
3941\begin_layout Standard
3942
3943\family typewriter
3944files
3945\begin_inset space ~
3946\end_inset
3947
3948
3949\begin_inset space ~
3950\end_inset
3951
3952
3953\begin_inset space ~
3954\end_inset
3955
3956
3957\begin_inset space ~
3958\end_inset
3959
3960
3961\begin_inset space ~
3962\end_inset
3963
3964
3965\begin_inset space ~
3966\end_inset
3967
39680.73
3969\end_layout
3970
3971\begin_layout Standard
3972
3973\family typewriter
3974mysql
3975\begin_inset space ~
3976\end_inset
3977
3978
3979\begin_inset space ~
3980\end_inset
3981
3982
3983\begin_inset space ~
3984\end_inset
3985
3986
3987\begin_inset space ~
3988\end_inset
3989
3990
3991\begin_inset space ~
3992\end_inset
3993
3994
3995\begin_inset space ~
3996\end_inset
3997
39984.7
3999\family default
40001
4001\begin_inset Newline newline
4002\end_inset
4003
4004 
4005\begin_inset Newline newline
4006\end_inset
4007
4008The winner is clearly the
4009\begin_inset Quotes eld
4010\end_inset
4011
4012flat file
4013\begin_inset Quotes erd
4014\end_inset
4015
4016 implementation.
4017 Disadvantage of this solution (and also of the
4018\begin_inset Quotes eld
4019\end_inset
4020
4021files
4022\begin_inset Quotes erd
4023\end_inset
4024
4025 and
4026\begin_inset Quotes eld
4027\end_inset
4028
4029mysql
4030\begin_inset Quotes erd
4031\end_inset
4032
4033 solution) is the amount of space that is used for the database: each line
4034 takes 4 Kbyte.
4035 The
4036\begin_inset Quotes eld
4037\end_inset
4038
4039gdbm
4040\begin_inset Quotes erd
4041\end_inset
4042
4043 solution does not have this problem.
4044 
4045\begin_inset Newline newline
4046\end_inset
4047
4048Normally, the stopos software will be used using an http server, so it is
4049 more relevant to have look at the timings in this case.
4050 As an example, that man page of bash (5459 lines) was put in the pool,
4051 and all lines were retrieved and removed:
4052\begin_inset Newline newline
4053\end_inset
4054
4055
4056\begin_inset Newline newline
4057\end_inset
4058
4059
4060\family typewriter
4061database
4062\begin_inset space ~
4063\end_inset
4064
4065
4066\begin_inset space ~
4067\end_inset
4068
4069time(sec)
4070\end_layout
4071
4072\begin_layout Standard
4073
4074\family typewriter
4075gdbm
4076\begin_inset space ~
4077\end_inset
4078
4079
4080\begin_inset space ~
4081\end_inset
4082
4083
4084\begin_inset space ~
4085\end_inset
4086
4087
4088\begin_inset space ~
4089\end_inset
4090
4091
4092\begin_inset space ~
4093\end_inset
4094
4095
4096\begin_inset space ~
4097\end_inset
4098
4099
4100\begin_inset space ~
4101\end_inset
4102
4103187
4104\end_layout
4105
4106\begin_layout Standard
4107
4108\family typewriter
4109flatfile
4110\begin_inset space ~
4111\end_inset
4112
4113
4114\begin_inset space ~
4115\end_inset
4116
4117
4118\begin_inset space ~
4119\end_inset
4120
4121182
4122\end_layout
4123
4124\begin_layout Standard
4125
4126\family typewriter
4127files
4128\begin_inset space ~
4129\end_inset
4130
4131
4132\begin_inset space ~
4133\end_inset
4134
4135
4136\begin_inset space ~
4137\end_inset
4138
4139
4140\begin_inset space ~
4141\end_inset
4142
4143
4144\begin_inset space ~
4145\end_inset
4146
4147
4148\begin_inset space ~
4149\end_inset
4150
4151190
4152\end_layout
4153
4154\begin_layout Standard
4155
4156\family typewriter
4157mysql
4158\begin_inset space ~
4159\end_inset
4160
4161
4162\begin_inset space ~
4163\end_inset
4164
4165
4166\begin_inset space ~
4167\end_inset
4168
4169
4170\begin_inset space ~
4171\end_inset
4172
4173
4174\begin_inset space ~
4175\end_inset
4176
4177
4178\begin_inset space ~
4179\end_inset
4180
4181
4182\family default
4183309
4184\begin_inset Newline newline
4185\end_inset
4186
4187
4188\begin_inset Newline newline
4189\end_inset
4190
4191These timings show, that for performance reasons, there is not much difference
4192 between
4193\begin_inset Quotes eld
4194\end_inset
4195
4196gdbm
4197\begin_inset Quotes erd
4198\end_inset
4199
4200,
4201\begin_inset Quotes eld
4202\end_inset
4203
4204flatfile
4205\begin_inset Quotes erd
4206\end_inset
4207
4208 and
4209\begin_inset Quotes eld
4210\end_inset
4211
4212files
4213\begin_inset Quotes erd
4214\end_inset
4215
4216.
4217 We choose the
4218\begin_inset Quotes eld
4219\end_inset
4220
4221gdbm
4222\begin_inset Quotes erd
4223\end_inset
4224
4225 implementation, because of it's economy with disk space used.
4226\end_layout
4227
4228\begin_layout Subsection*
4229Installation
4230\end_layout
4231
4232\begin_layout Standard
4233The source tree contains a file
4234\begin_inset Quotes eld
4235\end_inset
4236
4237Make.inc
4238\begin_inset Quotes erd
4239\end_inset
4240
4241, defining variables that determine the version of the program, the installation
4242 directories and which database versions should be produced.
4243 This file will be included in the Makefile.
4244 A trivial script
4245\begin_inset Quotes eld
4246\end_inset
4247
4248makeit
4249\begin_inset Quotes erd
4250\end_inset
4251
4252 sets the umask and calls make(1) to create and install the executables.
4253 The script
4254\begin_inset Quotes eld
4255\end_inset
4256
4257clean
4258\begin_inset Quotes erd
4259\end_inset
4260
4261 executes a
4262\begin_inset Quotes eld
4263\end_inset
4264
4265make clean
4266\begin_inset Quotes erd
4267\end_inset
4268
4269 to get rid of compilation products.
4270\end_layout
4271
4272\begin_layout Subsubsection*
4273Extra tools
4274\end_layout
4275
4276\begin_layout Standard
4277The source tree also contains the scripts
4278\begin_inset Quotes eld
4279\end_inset
4280
4281sara-get-num-cores
4282\begin_inset Quotes erd
4283\end_inset
4284
4285 and
4286\begin_inset Quotes eld
4287\end_inset
4288
4289sara-get-mem-size
4290\begin_inset Quotes erd
4291\end_inset
4292
4293, along with their man pages.
4294 Sara-get-num-cores prints the number of available cores, while sara-get-mem-siz
4295e prints the amount of memory.
4296 Both numbers can be used to determine how many processes should run in
4297 parallel on a given node.
4298 These scripts and man pages will be installed also by calling
4299\begin_inset Quotes eld
4300\end_inset
4301
4302./makeit
4303\begin_inset Quotes erd
4304\end_inset
4305
4306.
4307\end_layout
4308
4309\begin_layout Subsection*
4310Example of a job using stopos
4311\end_layout
4312
4313\begin_layout Standard
4314\begin_inset listings
4315inline false
4316status open
4317
4318\begin_layout Plain Layout
4319
43201  #PBS -lnodes=1 -lwalltime=3:00:00 
4321\end_layout
4322
4323\begin_layout Plain Layout
4324
43252  module load stopos 
4326\end_layout
4327
4328\begin_layout Plain Layout
4329
43303  ncores=`sara-get-num-cores` 
4331\end_layout
4332
4333\begin_layout Plain Layout
4334
43354  for ((i=1; i<=ncores; i++)) ; do 
4336\end_layout
4337
4338\begin_layout Plain Layout
4339
43405  ( 
4341\end_layout
4342
4343\begin_layout Plain Layout
4344
43456    for ((j=1; j<=10; j++)) ; do 
4346\end_layout
4347
4348\begin_layout Plain Layout
4349
43507      stopos next -p pool1 
4351\end_layout
4352
4353\begin_layout Plain Layout
4354
43558        if [ "$STOPOS_RC" != "OK" ]; then 
4356\end_layout
4357
4358\begin_layout Plain Layout
4359
43609        break
4361\end_layout
4362
4363\begin_layout Plain Layout
4364
436510      fi
4366\end_layout
4367
4368\begin_layout Plain Layout
4369
437011      eval "scanner $STOPOS_VALUE"
4371\end_layout
4372
4373\begin_layout Plain Layout
4374
437512      stopos remove -p pool1
4376\end_layout
4377
4378\begin_layout Plain Layout
4379
438013    done
4381\end_layout
4382
4383\begin_layout Plain Layout
4384
438514  ) &
4386\end_layout
4387
4388\begin_layout Plain Layout
4389
439015  done
4391\end_layout
4392
4393\begin_layout Plain Layout
4394
439516  wait
4396\end_layout
4397
4398\end_inset
4399
4400
4401\end_layout
4402
4403\begin_layout Itemize
44041 Note: we do not specify how many cores a node should have.
4405\end_layout
4406
4407\begin_layout Itemize
44082 Load the stopos module.
4409\end_layout
4410
4411\begin_layout Itemize
44123 Determine the number of cores available on this node: Module stopos contains
4413 'sara-get-num-cores': this command prints the number of cores available
4414 on the node.
4415 The command 'sara-get-mem-size' prints the available amount of memory in
4416 Mbytes.
4417 We do not use it here, but this number could be interesting when running
4418 memory intensive programs.
4419\end_layout
4420
4421\begin_layout Itemize
44224 Loop over the number of cores.
4423 The content of the loop is placed in the background (see 14)
4424\end_layout
4425
4426\begin_layout Itemize
44276 Loop over the number of the desired number of repetitions.
4428 
4429\end_layout
4430
4431\begin_layout Itemize
44327 Here we get the value of the next line, delivered in the environment variable
4433 STOPOS_VALUE.
4434 The environment variable STOPOS_RC has the value "OK" if everything went
4435 ok.
4436 STOPOS_KEY contains a value that can be used to remove the line from the
4437 pool.
4438 We use the pool 'pool1', created before.
4439\end_layout
4440
4441\begin_layout Itemize
44428 ..
4443 10 Break out the loop if stopos fails (usually because there are no lines
4444 left).
4445 
4446\end_layout
4447
4448\begin_layout Itemize
444911 Use the parameters in $STOPOS_VALUE.
4450 The 'eval' and quotes ensure that spaces and quotes in the parameters are
4451 correctly interpreted.
4452 
4453\end_layout
4454
4455\begin_layout Itemize
445612 Remove the line we just processed from the pool.
4457 
4458\end_layout
4459
4460\begin_layout Itemize
446114 Notice the &: this takes care that ncores instantiations of scanner will
4462 be started in parallel.
4463 
4464\end_layout
4465
4466\begin_layout Itemize
446716 The 'wait' ensures that the job script will only finish after all background
4468 processes are ended.
4469\end_layout
4470
4471\begin_layout Itemize
4472The job can be submitted a number of times with one 'qsub' command, for
4473 example:
4474\family typewriter
4475qsub -t 1-100 job
4476\end_layout
4477
4478\begin_layout Subsection*
4479Man page of stopos
4480\end_layout
4481
4482\begin_layout Standard
4483
4484\family typewriter
4485\begin_inset listings
4486inline false
4487status open
4488
4489\begin_layout Plain Layout
4490
4491\end_layout
4492
4493\begin_layout Plain Layout
4494
4495\end_layout
4496
4497\begin_layout Plain Layout
4498
4499STOPOS(1)                                                            STOPOS(1)
4500\end_layout
4501
4502\begin_layout Plain Layout
4503
4504\end_layout
4505
4506\begin_layout Plain Layout
4507
4508\end_layout
4509
4510\begin_layout Plain Layout
4511
4512\end_layout
4513
4514\begin_layout Plain Layout
4515
4516NAME
4517\end_layout
4518
4519\begin_layout Plain Layout
4520
4521       stopos - alias to call the program stoposclient and eval it's output
4522\end_layout
4523
4524\begin_layout Plain Layout
4525
4526\end_layout
4527
4528\begin_layout Plain Layout
4529
4530SYNOPSIS
4531\end_layout
4532
4533\begin_layout Plain Layout
4534
4535       stopos  -h,--help
4536\end_layout
4537
4538\begin_layout Plain Layout
4539
4540\end_layout
4541
4542\begin_layout Plain Layout
4543
4544       stopos  -v,--version
4545\end_layout
4546
4547\begin_layout Plain Layout
4548
4549\end_layout
4550
4551\begin_layout Plain Layout
4552
4553       stopos  create    [ -p,--pool POOL ]
4554\end_layout
4555
4556\begin_layout Plain Layout
4557
4558\end_layout
4559
4560\begin_layout Plain Layout
4561
4562       stopos  status    [ -p,--pool POOL]
4563\end_layout
4564
4565\begin_layout Plain Layout
4566
4567\end_layout
4568
4569\begin_layout Plain Layout
4570
4571       stopos  purge     [ -p,--pool POOL ]
4572\end_layout
4573
4574\begin_layout Plain Layout
4575
4576\end_layout
4577
4578\begin_layout Plain Layout
4579
4580       stopos  pools
4581\end_layout
4582
4583\begin_layout Plain Layout
4584
4585\end_layout
4586
4587\begin_layout Plain Layout
4588
4589       stopos  add       [ -p,--pool POOL ] [ FILENAME ]
4590\end_layout
4591
4592\begin_layout Plain Layout
4593
4594\end_layout
4595
4596\begin_layout Plain Layout
4597
4598       stopos  next      [ -p,--pool POOL ] [ -m,--multi ]
4599\end_layout
4600
4601\begin_layout Plain Layout
4602
4603\end_layout
4604
4605\begin_layout Plain Layout
4606
4607       stopos  remove    [ -p,--pool POOL] [ KEY  ]
4608\end_layout
4609
4610\begin_layout Plain Layout
4611
4612\end_layout
4613
4614\begin_layout Plain Layout
4615
4616       stopos  dump      [ -p,--pool POOL]
4617\end_layout
4618
4619\begin_layout Plain Layout
4620
4621\end_layout
4622
4623\begin_layout Plain Layout
4624
4625       The -q,--quiet flag supresses most output.
4626\end_layout
4627
4628\begin_layout Plain Layout
4629
4630\end_layout
4631
4632\begin_layout Plain Layout
4633
4634DESCRIPTION
4635\end_layout
4636
4637\begin_layout Plain Layout
4638
4639       Stopos  is an alias using the program stoposclient, an utility to
4640 store
4641\end_layout
4642
4643\begin_layout Plain Layout
4644
4645       and retrieve text lines in a pool.
4646  In general, the text lines are used
4647\end_layout
4648
4649\begin_layout Plain Layout
4650
4651       as command parameters, see Examples.
4652\end_layout
4653
4654\begin_layout Plain Layout
4655
4656\end_layout
4657
4658\begin_layout Plain Layout
4659
4660COMMANDS
4661\end_layout
4662
4663\begin_layout Plain Layout
4664
4665       -h,--help
4666\end_layout
4667
4668\begin_layout Plain Layout
4669
4670           Prints usage information.
4671\end_layout
4672
4673\begin_layout Plain Layout
4674
4675\end_layout
4676
4677\begin_layout Plain Layout
4678
4679       -v,--version
4680\end_layout
4681
4682\begin_layout Plain Layout
4683
4684           Prints version.
4685\end_layout
4686
4687\begin_layout Plain Layout
4688
4689\end_layout
4690
4691\begin_layout Plain Layout
4692
4693       create [ -p,--pool POOL ] [ FILENAME ]
4694\end_layout
4695
4696\begin_layout Plain Layout
4697
4698           Creates new pool.
4699  A pool with the same name will be removed first.
4700\end_layout
4701
4702\begin_layout Plain Layout
4703
4704\end_layout
4705
4706\begin_layout Plain Layout
4707
4708       status
4709\end_layout
4710
4711\begin_layout Plain Layout
4712
4713           Prints to standard error       environment variable
4714\end_layout
4715
4716\begin_layout Plain Layout
4717
4718           ---------------------------------------------------
4719\end_layout
4720
4721\begin_layout Plain Layout
4722
4723           total number of lines added      STOPOS_COUNT
4724\end_layout
4725
4726\begin_layout Plain Layout
4727
4728           number of lines present          STOPOS_PRESENT
4729\end_layout
4730
4731\begin_layout Plain Layout
4732
4733           number of lines never committed  STOPOS_PRESENT0
4734\end_layout
4735
4736\begin_layout Plain Layout
4737
4738\end_layout
4739
4740\begin_layout Plain Layout
4741
4742\end_layout
4743
4744\begin_layout Plain Layout
4745
4746       purge [-p,--pool POOL ]
4747\end_layout
4748
4749\begin_layout Plain Layout
4750
4751           Removes the pool.
4752\end_layout
4753
4754\begin_layout Plain Layout
4755
4756\end_layout
4757
4758\begin_layout Plain Layout
4759
4760       pools [-p,--pool POOL ]
4761\end_layout
4762
4763\begin_layout Plain Layout
4764
4765           Lists on standard error the pools available.
4766 The environment
4767\end_layout
4768
4769\begin_layout Plain Layout
4770
4771           variable STOPOS_VALUE is set accordingly.
4772\end_layout
4773
4774\begin_layout Plain Layout
4775
4776\end_layout
4777
4778\begin_layout Plain Layout
4779
4780       add [ -p,--pool POOL ] [ FILENAME ]
4781\end_layout
4782
4783\begin_layout Plain Layout
4784
4785           Adds lines from FILENAME, default from stdin.
4786  The  environ‐
4787\end_layout
4788
4789\begin_layout Plain Layout
4790
4791           ment variables STOPOS_RC and STOPS_KEY are set.
4792\end_layout
4793
4794\begin_layout Plain Layout
4795
4796           NOTE: when stopos is reading from a pipe as in:
4797\end_layout
4798
4799\begin_layout Plain Layout
4800
4801              cat parmfile | stopos add
4802\end_layout
4803
4804\begin_layout Plain Layout
4805
4806           no environment variables are set.
4807\end_layout
4808
4809\begin_layout Plain Layout
4810
4811\end_layout
4812
4813\begin_layout Plain Layout
4814
4815       next [ -p,--pool POOL ] [ -m,--multi ]
4816\end_layout
4817
4818\begin_layout Plain Layout
4819
4820           Gets the next line from the pool.
4821  The environment variables
4822\end_layout
4823
4824\begin_layout Plain Layout
4825
4826           STOPOS_VALUE, STOPOS_COMMITTED and STOPOS_KEY are  set,  see
4827\end_layout
4828
4829\begin_layout Plain Layout
4830
4831           ENVIRONMENT.
4832   By default, the same line in the pool will be
4833\end_layout
4834
4835\begin_layout Plain Layout
4836
4837           produced only once.
4838 When all lines  are  commited,  STOPS_RC
4839\end_layout
4840
4841\begin_layout Plain Layout
4842
4843           will  not  be  equal to OK.
4844 Using the --multi flag, the same
4845\end_layout
4846
4847\begin_layout Plain Layout
4848
4849           line can be produced more than  once,  if  necessary  stopos
4850\end_layout
4851
4852\begin_layout Plain Layout
4853
4854           will  wrap  around.
4855   This  can  be  useful for dealing with
4856\end_layout
4857
4858\begin_layout Plain Layout
4859
4860           crashed jobs.
4861\end_layout
4862
4863\begin_layout Plain Layout
4864
4865\end_layout
4866
4867\begin_layout Plain Layout
4868
4869       remove [-p,--pool POOL ] [ KEY ]
4870\end_layout
4871
4872\begin_layout Plain Layout
4873
4874           Removes the line with key KEY as ready.
4875  If not specified on
4876\end_layout
4877
4878\begin_layout Plain Layout
4879
4880           the  commandline,  the  value  of  environment variable STO‐
4881\end_layout
4882
4883\begin_layout Plain Layout
4884
4885           POS_KEY is used.
4886\end_layout
4887
4888\begin_layout Plain Layout
4889
4890\end_layout
4891
4892\begin_layout Plain Layout
4893
4894       dump [-p,--pool POOL ]
4895\end_layout
4896
4897\begin_layout Plain Layout
4898
4899           Reads the next available line, and puts it, preceded by it's
4900\end_layout
4901
4902\begin_layout Plain Layout
4903
4904           key  and  number of commitments in environment variable STO‐
4905\end_layout
4906
4907\begin_layout Plain Layout
4908
4909           POS_VALUE.
4910 The first call will produce the first line.
4911  When
4912\end_layout
4913
4914\begin_layout Plain Layout
4915
4916           all  lines have been delivered, STOPOS_RC gets a value other
4917\end_layout
4918
4919\begin_layout Plain Layout
4920
4921           than OK.
4922 A subsequent call will  start  the  dump  with  the
4923\end_layout
4924
4925\begin_layout Plain Layout
4926
4927           first line again.
4928\end_layout
4929
4930\begin_layout Plain Layout
4931
4932\end_layout
4933
4934\begin_layout Plain Layout
4935
4936OPTIONS
4937\end_layout
4938
4939\begin_layout Plain Layout
4940
4941       -p,--pool POOL
4942\end_layout
4943
4944\begin_layout Plain Layout
4945
4946           POOL is a unique name of the pool.
4947  Default: pool.
4948  The name
4949\end_layout
4950
4951\begin_layout Plain Layout
4952
4953           of the pool can also be set using the  environment  variable
4954\end_layout
4955
4956\begin_layout Plain Layout
4957
4958           STOPOS_POOL.
4959 The command line flag has precedence.
4960\end_layout
4961
4962\begin_layout Plain Layout
4963
4964\end_layout
4965
4966\begin_layout Plain Layout
4967
4968\end_layout
4969
4970\begin_layout Plain Layout
4971
4972ENVIRONMENT
4973\end_layout
4974
4975\begin_layout Plain Layout
4976
4977       Stopos sets the following environment variables:
4978\end_layout
4979
4980\begin_layout Plain Layout
4981
4982\end_layout
4983
4984\begin_layout Plain Layout
4985
4986       STOPOS_RC if the value is OK, than no errors were found
4987\end_layout
4988
4989\begin_layout Plain Layout
4990
4991\end_layout
4992
4993\begin_layout Plain Layout
4994
4995       STOPOS_KEY  contains  the keyvalue of the line produced with the
4996\end_layout
4997
4998\begin_layout Plain Layout
4999
5000       'next' command
5001\end_layout
5002
5003\begin_layout Plain Layout
5004
5005\end_layout
5006
5007\begin_layout Plain Layout
5008
5009       STOPOS_COMMITTED contains the number of times the line has  been
5010\end_layout
5011
5012\begin_layout Plain Layout
5013
5014       committed
5015\end_layout
5016
5017\begin_layout Plain Layout
5018
5019\end_layout
5020
5021\begin_layout Plain Layout
5022
5023       STOPOS_VALUE contains the line produced by the 'next' command or
5024\end_layout
5025
5026\begin_layout Plain Layout
5027
5028       the result of the 'pools' command
5029\end_layout
5030
5031\begin_layout Plain Layout
5032
5033\end_layout
5034
5035\begin_layout Plain Layout
5036
5037       STOPOS_COUNT STOPOS_PRESENT  STOPOS_PRESENT0  see  the  'status'
5038\end_layout
5039
5040\begin_layout Plain Layout
5041
5042       command above
5043\end_layout
5044
5045\begin_layout Plain Layout
5046
5047\end_layout
5048
5049\begin_layout Plain Layout
5050
5051       Stopos uses the following environment variables:
5052\end_layout
5053
5054\begin_layout Plain Layout
5055
5056\end_layout
5057
5058\begin_layout Plain Layout
5059
5060       STOPOS_POOL see the description of the --pool flag under OPTIONS
5061\end_layout
5062
5063\begin_layout Plain Layout
5064
5065\end_layout
5066
5067\begin_layout Plain Layout
5068
5069       STOPOS_KEY see the 'remove' flag above
5070\end_layout
5071
5072\begin_layout Plain Layout
5073
5074\end_layout
5075
5076\begin_layout Plain Layout
5077
5078       STOPOS_SERVER_URL  the  url  stopos  uses  to access the server,
5079\end_layout
5080
5081\begin_layout Plain Layout
5082
5083       default http://stopos.osd.surfsara.nl/cgi-bin/stoposserver
5084\end_layout
5085
5086\begin_layout Plain Layout
5087
5088\end_layout
5089
5090\begin_layout Plain Layout
5091
5092\end_layout
5093
5094\begin_layout Plain Layout
5095
5096EXAMPLES
5097\end_layout
5098
5099\begin_layout Plain Layout
5100
5101       Create a pool with the first 10 lines from the man page of sed:
5102\end_layout
5103
5104\begin_layout Plain Layout
5105
5106\end_layout
5107
5108\begin_layout Plain Layout
5109
5110            man sed | head > parmfile
5111\end_layout
5112
5113\begin_layout Plain Layout
5114
5115            stopos create
5116\end_layout
5117
5118\begin_layout Plain Layout
5119
5120            stopos add parmfile
5121\end_layout
5122
5123\begin_layout Plain Layout
5124
5125\end_layout
5126
5127\begin_layout Plain Layout
5128
5129           Get a line:
5130\end_layout
5131
5132\begin_layout Plain Layout
5133
5134\end_layout
5135
5136\begin_layout Plain Layout
5137
5138            stopos next
5139\end_layout
5140
5141\begin_layout Plain Layout
5142
5143\end_layout
5144
5145\begin_layout Plain Layout
5146
5147           The environment variable STOPOS_VALUE contains  now  one  of
5148\end_layout
5149
5150\begin_layout Plain Layout
5151
5152           the first ten lines of the man page of sed.
5153\end_layout
5154
5155\begin_layout Plain Layout
5156
5157\end_layout
5158
5159\begin_layout Plain Layout
5160
5161           You can use this in a command like this:
5162\end_layout
5163
5164\begin_layout Plain Layout
5165
5166\end_layout
5167
5168\begin_layout Plain Layout
5169
5170            eval "myparser $STOPOS_VALUE"
5171\end_layout
5172
5173\begin_layout Plain Layout
5174
5175\end_layout
5176
5177\begin_layout Plain Layout
5178
5179           The line can be removed by:
5180\end_layout
5181
5182\begin_layout Plain Layout
5183
5184\end_layout
5185
5186\begin_layout Plain Layout
5187
5188            stopos remove
5189\end_layout
5190
5191\begin_layout Plain Layout
5192
5193\end_layout
5194
5195\begin_layout Plain Layout
5196
5197           Finally, the pool can be completely removed by:
5198\end_layout
5199
5200\begin_layout Plain Layout
5201
5202\end_layout
5203
5204\begin_layout Plain Layout
5205
5206            stopos purge
5207\end_layout
5208
5209\begin_layout Plain Layout
5210
5211\end_layout
5212
5213\begin_layout Plain Layout
5214
5215\end_layout
5216
5217\begin_layout Plain Layout
5218
5219FILES
5220\end_layout
5221
5222\begin_layout Plain Layout
5223
5224       $HOME/.stopos/id
5225\end_layout
5226
5227\begin_layout Plain Layout
5228
5229       This  file  is  set  by  stopos  to  store  an unique id, which,
5230\end_layout
5231
5232\begin_layout Plain Layout
5233
5234       together with the login name, will be  used  to  identify  which
5235\end_layout
5236
5237\begin_layout Plain Layout
5238
5239       pools are yours.
5240\end_layout
5241
5242\begin_layout Plain Layout
5243
5244\end_layout
5245
5246\begin_layout Plain Layout
5247
5248SEE ALSO
5249\end_layout
5250
5251\begin_layout Plain Layout
5252
5253       sara-get-mem-size(1),   sara-get-num-cores(1),  stoposclient(1),
5254\end_layout
5255
5256\begin_layout Plain Layout
5257
5258       stoposdump(1)
5259\end_layout
5260
5261\begin_layout Plain Layout
5262
5263\end_layout
5264
5265\begin_layout Plain Layout
5266
5267AUTHORS
5268\end_layout
5269
5270\begin_layout Plain Layout
5271
5272       Willem Vermin
5273\end_layout
5274
5275\begin_layout Plain Layout
5276
5277\end_layout
5278
5279\begin_layout Plain Layout
5280
5281BUGS
5282\end_layout
5283
5284\begin_layout Plain Layout
5285
5286       No bugs known yet.
5287\end_layout
5288
5289\begin_layout Plain Layout
5290
5291\end_layout
5292
5293\begin_layout Plain Layout
5294
5295\end_layout
5296
5297\begin_layout Plain Layout
5298
5299\end_layout
5300
5301\begin_layout Plain Layout
5302
5303                                                                     STOPOS(1)
5304\end_layout
5305
5306\begin_layout Plain Layout
5307
5308\end_layout
5309
5310\end_inset
5311
5312
5313\end_layout
5314
5315\end_body
5316\end_document
Note: See TracBrowser for help on using the repository browser.