Sali listenning port server

[VincentM]

Hi,

I just wanted to know, when i start the SALI, with the netstat command I see that teh server is listening on ports 6969 and 8181. But not on the port 873.
Should the server listening on this port? Or should i have to run a rsync daemon with the config file created with the "sali rsync" command ?

Best Regards,

Vincent

[dennis]

Replying to VincentM:

Hi,

I just wanted to know, when i start the SALI, with the netstat command I see that teh server is listening on ports 6969 and 8181. But not on the port 873.
Should the server listening on this port? Or should i have to run a rsync daemon with the config file created with the "sali rsync" command ?

I just noticed that we completely forgotten to document this. The answer to your question is Yes. You have to run a rsync daemon with the configuration file that SALI has generated for you.

The reasoning behind this, that most distribution also install a init.d script to start the rsync as a daemon. I shall update our documentation to make this more clear.

Thanks for reporting.

Best Regards,

Vincent

[anonymous]

HI,

Thanks for your quick answer. This part is more clear for me now.
But I always have questions about how to use sali but I don't find answers in the documentation and I've never use system imager.

First, i'm not clear with the script things. I read it exists the masterscript.new. Should i have to use it and rename it with the name of my image. For example if my image name is ubuntu, i have to rename masterscript.new in ubuntu.master ?

And the second things is with Bittorrent protocol, I can't find how to create the torrent file.
It hasn't to be created automatically ?

I just want to test to install a new node with rsync and bit torrent.

Best regards,

Vincent

[bas]

Replying to anonymous:

HI,

Thanks for your quick answer. This part is more clear for me now.
But I always have questions about how to use sali but I don't find answers in the documentation and I've never use system imager.

That is good. we must update the docs to point that 'systemimager' knowledge is not needed.

First, i'm not clear with the script things. I read it exists the masterscript.new. Should i have to use it and rename it with the name of my image. For example if my image name is ubuntu, i have to rename masterscript.new in ubuntu.master ?

That is correct.

And the second things is with Bittorrent protocol, I can't find how to create the torrent file.
It hasn't to be created automatically ?

in your sali.cfg you specifiy which image must also be monitored. If it updated it will generate the bittorrent file(s) for you:

 the auto update feature for tarballs and metafiles
[update]
enabled             : True                      ; required
imagedir            : %(maindir)s/images        ; required
images              : amd64_squeeze_lisa        ; required separate with ,
sleeptime           : 300                       ; optional ( default is 300 )
compress            : True                      ; optional ( default is False )
tarcommand          : /sbin/tar                 ; optional ( default is search )
; There are 2 ways to specify the trackers/announcers
trackeradress       : http://0.0.0.0:6969/announce

In this example images : amd64_squeeze_lisa will be converted to bittorrent.

good luck

[anonymous]

Replying to bas:

Replying to anonymous:

HI,

Thanks for your quick answer. This part is more clear for me now.
But I always have questions about how to use sali but I don't find answers in the documentation and I've never use system imager.

That is good. we must update the docs to point that 'systemimager' knowledge is not needed.

First, i'm not clear with the script things. I read it exists the masterscript.new. Should i have to use it and rename it with the name of my image. For example if my image name is ubuntu, i have to rename masterscript.new in ubuntu.master ?

That is correct.

And the second things is with Bittorrent protocol, I can't find how to create the torrent file.
It hasn't to be created automatically ?

in your sali.cfg you specifiy which image must also be monitored. If it updated it will generate the bittorrent file(s) for you:

 the auto update feature for tarballs and metafiles
[update]
enabled             : True                      ; required
imagedir            : %(maindir)s/images        ; required
images              : amd64_squeeze_lisa        ; required separate with ,
sleeptime           : 300                       ; optional ( default is 300 )
compress            : True                      ; optional ( default is False )
tarcommand          : /sbin/tar                 ; optional ( default is search )
; There are 2 ways to specify the trackers/announcers
trackeradress       : http://0.0.0.0:6969/announce

In this example images : amd64_squeeze_lisa will be converted to bittorrent.

good luck

I was pretty close, i wanted to change this line but i wasn't sure.

Thank a lot for your reactivity

Bests

[anonymous]

Hi,

I have a Sali system working with rsync but not with Bittorrent.

I use the same masterscript for both methods. Is that correct?

Everything is allrigth, it says "Starting download tarball..." but just right after this it says :

killed
killed
killed
killed
killed
killed
killed
killed

Finished running install script

killed
killed
killed
killed
sh: 90: unknown operand
killed
killed
sh: 0: unknown operand
killed
killed
killed

And then I can't access the shell to see the logs.

Do you have any idea of what could be the problem?

Best regards,

Vincent

[bas]

Vincent that is good to hear that rsync works. Now for Bittorrent we need some more info:

• pxeconfig file, our look like this:

    append initrd=sali/default/initrd.img SSHD=y MONITOR_SERVER=si_monitor MONITOR_CONSOLE=yes SCRIPTNAME=amd64_squeeze_lisa VERBOSE_LEVEL=2 PROTOCOL=bittorrent GRUB2=yes STAGING=/tmp rw
  

• how much memory does your system have. The tar.gz file is copied to memory. Our images are in size:

  -rw-r--r-- 1 root root 2.6G Jan 26 12:12 image-amd64_squeeze_lisa.tar.gz
  -rw-r--r-- 1 root root 203K Jan 26 12:12 image-amd64_squeeze_lisa.tar.gz.torrent
  

• you can also specify staging /a/tmp, so the tar.gz will be downloaded to disk. Is not that fast.

• set VERBOSE_LEVEL=256, you will get a lot of messages.

[anonymous]

Indeed, it was the tarball who didn't fit in the system memory. We add RAM, it's OK now.

So now the masterscript execute, and the download "start".
But the message "Downlading 0 percent at ..." is looping, apparently the download does not progress on the node.

I made a test with a computer with a torrent client (who downloaded the torrent normally) and I see that this computer is uploading the torrent when I start installation of a new node. So I think the new node have download the torrent in memory but he seems to be lock in a loop with the previous message. I don't see any messages about the extraction of the tarball.

I have a VERBOSE_LEVEL=256 and don't see error message.

Do you know this problem?

Best regards,

Vincent

[anonymous]

I have take a look at the code and it appears that we are blocked in this function:

96 aria_monitor_log(){
97 FILE=$1
98
99 DONE=0
100 while [ $DONE -lt 1 ]
101 do
102 PROGRESS=$(aria_get_rpc_progress $FILE)
103 SPEED=$(aria_get_rpc_speed $FILE)
104 case $PROGRESS in
105 *100*)
106 p_comment 0 "Done with downloading file"
107 DONE=1
108 ;;
109
110 *)
111 p_comment 0 "Downloading: ${PROGRESS} percent at ${SPEED} bytes/sec"
112 sleep 5
113 ;;
114 esac
115 done
116 }

The value of $PROGRESS don't grow up so we can't leave the loop..

Best regards,

Vincent

[bas]

Vincent you can turn of the monitoring stuff by leaving out:

• MONITOR_CONSOLE=yes

in the pxeconfig file. Which version do you use and if you ssh to the node you can telnet to the bittorrent port. There is no firewall in between?

[anonymous]

MONITOR_CONSOLE is already on yes.

I use the 1.5.6 version and when I start sali as a shell, and when I try to telnet on the bittorrent port (6969) I've the following message:
" Connection closed by foreign host"

There no firewall because we accept all with iptables.

What I don't understand is that it works with a classic bittorrent client..

[dennis]

Can you post the contents of your sali.cfg. And the output of the following command:

ps -ALf|grep sali

[anonymous]

Replying to dennis:

Can you post the contents of your sali.cfg. And the output of the following command:

ps -ALf|grep sali

Here is the result of the following command ps -ALf|grep sali :

root@proj1:~# ps -ALf | grep sali
root      1858     1  1858  0    4 Feb08 ?        00:00:00 /usr/bin/python /usr/local/bin/sali_server /etc/sali/sali.cfg start
root      1858     1  1860  0    4 Feb08 ?        00:00:51 /usr/bin/python /usr/local/bin/sali_server /etc/sali/sali.cfg start
root      1858     1  1861  0    4 Feb08 ?        00:19:31 /usr/bin/python /usr/local/bin/sali_server /etc/sali/sali.cfg start
root      1858     1  1863  0    4 Feb08 ?        00:00:17 /usr/bin/python /usr/local/bin/sali_server /etc/sali/sali.cfg start
root     13329     1 13329  0    1 Feb11 ?        00:00:00 rsync --daemon --config=/etc/sali/rsyncd.conf
root     22684 22624 22684  0    1 13:42 pts/1    00:00:00 grep --color=auto sali 

and here my sali.cfg :

root@proj1:~# cat /etc/sali/sali.cfg 
; "SALI"
;
; Copyright (c) 2010 SARA Computing and Networking Services
;
;   $Id: sali.cfg 495 2012-05-02 10:01:43Z bas $

[DEFAULT]
maindir             : /data/sali                ; required
logdir              : /var/log/sali             ; required
logfile             : %(logdir)s/sali.log      ; required
loglevel            : 1                         ; required
pidfile             : /var/run/sali.pid         ; required
serveraddress       : 192.168.1.1            ; required
configdir           : /etc/sali

[rsync]
stubdir             : %(configdir)s/rsync_stubs/
configfile          : %(configdir)s/rsyncd.conf
stubsext            : conf
port                : 873
quiet               : False

[monitor]
enabled             : True
port                : 8181
filedestination     : %(maindir)s/monitor.data
si_monitortk        : True
si_clients          : /var/lib/systemimager/clients.xml

; the auto update feature for tarballs and metafiles
[update]
enabled             : True                     ; required
imagedir            : %(maindir)s/images        ; required
images              : ubuntu          ; required separate with ,
sleeptime           : 300                       ; optional ( default is 300 )
compress            : True                      ; optional ( default is False )
tarcommand          : /sbin/tar                 ; optional ( default is search )
trackeradress       : http://%(serveraddress)s:6969/announce

; global bittorrent options
[bittorrent]
torrentsdir         : %(maindir)s/torrents    ; required when tracker or bittorrent enabled
enable_bittorrent   : True                   ; required
enable_tracker      : True                      ; required

; for arguments check the man page of bttrack
[tracker]
port                : 6969
logfile             : %(logdir)s/tracker.log
dfile               : %(logdir)s/dstate               ; required if tracker is enabled
allow_get           : 1

; for arguments check the man page of bittorrent-downloader
[seeder]
rerequest_interval  : 1
max_upload_rate     : 0 }}}

[dennis]

The machine you are trying to install is on the same network as the imageserver (192.168.1.1)?

Perhaps you can change the option trackeradress to ​http://0.0.0.0:6969

This means that SALI will figure out which interfaces are configured on your machine. And for each ip address an entry will be added within the .torrent so you can have more then 1 announcer.

[dennis]

Replying to dennis:

The machine you are trying to install is on the same network as the imageserver (192.168.1.1)?

Perhaps you can change the option trackeradress to ​http://0.0.0.0:6969

This means that SALI will figure out which interfaces are configured on your machine. And for each ip address an entry will be added within the .torrent so you can have more then 1 announcer.

Plus you must delete the .torrent file in /data/sali/torrents and restart the sali_server daemon.

[anonymous]

Yes all the machines are on the same LAN. We have set a DHCP to attribute IP addresses.

I have made the change of the trackeradress but it's still not working. On my machine with utorrent I can download the tarball.

On a ubuntu machine we try with the commande :

aria2c image-ubuntu.tar.gz.torrent

But me have the following error in aria:

errorCode=25 Bencode decoding failed: A positive integer expected but none found

[dennis]

Hmm, that is an error we haven't seen before. Can you try to run aria2c with option --bt-seed-unverified=true

And a couple of questions:

• Which version of Python do you use?
• Any errors /var/log/sali/sali.log

Can you also try to login into the SALI environment with SSH and run the aria2c command by hand. I'm curious what output is generated.

[anonymous]

We use python's version 2.7.3

here is the log, I don't see any errors :

2013-02-12 16:42:50,734 - DaemonCtl    - INFO    - starting as daemon
2013-02-12 16:42:50,751 - SaliMonitor  - INFO    - stopped
2013-02-12 16:42:50,751 - UpdateImages - INFO    - stopped
2013-02-12 16:42:50,751 - Seeder       - INFO    - stopped
2013-02-12 16:42:50,752 - Tracker      - DEBUG   - removing dstate file
2013-02-12 16:42:50,752 - Tracker      - INFO    - stopped
2013-02-12 16:42:51,066 - SaliServer   - INFO    - starting threads
2013-02-12 16:42:51,093 - UpdateImages - INFO    - started
2013-02-12 16:42:51,097 - Tracker      - INFO    - started
2013-02-12 16:42:51,097 - Tracker      - DEBUG   - starting tracker with args "--dfile /var/log/sali/dstate --allow_get 1 --logfile /var/log/sali/tracker.log --port 6969 --allowed_dir /data/sali/torrents"
2013-02-12 16:42:51,099 - Seeder       - INFO    - started
2013-02-12 16:42:51,105 - UpdateImages - INFO    - starting check
2013-02-12 16:42:51,106 - Seeder       - DEBUG   - start seeder with args "/data/sali/torrents --max_upload_rate 0 --rerequest_interval 1"
2013-02-12 16:42:51,107 - SaliMonitor  - INFO    - started
2013-02-12 16:42:51,237 - UpdateImages - INFO    -   ubuntu: tarball of image is up to date
2013-02-12 16:42:51,238 - CreateImage  - INFO    -     ubuntu: does not have a torrent metafile, creating it now
2013-02-12 16:42:51,238 - CreateImage  - INFO    -     ubuntu: creating torrent file
2013-02-12 16:43:11,066 - CreateImage  - INFO    -        torrent file has been created
2013-02-12 16:43:11,122 - UpdateImages - INFO    - sleeping for 5 minutes

Maybe I had to reinstall the server ? I used the "Other distribution" way, maybe I have to use the debian way because I am on ubuntu?

[dennis]

• Can you attach the .torrent file to this ticket.
• Also try to login to the SALI environment with SSH and run aria2c by hand
  • root password is SaliRoot
  • Perhaps you also have to add SSHD=y to the pxe file

[anonymous]

When we connect to the node with SSH and we downlaod manually the .torrent file and the we run aria2 with this torrent file. It works !

96 aria_monitor_log(){
97 FILE=$1
98 
99 DONE=0
100 while [ $DONE -lt 1 ]
101 do 
102 PROGRESS=$(aria_get_rpc_progress $FILE)
103 SPEED=$(aria_get_rpc_speed $FILE)
104 case $PROGRESS in
105 *100*)
106 p_comment 0 "Done with downloading file"
107 DONE=1
108 ;; 
109 
110 *)
111 p_comment 0 "Downloading: ${PROGRESS} percent at ${SPEED} bytes/sec"
112 sleep 5
113 ;;
114 esac 
115 done
116 }

I think we are blocked in this loop because we never reach the leaving condition that is PROGRESS = 100

[dennis]

Replying to anonymous:

When we connect to the node with SSH and we downlaod manually the .torrent file and the we run aria2 with this torrent file. It works !

96 aria_monitor_log(){
97 FILE=$1
98 
99 DONE=0
100 while [ $DONE -lt 1 ]
101 do 
102 PROGRESS=$(aria_get_rpc_progress $FILE)
103 SPEED=$(aria_get_rpc_speed $FILE)
104 case $PROGRESS in
105 *100*)
106 p_comment 0 "Done with downloading file"
107 DONE=1
108 ;; 
109 
110 *)
111 p_comment 0 "Downloading: ${PROGRESS} percent at ${SPEED} bytes/sec"
112 sleep 5
113 ;;
114 esac 
115 done
116 }

I think we are blocked in this loop because we never reach the leaving condition that is PROGRESS = 100

Owke, great news it works a bit :p, we are going to investigate this further.

We shall modify the part of the script to make it more VERBOSE. Then we ask you again to test it, I'm curious what the output is of the aria_get_rcp_progress.

Thanks for testing.

[anonymous]

Thank you, we will wait for your modifications.

We are students and we are using SALI for a university project.
So we will continue with rsync protocol.

We want to be able to download with SALI A big file in a directory or on a Partition that we call "reservoir". We have to modify some function in the initrd. Is that complicated to compile the initrd ? we have tried to do a

make build

But there so many questions to answer, and finally we have errors. Is there any documentation about that ?

[bas]

Replying to anonymous:

Thank you, we will wait for your modifications.

We are students and we are using SALI for a university project.
So we will continue with rsync protocol.

We want to be able to download with SALI A big file in a directory or on a Partition that we call "reservoir". We have to modify some function in the initrd. Is that complicated to compile the initrd ? we have tried to do a

make build

But there so many questions to answer, and finally we have errors. Is there any documentation about that ?

You can start here. I never heard about questions for which os/kernel/architecture do you build (i386/x86_64 is supported), seems like you get kernel config questions? You must run debian squeeze for the build process!

• https://oss.trac.surfsara.nl/sali/wiki/SaliRequirements

Do not use the latest version but the stable 1.5 branch:

• svn co https://oss.trac.surfsara.nl/sali/svn/branches/1.5

[anonymous]

We use this ​http://www.thegeekstuff.com/2009/07/how-to-view-modify-and-recreate-initrd-img/
to modify the functions in the file "/etc/init.d/functions" . It's a quickly way to modify the initrd.

We search the problem for bittorrent, so we have added a shellout in the function aria_monitor_log() to see the errors.
In the file aria.log.err we have the following message :

/usr/bin/xmlrpc: not found

Maybe this is the origin of our problem ?

[dennis]

Replying to anonymous:

We use this ​http://www.thegeekstuff.com/2009/07/how-to-view-modify-and-recreate-initrd-img/
to modify the functions in the file "/etc/init.d/functions" . It's a quickly way to modify the initrd.

We search the problem for bittorrent, so we have added a shellout in the function aria_monitor_log() to see the errors.
In the file aria.log.err we have the following message :

/usr/bin/xmlrpc: not found

Maybe this is the origin of our problem ?

I assume that you are using version 1.5.6? I started a SALI rescue environment and issued the following commands:

gb-r41n1:~ root# cat /etc/sali_version 
1.5.6
gb-r41n1:~ root# ls -ld /usr/bin/xmlrpc
-rwxr-xr-x    1 root     root         29000 Sep 27 09:48 /usr/bin/xmlrpc
gb-r41n1:~ root# 

Perhaps something is wrong during repacking of the initrd (we are using bzip2 as compression).

May I ask what your are trying to modify to the /etc/initd.d/functions?

[anonymous]

I have run on the node SALI as a shell with an initrd freshly download from the SALI website.

Here is the result of the command you made, on my node:

sali:~ root# cat /etc/sali_version 
1.5.6
sali:~ root# ls -ld /usr/bin/xmlrpc
ls: /usr/bin/xmlrpc: No such file or directory
sali:~ root# 

I try to modify the functions to find out the problem with bittorrent.

[dennis]

Replying to anonymous:

I have run on the node SALI as a shell with an initrd freshly download from the SALI website.

Here is the result of the command you made, on my node:

sali:~ root# cat /etc/sali_version 
1.5.6
sali:~ root# ls -ld /usr/bin/xmlrpc
ls: /usr/bin/xmlrpc: No such file or directory
sali:~ root# 

I try to modify the functions to find out the problem with bittorrent.

Ah you are using the i386 version of SALI, I think something went wrong during the compilation. You can try to use the x86_64. This version can also be used to install en 32bit OS.

• ​ftp://ftp.surfsara.nl/sali/sali.tar.gz

The i386 is meant for embedded systems. Thanks for reporting this, we are going to fix this.

[anonymous]

I try with the initrd from the link, the x86_64 version but i have the following error message:

this kernel requires an x86-64 CPU, but only detected an i686CPU

We are trying to use the

/tmp/aria.log

file to extract speed and progress information from it.

[dennis]

Replying to anonymous:

I try with the initrd from the link, the x86_64 version but i have the following error message:

this kernel requires an x86-64 CPU, but only detected an i686CPU

We are trying to use the

/tmp/aria.log

file to extract speed and progress information from it.

We shall build a new version. We give you an update.

[anonymous]

I have an other question. I want to copy a directory in a repertoire for example:

/reservoir

I succeed to transfer the file on the

/a

but then I want to copy this in the "reservoir" directory. I can't figure out how to do this. Do you have any idea to do that ?

[bas]

Replying to dennis:

Replying to anonymous:

I try with the initrd from the link, the x86_64 version but i have the following error message:

this kernel requires an x86-64 CPU, but only detected an i686CPU

We are trying to use the

/tmp/aria.log

file to extract speed and progress information from it.

We shall build a new version. We give you an update.

There is a new i386 version available on our ftp server.

[bas]

Replying to anonymous:

I have an other question. I want to copy a directory in a repertoire for example:

/reservoir

I succeed to transfer the file on the

/a

but then I want to copy this in the "reservoir" directory. I can't figure out how to do this. Do you have any idea to do that ?

I have a question for you. What kind of system due you work on and what are you trying to acomplish? SALI is mostly used to install systems from scratch. SALI is the root system '/' and the disk partions of the local filesystem are mount under /a. That is the root directory of the system if it is rebooted.

[anonymous]

Replying to bas:

Replying to dennis:

Replying to anonymous:

I try with the initrd from the link, the x86_64 version but i have the following error message:

this kernel requires an x86-64 CPU, but only detected an i686CPU

We are trying to use the

/tmp/aria.log

file to extract speed and progress information from it.

We shall build a new version. We give you an update.

There is a new i386 version available on our ftp server.

Thank you, now it's working with the new i386 version.

[anonymous]

Replying to bas:

Replying to anonymous:

I have an other question. I want to copy a directory in a repertoire for example:

/reservoir

I succeed to transfer the file on the

/a

but then I want to copy this in the "reservoir" directory. I can't figure out how to do this. Do you have any idea to do that ?

I have a question for you. What kind of system due you work on and what are you trying to acomplish? SALI is mostly used to install systems from scratch. SALI is the root system '/' and the disk partions of the local filesystem are mount under /a. That is the root directory of the system if it is rebooted.

We are trying to use SALI to transfer Virtual Machines.
In a first time we use SALI to install a system like ubuntu, and we create a partition named "reservoir" where we gonna store the virtual machines. If there is only one disk we create a partition, otherwise if there is two disks we use the second one to store the virtual machines.

So we try to figure out how to store directly in

/reservoir

our virtual machine.

[bas]

will close this ticket. As bittorrent/rsync works.