Opened 8 years ago
Closed 8 years ago
#57 closed enhancement (fixed)
Sali listenning port server
Reported by: | VincentM | Owned by: | dennis |
---|---|---|---|
Priority: | major | Milestone: | |
Component: | sali server tools | Version: | 1.5.5 |
Keywords: | Cc: |
Description
Hi,
I just wanted to know, when i start the SALI, with the netstat command I see that teh server is listening on ports 6969 and 8181. But not on the port 873.
Should the server listening on this port? Or should i have to run a rsync daemon with the config file created with the "sali rsync" command ?
Best Regards,
Vincent
Attachments (1)
Change History (37)
comment:1 in reply to: ↑ description Changed 8 years ago by dennis
comment:2 Changed 8 years ago by dennis
- Owner set to dennis
- Status changed from new to accepted
comment:3 Changed 8 years ago by dennis
- Milestone SALI 1.6.0 deleted
comment:4 follow-up: ↓ 5 Changed 8 years ago by anonymous
HI,
Thanks for your quick answer. This part is more clear for me now.
But I always have questions about how to use sali but I don't find answers in the documentation and I've never use system imager.
First, i'm not clear with the script things. I read it exists the masterscript.new. Should i have to use it and rename it with the name of my image. For example if my image name is ubuntu, i have to rename masterscript.new in ubuntu.master ?
And the second things is with Bittorrent protocol, I can't find how to create the torrent file.
It hasn't to be created automatically ?
I just want to test to install a new node with rsync and bit torrent.
Best regards,
Vincent
comment:5 in reply to: ↑ 4 ; follow-up: ↓ 6 Changed 8 years ago by bas
Replying to anonymous:
HI,
Thanks for your quick answer. This part is more clear for me now.
But I always have questions about how to use sali but I don't find answers in the documentation and I've never use system imager.
That is good. we must update the docs to point that 'systemimager' knowledge is not needed.
First, i'm not clear with the script things. I read it exists the masterscript.new. Should i have to use it and rename it with the name of my image. For example if my image name is ubuntu, i have to rename masterscript.new in ubuntu.master ?
That is correct.
And the second things is with Bittorrent protocol, I can't find how to create the torrent file.
It hasn't to be created automatically ?
in your sali.cfg you specifiy which image must also be monitored. If it updated it will generate the bittorrent file(s) for you:
the auto update feature for tarballs and metafiles [update] enabled : True ; required imagedir : %(maindir)s/images ; required images : amd64_squeeze_lisa ; required separate with , sleeptime : 300 ; optional ( default is 300 ) compress : True ; optional ( default is False ) tarcommand : /sbin/tar ; optional ( default is search ) ; There are 2 ways to specify the trackers/announcers trackeradress : http://0.0.0.0:6969/announce
In this example images : amd64_squeeze_lisa will be converted to bittorrent.
good luck
comment:6 in reply to: ↑ 5 Changed 8 years ago by anonymous
Replying to bas:
Replying to anonymous:
HI,
Thanks for your quick answer. This part is more clear for me now.
But I always have questions about how to use sali but I don't find answers in the documentation and I've never use system imager.
That is good. we must update the docs to point that 'systemimager' knowledge is not needed.
First, i'm not clear with the script things. I read it exists the masterscript.new. Should i have to use it and rename it with the name of my image. For example if my image name is ubuntu, i have to rename masterscript.new in ubuntu.master ?
That is correct.
And the second things is with Bittorrent protocol, I can't find how to create the torrent file.
It hasn't to be created automatically ?
in your sali.cfg you specifiy which image must also be monitored. If it updated it will generate the bittorrent file(s) for you:
the auto update feature for tarballs and metafiles [update] enabled : True ; required imagedir : %(maindir)s/images ; required images : amd64_squeeze_lisa ; required separate with , sleeptime : 300 ; optional ( default is 300 ) compress : True ; optional ( default is False ) tarcommand : /sbin/tar ; optional ( default is search ) ; There are 2 ways to specify the trackers/announcers trackeradress : http://0.0.0.0:6969/announceIn this example images : amd64_squeeze_lisa will be converted to bittorrent.
good luck
I was pretty close, i wanted to change this line but i wasn't sure.
Thank a lot for your reactivity
Bests
comment:7 Changed 8 years ago by anonymous
Hi,
I have a Sali system working with rsync but not with Bittorrent.
I use the same masterscript for both methods. Is that correct?
Everything is allrigth, it says "Starting download tarball..." but just right after this it says :
killed
killed
killed
killed
killed
killed
killed
killed
- Finished running install script
killed
killed
killed
sh: 90: unknown operand
killed
killed
sh: 0: unknown operand
killed
killed
killed
And then I can't access the shell to see the logs.
Do you have any idea of what could be the problem?
Best regards,
Vincent
comment:8 Changed 8 years ago by bas
Vincent that is good to hear that rsync works. Now for Bittorrent we need some more info:
- pxeconfig file, our look like this:
append initrd=sali/default/initrd.img SSHD=y MONITOR_SERVER=si_monitor MONITOR_CONSOLE=yes SCRIPTNAME=amd64_squeeze_lisa VERBOSE_LEVEL=2 PROTOCOL=bittorrent GRUB2=yes STAGING=/tmp rw
- how much memory does your system have. The tar.gz file is copied to memory. Our images are in size:
-rw-r--r-- 1 root root 2.6G Jan 26 12:12 image-amd64_squeeze_lisa.tar.gz -rw-r--r-- 1 root root 203K Jan 26 12:12 image-amd64_squeeze_lisa.tar.gz.torrent
- you can also specify staging /a/tmp, so the tar.gz will be downloaded to disk. Is not that fast.
- set VERBOSE_LEVEL=256, you will get a lot of messages.
comment:9 Changed 8 years ago by anonymous
Indeed, it was the tarball who didn't fit in the system memory. We add RAM, it's OK now.
So now the masterscript execute, and the download "start".
But the message "Downlading 0 percent at ..." is looping, apparently the download does not progress on the node.
I made a test with a computer with a torrent client (who downloaded the torrent normally) and I see that this computer is uploading the torrent when I start installation of a new node. So I think the new node have download the torrent in memory but he seems to be lock in a loop with the previous message. I don't see any messages about the extraction of the tarball.
I have a VERBOSE_LEVEL=256 and don't see error message.
Do you know this problem?
Best regards,
Vincent
comment:10 Changed 8 years ago by anonymous
I have take a look at the code and it appears that we are blocked in this function:
96 aria_monitor_log(){
97 FILE=$1
98
99 DONE=0
100 while [ $DONE -lt 1 ]
101 do
102 PROGRESS=$(aria_get_rpc_progress $FILE)
103 SPEED=$(aria_get_rpc_speed $FILE)
104 case $PROGRESS in
105 *100*)
106 p_comment 0 "Done with downloading file"
107 DONE=1
108 ;;
109
110 *)
111 p_comment 0 "Downloading: ${PROGRESS} percent at ${SPEED} bytes/sec"
112 sleep 5
113 ;;
114 esac
115 done
116 }
The value of $PROGRESS don't grow up so we can't leave the loop..
Best regards,
Vincent
comment:11 Changed 8 years ago by bas
Vincent you can turn of the monitoring stuff by leaving out:
- MONITOR_CONSOLE=yes
in the pxeconfig file. Which version do you use and if you ssh to the node you can telnet to the bittorrent port. There is no firewall in between?
comment:12 Changed 8 years ago by anonymous
MONITOR_CONSOLE is already on yes.
I use the 1.5.6 version and when I start sali as a shell, and when I try to telnet on the bittorrent port (6969) I've the following message:
" Connection closed by foreign host"
There no firewall because we accept all with iptables.
What I don't understand is that it works with a classic bittorrent client..
comment:13 follow-up: ↓ 14 Changed 8 years ago by dennis
Can you post the contents of your sali.cfg. And the output of the following command:
ps -ALf|grep sali
comment:14 in reply to: ↑ 13 Changed 8 years ago by anonymous
Replying to dennis:
Can you post the contents of your sali.cfg. And the output of the following command:
ps -ALf|grep sali
Here is the result of the following command ps -ALf|grep sali :
root@proj1:~# ps -ALf | grep sali root 1858 1 1858 0 4 Feb08 ? 00:00:00 /usr/bin/python /usr/local/bin/sali_server /etc/sali/sali.cfg start root 1858 1 1860 0 4 Feb08 ? 00:00:51 /usr/bin/python /usr/local/bin/sali_server /etc/sali/sali.cfg start root 1858 1 1861 0 4 Feb08 ? 00:19:31 /usr/bin/python /usr/local/bin/sali_server /etc/sali/sali.cfg start root 1858 1 1863 0 4 Feb08 ? 00:00:17 /usr/bin/python /usr/local/bin/sali_server /etc/sali/sali.cfg start root 13329 1 13329 0 1 Feb11 ? 00:00:00 rsync --daemon --config=/etc/sali/rsyncd.conf root 22684 22624 22684 0 1 13:42 pts/1 00:00:00 grep --color=auto sali
and here my sali.cfg :
root@proj1:~# cat /etc/sali/sali.cfg ; "SALI" ; ; Copyright (c) 2010 SARA Computing and Networking Services ; ; $Id: sali.cfg 495 2012-05-02 10:01:43Z bas $ [DEFAULT] maindir : /data/sali ; required logdir : /var/log/sali ; required logfile : %(logdir)s/sali.log ; required loglevel : 1 ; required pidfile : /var/run/sali.pid ; required serveraddress : 192.168.1.1 ; required configdir : /etc/sali [rsync] stubdir : %(configdir)s/rsync_stubs/ configfile : %(configdir)s/rsyncd.conf stubsext : conf port : 873 quiet : False [monitor] enabled : True port : 8181 filedestination : %(maindir)s/monitor.data si_monitortk : True si_clients : /var/lib/systemimager/clients.xml ; the auto update feature for tarballs and metafiles [update] enabled : True ; required imagedir : %(maindir)s/images ; required images : ubuntu ; required separate with , sleeptime : 300 ; optional ( default is 300 ) compress : True ; optional ( default is False ) tarcommand : /sbin/tar ; optional ( default is search ) trackeradress : http://%(serveraddress)s:6969/announce ; global bittorrent options [bittorrent] torrentsdir : %(maindir)s/torrents ; required when tracker or bittorrent enabled enable_bittorrent : True ; required enable_tracker : True ; required ; for arguments check the man page of bttrack [tracker] port : 6969 logfile : %(logdir)s/tracker.log dfile : %(logdir)s/dstate ; required if tracker is enabled allow_get : 1 ; for arguments check the man page of bittorrent-downloader [seeder] rerequest_interval : 1 max_upload_rate : 0 }}}
comment:15 follow-up: ↓ 16 Changed 8 years ago by dennis
The machine you are trying to install is on the same network as the imageserver (192.168.1.1)?
Perhaps you can change the option trackeradress to http://0.0.0.0:6969
This means that SALI will figure out which interfaces are configured on your machine. And for each ip address an entry will be added within the .torrent so you can have more then 1 announcer.
comment:16 in reply to: ↑ 15 Changed 8 years ago by dennis
Replying to dennis:
The machine you are trying to install is on the same network as the imageserver (192.168.1.1)?
Perhaps you can change the option trackeradress to http://0.0.0.0:6969
This means that SALI will figure out which interfaces are configured on your machine. And for each ip address an entry will be added within the .torrent so you can have more then 1 announcer.
Plus you must delete the .torrent file in /data/sali/torrents and restart the sali_server daemon.
comment:17 Changed 8 years ago by anonymous
Yes all the machines are on the same LAN. We have set a DHCP to attribute IP addresses.
I have made the change of the trackeradress but it's still not working. On my machine with utorrent I can download the tarball.
On a ubuntu machine we try with the commande :
aria2c image-ubuntu.tar.gz.torrent
But me have the following error in aria:
errorCode=25 Bencode decoding failed: A positive integer expected but none found
comment:18 Changed 8 years ago by dennis
Hmm, that is an error we haven't seen before. Can you try to run aria2c with option --bt-seed-unverified=true
And a couple of questions:
- Which version of Python do you use?
- Any errors /var/log/sali/sali.log
Can you also try to login into the SALI environment with SSH and run the aria2c command by hand. I'm curious what output is generated.
comment:19 Changed 8 years ago by anonymous
We use python's version 2.7.3
here is the log, I don't see any errors :
2013-02-12 16:42:50,734 - DaemonCtl - INFO - starting as daemon 2013-02-12 16:42:50,751 - SaliMonitor - INFO - stopped 2013-02-12 16:42:50,751 - UpdateImages - INFO - stopped 2013-02-12 16:42:50,751 - Seeder - INFO - stopped 2013-02-12 16:42:50,752 - Tracker - DEBUG - removing dstate file 2013-02-12 16:42:50,752 - Tracker - INFO - stopped 2013-02-12 16:42:51,066 - SaliServer - INFO - starting threads 2013-02-12 16:42:51,093 - UpdateImages - INFO - started 2013-02-12 16:42:51,097 - Tracker - INFO - started 2013-02-12 16:42:51,097 - Tracker - DEBUG - starting tracker with args "--dfile /var/log/sali/dstate --allow_get 1 --logfile /var/log/sali/tracker.log --port 6969 --allowed_dir /data/sali/torrents" 2013-02-12 16:42:51,099 - Seeder - INFO - started 2013-02-12 16:42:51,105 - UpdateImages - INFO - starting check 2013-02-12 16:42:51,106 - Seeder - DEBUG - start seeder with args "/data/sali/torrents --max_upload_rate 0 --rerequest_interval 1" 2013-02-12 16:42:51,107 - SaliMonitor - INFO - started 2013-02-12 16:42:51,237 - UpdateImages - INFO - ubuntu: tarball of image is up to date 2013-02-12 16:42:51,238 - CreateImage - INFO - ubuntu: does not have a torrent metafile, creating it now 2013-02-12 16:42:51,238 - CreateImage - INFO - ubuntu: creating torrent file 2013-02-12 16:43:11,066 - CreateImage - INFO - torrent file has been created 2013-02-12 16:43:11,122 - UpdateImages - INFO - sleeping for 5 minutes
Maybe I had to reinstall the server ? I used the "Other distribution" way, maybe I have to use the debian way because I am on ubuntu?
comment:20 Changed 8 years ago by dennis
- Can you attach the .torrent file to this ticket.
- Also try to login to the SALI environment with SSH and run aria2c by hand
- root password is SaliRoot
- Perhaps you also have to add SSHD=y to the pxe file
Changed 8 years ago by anonymous
comment:21 follow-up: ↓ 22 Changed 8 years ago by anonymous
When we connect to the node with SSH and we downlaod manually the .torrent file and the we run aria2 with this torrent file. It works !
96 aria_monitor_log(){ 97 FILE=$1 98 99 DONE=0 100 while [ $DONE -lt 1 ] 101 do 102 PROGRESS=$(aria_get_rpc_progress $FILE) 103 SPEED=$(aria_get_rpc_speed $FILE) 104 case $PROGRESS in 105 *100*) 106 p_comment 0 "Done with downloading file" 107 DONE=1 108 ;; 109 110 *) 111 p_comment 0 "Downloading: ${PROGRESS} percent at ${SPEED} bytes/sec" 112 sleep 5 113 ;; 114 esac 115 done 116 }
I think we are blocked in this loop because we never reach the leaving condition that is PROGRESS = 100
comment:22 in reply to: ↑ 21 Changed 8 years ago by dennis
Replying to anonymous:
When we connect to the node with SSH and we downlaod manually the .torrent file and the we run aria2 with this torrent file. It works !
96 aria_monitor_log(){ 97 FILE=$1 98 99 DONE=0 100 while [ $DONE -lt 1 ] 101 do 102 PROGRESS=$(aria_get_rpc_progress $FILE) 103 SPEED=$(aria_get_rpc_speed $FILE) 104 case $PROGRESS in 105 *100*) 106 p_comment 0 "Done with downloading file" 107 DONE=1 108 ;; 109 110 *) 111 p_comment 0 "Downloading: ${PROGRESS} percent at ${SPEED} bytes/sec" 112 sleep 5 113 ;; 114 esac 115 done 116 }I think we are blocked in this loop because we never reach the leaving condition that is PROGRESS = 100
Owke, great news it works a bit :p, we are going to investigate this further.
We shall modify the part of the script to make it more VERBOSE. Then we ask you again to test it, I'm curious what the output is of the aria_get_rcp_progress.
Thanks for testing.
comment:23 follow-up: ↓ 24 Changed 8 years ago by anonymous
Thank you, we will wait for your modifications.
We are students and we are using SALI for a university project.
So we will continue with rsync protocol.
We want to be able to download with SALI A big file in a directory or on a Partition that we call "reservoir". We have to modify some function in the initrd. Is that complicated to compile the initrd ? we have tried to do a
make build
But there so many questions to answer, and finally we have errors. Is there any documentation about that ?
comment:24 in reply to: ↑ 23 Changed 8 years ago by bas
Replying to anonymous:
Thank you, we will wait for your modifications.
We are students and we are using SALI for a university project.
So we will continue with rsync protocol.
We want to be able to download with SALI A big file in a directory or on a Partition that we call "reservoir". We have to modify some function in the initrd. Is that complicated to compile the initrd ? we have tried to do a
make buildBut there so many questions to answer, and finally we have errors. Is there any documentation about that ?
You can start here. I never heard about questions for which os/kernel/architecture do you build (i386/x86_64 is supported), seems like you get kernel config questions? You must run debian squeeze for the build process!
Do not use the latest version but the stable 1.5 branch:
comment:25 follow-up: ↓ 26 Changed 8 years ago by anonymous
We use this http://www.thegeekstuff.com/2009/07/how-to-view-modify-and-recreate-initrd-img/
to modify the functions in the file "/etc/init.d/functions" . It's a quickly way to modify the initrd.
We search the problem for bittorrent, so we have added a shellout in the function aria_monitor_log() to see the errors.
In the file aria.log.err we have the following message :
/usr/bin/xmlrpc: not found
Maybe this is the origin of our problem ?
comment:26 in reply to: ↑ 25 Changed 8 years ago by dennis
Replying to anonymous:
We use this http://www.thegeekstuff.com/2009/07/how-to-view-modify-and-recreate-initrd-img/
to modify the functions in the file "/etc/init.d/functions" . It's a quickly way to modify the initrd.
We search the problem for bittorrent, so we have added a shellout in the function aria_monitor_log() to see the errors.
In the file aria.log.err we have the following message :
/usr/bin/xmlrpc: not foundMaybe this is the origin of our problem ?
I assume that you are using version 1.5.6? I started a SALI rescue environment and issued the following commands:
gb-r41n1:~ root# cat /etc/sali_version 1.5.6 gb-r41n1:~ root# ls -ld /usr/bin/xmlrpc -rwxr-xr-x 1 root root 29000 Sep 27 09:48 /usr/bin/xmlrpc gb-r41n1:~ root#
Perhaps something is wrong during repacking of the initrd (we are using bzip2 as compression).
May I ask what your are trying to modify to the /etc/initd.d/functions?
comment:27 follow-up: ↓ 28 Changed 8 years ago by anonymous
I have run on the node SALI as a shell with an initrd freshly download from the SALI website.
Here is the result of the command you made, on my node:
sali:~ root# cat /etc/sali_version 1.5.6 sali:~ root# ls -ld /usr/bin/xmlrpc ls: /usr/bin/xmlrpc: No such file or directory sali:~ root#
I try to modify the functions to find out the problem with bittorrent.
comment:28 in reply to: ↑ 27 Changed 8 years ago by dennis
Replying to anonymous:
I have run on the node SALI as a shell with an initrd freshly download from the SALI website.
Here is the result of the command you made, on my node:
sali:~ root# cat /etc/sali_version 1.5.6 sali:~ root# ls -ld /usr/bin/xmlrpc ls: /usr/bin/xmlrpc: No such file or directory sali:~ root#I try to modify the functions to find out the problem with bittorrent.
Ah you are using the i386 version of SALI, I think something went wrong during the compilation. You can try to use the x86_64. This version can also be used to install en 32bit OS.
The i386 is meant for embedded systems. Thanks for reporting this, we are going to fix this.
comment:29 follow-up: ↓ 30 Changed 8 years ago by anonymous
I try with the initrd from the link, the x86_64 version but i have the following error message:
this kernel requires an x86-64 CPU, but only detected an i686CPU
We are trying to use the
/tmp/aria.log
file to extract speed and progress information from it.
comment:30 in reply to: ↑ 29 ; follow-up: ↓ 32 Changed 8 years ago by dennis
Replying to anonymous:
I try with the initrd from the link, the x86_64 version but i have the following error message:
this kernel requires an x86-64 CPU, but only detected an i686CPUWe are trying to use the
/tmp/aria.logfile to extract speed and progress information from it.
We shall build a new version. We give you an update.
comment:31 follow-up: ↓ 33 Changed 8 years ago by anonymous
I have an other question. I want to copy a directory in a repertoire for example:
/reservoir
I succeed to transfer the file on the
/a
but then I want to copy this in the "reservoir" directory. I can't figure out how to do this. Do you have any idea to do that ?
comment:32 in reply to: ↑ 30 ; follow-up: ↓ 34 Changed 8 years ago by bas
Replying to dennis:
Replying to anonymous:
I try with the initrd from the link, the x86_64 version but i have the following error message:
this kernel requires an x86-64 CPU, but only detected an i686CPUWe are trying to use the
/tmp/aria.logfile to extract speed and progress information from it.
We shall build a new version. We give you an update.
There is a new i386 version available on our ftp server.
comment:33 in reply to: ↑ 31 ; follow-up: ↓ 35 Changed 8 years ago by bas
Replying to anonymous:
I have an other question. I want to copy a directory in a repertoire for example:
/reservoirI succeed to transfer the file on the
/abut then I want to copy this in the "reservoir" directory. I can't figure out how to do this. Do you have any idea to do that ?
I have a question for you. What kind of system due you work on and what are you trying to acomplish? SALI is mostly used to install systems from scratch. SALI is the root system '/' and the disk partions of the local filesystem are mount under /a. That is the root directory of the system if it is rebooted.
comment:34 in reply to: ↑ 32 Changed 8 years ago by anonymous
Replying to bas:
Replying to dennis:
Replying to anonymous:
I try with the initrd from the link, the x86_64 version but i have the following error message:
this kernel requires an x86-64 CPU, but only detected an i686CPUWe are trying to use the
/tmp/aria.logfile to extract speed and progress information from it.
We shall build a new version. We give you an update.
There is a new i386 version available on our ftp server.
Thank you, now it's working with the new i386 version.
comment:35 in reply to: ↑ 33 Changed 8 years ago by anonymous
Replying to bas:
Replying to anonymous:
I have an other question. I want to copy a directory in a repertoire for example:
/reservoirI succeed to transfer the file on the
/abut then I want to copy this in the "reservoir" directory. I can't figure out how to do this. Do you have any idea to do that ?
I have a question for you. What kind of system due you work on and what are you trying to acomplish? SALI is mostly used to install systems from scratch. SALI is the root system '/' and the disk partions of the local filesystem are mount under /a. That is the root directory of the system if it is rebooted.
We are trying to use SALI to transfer Virtual Machines.
In a first time we use SALI to install a system like ubuntu, and we create a partition named "reservoir" where we gonna store the virtual machines. If there is only one disk we create a partition, otherwise if there is two disks we use the second one to store the virtual machines.
So we try to figure out how to store directly in
/reservoir
our virtual machine.
comment:36 Changed 8 years ago by bas
- Resolution set to fixed
- Status changed from accepted to closed
will close this ticket. As bittorrent/rsync works.
Replying to VincentM:
I just noticed that we completely forgotten to document this. The answer to your question is Yes. You have to run a rsync daemon with the configuration file that SALI has generated for you.
The reasoning behind this, that most distribution also install a init.d script to start the rsync as a daemon. I shall update our documentation to make this more clear.
Thanks for reporting.