User Tools
operations:documentation.ivs.bonnetransfer
This wiki is not maintained! Do not use this when setting up AuScope experiments!
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision | |||
|
operations:documentation.ivs.bonnetransfer [2012/11/13 23:35] Jamie McCallum |
operations:documentation.ivs.bonnetransfer [2013/07/12 06:30] (current) Jamie McCallum |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| ====== Transferring Data to Bonn ====== | ====== Transferring Data to Bonn ====== | ||
| + | |||
| + | **Update: Transfers directly from mounted diskpacks have significantly higher rates of failure. Staging data to RAID is strongly recommended!** | ||
| + | |||
| + | **Update: The mark5B upgrade to SDK9.2 has adversely affected fuseMk5 performance. It is not uncommon for the program to crash, sometimes with kernel panic during copy operations. A workaround script using DIMino has been kludged together and is in ~oper/recovery/ but is still in development.** | ||
| + | |||
| * Use a VNC session to run the transfers - the hobart.phys.utas.edu.au:1 session is normally a good option. | * Use a VNC session to run the transfers - the hobart.phys.utas.edu.au:1 session is normally a good option. | ||
| * Open four terminal windows on the Mark5 unit the data will be sent from | * Open four terminal windows on the Mark5 unit the data will be sent from | ||
| - | * In the first terminal, make sure that neither fuseMk5 or dimino is running <code>ps -ef | grep dim</code> and <code>ps -ef | grep fuse</code> and that another transfer isn't running <code>ps -ef | grep tsuna</code>. | + | * In the first terminal, make sure that neither fuseMk5 or dimino is running <code>ps -ef | grep DIM</code> and <code>ps -ef | grep fuse</code> and that another transfer isn't running <code>ps -ef | grep tsuna</code>. |
| * If fuseMk5A is running and the wrong module is mounted, dismount with <code>fusermount -u <directory_name></code> | * If fuseMk5A is running and the wrong module is mounted, dismount with <code>fusermount -u <directory_name></code> | ||
| - | * Then use fuseMk5 (or fuseMk5A for unupgraded mark5 units) to mount the module: <code>fuseMk5 --bank=0 --verbose --cachesize=128000000 /mnt/diskpack</code> This will mount Bank A (set bank=1 for Bank B) at /mnt/diskpack. Wait for a message to say <code>.... Registering fuseMk5A to FUSE with: ....</code>. It can take a while. Background this process with Ctrl-z and <code> bg </code>. | + | * Then use fuseMk5 to mount the module: <code>fuseMk5 --bank=0 --verbose --cachesize=128000000 /mnt/diskpack</code> This will mount Bank A (set bank=1 for Bank B) at /mnt/diskpack. Wait for a message to say <code>.... Registering fuseMk5A to FUSE with: ....</code>. It can take a while. Background this process with Ctrl-z and <code> bg </code>. |
| * In a second terminal, change to the mounted directory and make sure the data are there <code>cd /mnt/diskpack | * In a second terminal, change to the mounted directory and make sure the data are there <code>cd /mnt/diskpack | ||
| ls | ls | ||
| Line 13: | Line 18: | ||
| </code> The [[http://www3.mpifr-bonn.mpg.de/cgi-bin/showtransfers.cgi | </code> The [[http://www3.mpifr-bonn.mpg.de/cgi-bin/showtransfers.cgi | ||
| |Bonn web page]] shows available servers and space. | |Bonn web page]] shows available servers and space. | ||
| - | (194.94.199.166 = sneezy2) | + | (194.94.199.163 = io10, 194.94.199.164 = io3, 194.94.199.166 = sneezy2) |
| * Start tsunami and set up for a transfer | * Start tsunami and set up for a transfer | ||
| Line 24: | Line 29: | ||
| * in the fourth terminal window, create an empty file with a name describing the start time of the transfer, rate etc: <code>touch 20110525033000_r1485_Hb_Bonn_300m_52100_sneezy2_start</code> This puts an entry on the [[http://www3.mpifr-bonn.mpg.de/cgi-bin/showtransfers.cgi | * in the fourth terminal window, create an empty file with a name describing the start time of the transfer, rate etc: <code>touch 20110525033000_r1485_Hb_Bonn_300m_52100_sneezy2_start</code> This puts an entry on the [[http://www3.mpifr-bonn.mpg.de/cgi-bin/showtransfers.cgi | ||
| |Bonn web page]] to say there's a transfer under way | |Bonn web page]] to say there's a transfer under way | ||
| - | * and send it to the web server <code>ncftpput ftp.mpifr-bonn.mpg.de /incoming/geodesy/transfers 0110525033000_r1485hb_Hb177_Bonn_300m_52100_sneezy2_start</code> | + | * and send it to the web server <code>ncftpput ftp.mpifr-bonn.mpg.de /incoming/geodesy/transfers 20110525033000_r1485hb_Hb177_Bonn_300m_52100_sneezy2_start</code> |
| * When the transfer finishes, send a stop message with <code>touch 20110525033000_Hb_stop | * When the transfer finishes, send a stop message with <code>touch 20110525033000_Hb_stop | ||
| ncftpput ftp.mpifr-bonn.mpg.de /incoming/geodesy/transfers 20110525033000_Hb177_stop</code> | ncftpput ftp.mpifr-bonn.mpg.de /incoming/geodesy/transfers 20110525033000_Hb177_stop</code> | ||
| - | It is also possible (and preferable) to export the data onto a RAID first. Currently, there are three RAIDS which can beused for temporary storage at Hobart - Vortex, Cornucopia and 131.217.63.140 (Currently unnamed and contains two separate RAIDS of 6 and 12 TB). These can be NFS mounted on the mk5 machine with <code>mount vortex.phys.utas.edu.au:/exports/vortex_internal/ /mnt/vortex</code> (as root). Once mounted the data can be copied with <code>cp /mnt/diskpack/r1485* /mnt/vortex/r1485hb/</code>. | + | It is also possible (and preferable) to export the data onto a RAID first. Currently, there are five RAIDS which can be used for temporary storage at Hobart - Vortex, Cornucopia, Jupiter (which contains two separate RAIDS of 6 and 12 TB) and Magilla. These can be NFS mounted on the mk5 machine with <code>mount vortex.phys.utas.edu.au:/exports/vortex_internal/ /mnt/vortex</code> (as root). Once mounted the data can be copied with <code>cp /mnt/diskpack/r1485* /mnt/vortex/r1485hb/</code>. |
| - | Once fully copied, please check the data has not been corrupted in the transfer. ssh into hex0 and NFS mount the RAID if necessary. Then run <code>directory2filelist /mnt/vortex/r1485hb Mark5B-256-16-1 > /tmp/ok.txt</code> . The output to the screen will be the list of those scans which show problems. It's not uncommon to see up to ~10% of the scans fail the test - these files should be deleted and retransferred to the RAID. | + | Once fully copied, please check the data has not been corrupted in the transfer. ssh into the NFS host as observer and run <code>./directory2list /exports/vortex_internal/r1485hb Mark5B-256-16-1</code> . The output to the screen will be the list of all the scans with the start mjd/time and a summary indicating if problems were found. If bad scans are found, a summary at the end will warn you, and list of bad scans will be made available. |
| - | The same <code>directory2filelist</code> procedure should be run after a transfer to Bonn completes to check for errors in transmission. | + | After a transfer to Bonn completes, you should run <code>directory2filelist /data3/r1/hobart12/r1485/ Mark5B-256-16-1</code> to check for errors in transmission. |
/home/www/auscope/opswiki/data/attic/operations/documentation.ivs.bonnetransfer.1352849748.txt.gz · Last modified: 2012/11/13 23:35 by Jamie McCallum
Page Tools
