AuScope VLBI Operations Wiki

User Tools

Site Tools

You are here: start » operations » documentation_ops2_austral15day
Trace:
operations:documentation_ops2_austral15day

This wiki is not maintained! Do not use this when setting up AuScope experiments!

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
Next revision Both sides next revision
operations:documentation_ops2_austral15day [2013/11/21 04:35]
Jim Lovell 		operations:documentation_ops2_austral15day [2013/12/05 03:33]
Jim Lovell [Module Swaps]
Line 3: Line 3:
   * [[#​schedule_overview_and_module_usage|Observing schedule and module usage overview]]   * [[#​schedule_overview_and_module_usage|Observing schedule and module usage overview]]
   * [[#​schedule_changeover|Schedule changeover procedure]]   * [[#​schedule_changeover|Schedule changeover procedure]]
 +  * [[#​module_swaps|Swapping Mark5 modules]]
   * [[#​observatory_checking|Observatory checking procedure]] (every 3 days)   * [[#​observatory_checking|Observatory checking procedure]] (every 3 days)
   * [[#​fringe_checks|Fringe Checks]]   * [[#​fringe_checks|Fringe Checks]]
Line 10: Line 11:
 Please read the [[operations:​documentation_austral15day_overview|Overview Page]] for information on the observations,​ start and stop times etc. The google calendar on the [[operations:​schedule|schedules page]] describes experiment and shift times. Please read the [[operations:​documentation_austral15day_overview|Overview Page]] for information on the observations,​ start and stop times etc. The google calendar on the [[operations:​schedule|schedules page]] describes experiment and shift times.
  
-The AUSTRAL schedules are named ''​A13nn''​ where ''​(nn = 01, 02, 03, ... , 15)''​. Interleaved with the AUSTRAL sessions ​are R1614, CRF79 and R4615. There is only a 5 minute gap between AUSTRAL observations,​ just enough time to change schedule files and modules (if needed). Every three days, an hour is set aside for detailed checks at each site. They are staggered so that only one antenna is out of the array at a time. Check times for AuScope sites are as follows:+The AUSTRAL schedules are named ''​A13nn''​ where ''​(nn = 01, 02, 03, ... , 15)''​. Interleaved with these are regular IVS sessions ​R1614, CRF79 and R4615. There is only a 5 minute gap between AUSTRAL observations,​ just enough time to change schedule files and modules (if needed). Every three days, an hour is set aside for detailed checks at each site. They are staggered so that only one antenna is out of the array at a time. Check times for AuScope sites are as follows:
  
 ^  Site  ^  Check time (UT)  ^  Check time (Hobart) ​ ^ ^  Site  ^  Check time (UT)  ^  Check time (Hobart) ​ ^
Line 17: Line 18:
 | Yg    | 04.00 - 05.00 | 15.00 - 16.00 | | Yg    | 04.00 - 05.00 | 15.00 - 16.00 |
  
-We are using 16 TB modules at each station. They will hold two days of data each, so module changes will occur at every second schedule change. The schedule files are written such that the same source is observed at the same sidereal time every day, so the actual schedule changes are NOT at the same UT every time. Note also that R1614, CRF79 and R4615 will go on different modules to the AUSTRALs. Please refer to the [[operations:​start#​auscope_ivs_module_allocation|Module Usage Spreadsheet]] for a list of which modules to use for which experiments. +We are using 16 TB modules at each AuScope ​station. They will hold two days of data each, so module changes will occur at every second schedule change. The schedule files are written such that the same source is observed at the same sidereal time every day, so the actual schedule changes are NOT at the same UT every time. Note also that R1614, CRF79 and R4615 will go on different modules to the AUSTRALs. Please refer to the [[operations:​start#​auscope_ivs_module_allocation|Module Usage Spreadsheet]] for a list of which modules to use for which experiments.
- +
-The sections below describe how operations differ from normal. +
  
 +The sections below describe how operations differ from normal. Note that it will be possible to monitor the Hart 15m antenna and possibly Warkworth as well.
 ===== Schedule files ===== ===== Schedule files =====
  
Line 28: Line 28:
  
 Please carry out the regular two-hourly checks as normal during the observations. Please carry out the regular two-hourly checks as normal during the observations.
-  * [[operations:​documentation.eRemoteControl_checklist_midob|Description of checklist parameters]]+  * [[operations:​documentation.eRemoteControl_checklist_midob|Description of checklist parameters]] ​for AuScope telescopes
  
 +It is also possible to monitor the 15m telescope at Hartebeeshoek and you may be asked to respond to problems when local staff are unavailable. ​
 +  * [[operations:​documentation.monitor_hart15|Monitoring the Hart 15m telescope]]
  
 +It may also be possible to monitor the Warkworth 12m antenna. Details to follow if this becomes possible.
 ===== Schedule Changeover ===== ===== Schedule Changeover =====
  
Line 45: Line 48:
 mk5=vsn?</​code>​ If it doesn'​t,​ then the VSN will need setting: mk5=vsn?</​code>​ If it doesn'​t,​ then the VSN will need setting:
     - [[operations:​van_setting|Setting a module VSN code]]     - [[operations:​van_setting|Setting a module VSN code]]
-  - Now start the new schedule. Include the '',#​1'' ​section ​so that you start the schedule on the first line, e.g. <​code>​schedule=a1303hb,#​1</​code>​+  - Now start the new schedule. Include the '',#​1'' ​suffix ​so that you start the schedule on the first line, e.g. <​code>​schedule=a1303hb,#​1</​code>​
  
 Once the new schedule has started: Once the new schedule has started:
  
   * [[operations:​startmessage|Send a start message to IVS]]. ​   * [[operations:​startmessage|Send a start message to IVS]]. ​
-  * Send off the log files to IVS. Refer to [[operations:​documentation.ivs.Logs|Processing logs and shipping data]] for details. Note that all modules for AUSTRAL experiments from Yarragadee and Katherine are shipped to Hobart, and all AUSTRAL data are eTransfer-ed to Curtin for correlation.+  * Send off the log files to IVS. Refer to the page [[operations:​documentation.ivs.commentsfile|Prepare the comments file and transfer the logs]] for details. Note that all modules for AUSTRAL experiments from Yarragadee and Katherine are shipped to Hobart, and all AUSTRAL data are eTransfer-ed to Curtin for correlation.
  
 +===== Module Swaps =====
  
 +Module swaps should be coordinated with local staff at the observatories. Someone at the site will phone when they want to change a module. Estimated module change times are available here:
 +  * [[operations:​documentation_austral15day_module_swaps|Module swapping timetable]]
  
 +It's important that the new module is powered down and up during a scan gap as this draws significant power and the Mark5 can't record to the other module at the same time (and will probably crash). So, when someone from the observatory calls, consult the observation summary file and find an upcoming period of time of at least one minute between a ''​Stop Data''​ and a ''​Start Data''​ to carry out the swap. For example, in the schedule below, the period between Scans ''​332-2104''​ and ''​332-2110''​ would be a good choice as there'​s a gap of almost 4 minutes (from ''​21:​06:​18''​ to ''​21:​10:​01''​). However, a module change between Scans ''​332-2118b''​ and ''​332-2122''​ would not be a good idea as the gap is only 25 seconds (from ''​21:​22:​29''​ to ''​21:​22:​54''​):​
  
-===== Observatory Checking =====+<​code>​ 
 +                                         ​Start ​    ​Stop ​                     Record 
 + ​Scan ​     Line#  Source ​  Az El Cable    Data      Data      Dur    Gbyte 
 +date = 2013NOV28 ​ DOY = 332 
 + ​332-2101 ​    7 0648-165 254  6 CCW    21:​01:​11 ​ 21:​02:​05 ​   0:54      .0 XXX    
 + ​332-2102b ​  24 1606+106 ​ 50 27 NEUTR  21:​02:​57 ​ 21:​04:​13 ​   1:16     ​6.9 ​ *     
 + ​332-2104 ​   41 1622-253 ​ 89 48 NEUTR  21:​04:​52 ​ 21:​06:​18 ​   1:26    16.6  *     
 + ​332-2110 ​   58 0454-810 188 31 CW     ​21:​10:​01 ​ 21:​11:​30 ​   1:29    27.6  *     
 + ​332-2111 ​   75 0230-790 184 26 CW     ​21:​11:​54 ​ 21:​18:​16 ​   6:22    39.0  *     
 + ​332-2118b ​  92 2052-474 140 13 CW     ​21:​18:​58 ​ 21:​22:​29 ​   3:31    87.9  *     
 + ​332-2122 ​  109 2300-683 166 18 CW     ​21:​22:​54 ​ 21:​24:​35 ​   1:41   ​114.9 ​ *     
 + ​332-2125 ​  126 0727-115 260  6 CW     ​21:​25:​51 ​ 21:​26:​21 ​   0:30   ​127.9 ​ *     
 +</​code>​
  
-Manual stop and start. An hour allocated but re-start ASAP. +=== Procedure ===
-Use a different log file for the system checks +
-Log MONICA output+
  
 +  * Wait for the scan to finish and observe that the four disk activity lights stop flashing
 +  * halt the schedule. This makes sure that no recording can possibly happen during the swap and crash the Mark5: <​code>​halt</​code>​
 +  * Ask the local person to turn the key on the used module and swap it with a blank one.
 +  * Wait for three green lights above the new module to indicate it is powered up and ready.
 +  * Resume the schedule with the cont command <​code>​cont</​code>​
 +  * Make a note in the log that the modules have been swapped. Include VSN codes in the message.
 +  * Send an email to transfers@postoffice.maths.utas.edu.au so that we know what's happened
 +===== Observatory Checking =====
  
-Sometimes, another IVS observation will follow from the one just finished. In this case, some of the setup procedure ​will need repeating. Here's a run-down of what needs doing and what can be skipped:+Note that the observatory checking ​procedure ​requires you to interrupt the schedule ​and miss some scans. This allows us to skip checks if necessary but also means if the checks take less than the allocated hour we can be observing again as soon as possible.
  
 +Firstly, check the schedule summary printout and find out which is the first scan to finish after the start of the check period. The idea is to stop the schedule while no data are being recorded. For example, if the check time is 19:00 UT, then in the schedule below, it could be halted after 19:05:45 UT but before 19:07:43 UT:
  
-=== Prepare the schedule file ===+<​code>​ 
 +                                         ​Start ​    ​Start ​    ​Stop ​                     Record 
 + ​Scan ​     Line#  Source ​  Az El Cable  Record ​     Data      Data      Dur    Gbyte 
 +date = 2013NOV21 ​ DOY = 325 
 +. 
 +. 
 +. 
 + ​325-1854 ​  140 1610-771 170  6 CCW    18:​53:​58 ​ 18:​54:​08 ​ 18:​58:​33 ​   4:25    35.0  *     
 + ​325-1859 ​  159 1255-316 119 19 CCW    18:​59:​02 ​ 18:​59:​12 ​ 19:​05:​45 ​   6:33    43.8  *     
 + ​325-1907a ​ 178 0131-522 218  7 CCW    19:​07:​43 ​ 19:​07:​53 ​ 19:​20:​57 ​  ​13:​04 ​   56.7  *     
 +. 
 +. 
 +. 
 +</​code>​
  
-**This should already be done and can be skipped.** However, you can check for schedule and procedure files on the pcfs machines. For example, for Hobart there should be the following files on pcfshb for r4556: +Now, wait for the scan gap, and then type: 
-<​code>​/​usr2/​sched/​r4556hb.snp +<​code>​halt</​code>​ 
-/​usr2/​proc/​r4556hb.prc</​code>​ +Now make a note in the log that the schedule ​is being stopped ​for system checks. ​e.g. <code>"​Halting ​schedule for system checks</​code>​
-> If they aren't there, follow ​the following steps to prepare them: +
->  * [[operations:​documentation.ivs.drudg_9.10.5|Preparing ​the schedule ​and procedure files]] +
->  * On occasions, the Hobart data are written to disk between scans for e-transfer to Tsukuba for ultra-rapid EOP observations. Some additional changes to the schedule files are required in this case: +
->    * [[operations:​documentation.ivs.drudg_ureop_9.10.5|Additional ​schedule ​preparation ​for Hb ultra-rapid EOP experiments]]+
  
-=== Timing PC Software === +We want to keep all the check data in a separate ​log, so open a log file called "check.log": 
- +<​code>​log=check</​code>​
-This software should still be running from the previous experiment. **Skip.** +
- +
-> For reference, here's a description of [[operations:​documentation.ivs.timingsw|software on the timing PC]] +
- +
-=== Field System Software === +
- +
-As you shouldn'​t have terminated the Field System, this should still be running. **Skip.** Again, for reference:​ +
-  * [[operations:​starting_fs|Starting the PC Field System Software]] +
-  * [[operations:​documentation:​fs_antenna_9.10.5|Some notes on antenna control and monitoring]] using the new version of the Field system +
- +
-=== System Monitor === +
- +
-Should still be running. **Skip.**  +
- +
->If you do need to re-start: [[operations:​starting_sysmon|Start the System Monitor]] +
- +
-=== e-Remote Ctrl === +
- +
-This should still be running too, but there are a few things to do here.  +
- +
->If for some reason you need to re-start eRemoteControl,​ here's how: +
->  * [[operations:​starting_eRemoteCtrl|Start e-Remote Ctrl]] +
-eRemoteControl will still be writing a local log file and pcfs_log_monitor will still be monitoring it. Although that log file has a name that doesn'​t match the new experiment, it's not essential to start a new local log (as it's only used locally and the master log on the pcfs is used at the correlator).  +
- +
-> If you do want to start a new local log and monitor it, click on the red dot icon in econtrol to make the green bar go away: \\ {{:​operations:​econtrol_log_recording.png|}}.  +
-> Then [[operations:​documentation.logmonitor_eremotectrl|set up the log monitor]] to provide warnings and alarms +
- +
-To keep all log messages related to the new experiment ​in the same file, the next thing to do is +
-  * [[operations:​eRemoteCtrl_newPCFSLog|Open ​a new PCFS log file]] +
-  * Now start the Checklist by loading the procedure file for the experiment, then ''​exper_initi''​. e.g for R1556 at Hobart: <​code>​proc=r1556hb +
-exper_initi</​code>​ +
- +
-The checklist should now be visible. The items in the list are described below and can be ticked off as you go, but many can be skipped in a follow-on experiment. +
- +
-=== The Before Observation Checklist === +
- +
-The RF and IF signal paths, and DBBC are now configured through the Field System.  +
- +
-== RF and IF signal path == +
- +
-Should already be setup and correct. **Skip.** +
- +
->For referenece here are notes on [[operations:​rf_if_check_9.10.5|configuring the RF and IF signal paths]] +
- +
-== DBBC Configuration == +
- +
-The DBBC should already be running and the PCFS should be able to communicate with it.  +
- +
->There should be no need to start any software on the DBBC but for reference here are the notes on setting up the DBBC: +
->  * [[operations:​documentation.DBBCSetupIVS_9.10.5|DBBC Setup]] +
- +
-At the start of every scan, the pcfs executes the ''​setupsx''​ or ''​setup8f''​ procedure (depending on experiment type) which sets the DBBC input channels, filters and BBC frequencies (and configues the Mark5). However, it's a good idea to execute it now so that a recorder test can be made: +
-  * For R1, R4, RD, RDV, T2, ANZ, CRDS, CRF, OHIG:<​code>​setupsx</​code>​ +
-  * For AUST, CONT:<​code>​setup8f</​code>​ +
-''​setupsx''​ (or ''​setup8f''​) calls ''​ifdsx''​ which sets up the DBBC Conditioning module inputs, filters and power levels, then ''​dbbcsx8''​ (or ''​dbbcsx4''​ for a 4 MHz DBBC mode)  which sets the BBC frequencies. Then if you type: +
-<​code>​iread</​code>​ +
-You will see what the current Conditioning module settings are. Output format is: +
-<​code><​time>/<​Module label>/<​IF input number>,<​Auto or Manual gain control>,<​Nyquist filter number>,<​Target power level>,<​attenuation>,<​actual power level></​code>​ +
-Check there’s agreement with what appears in ifdsx. The actual power level should agree pretty well with the target level. The attenuation number can be anywhere between 0 (none) and 63 (maximum). If you see it at 0 or 63, it means the Conditioning module is having trouble getting the power to the right level. You may want to contact the on-call person in this case for help on getting the levels in range. +
- +
-Then type: +
-<​code>​bread</​code>​ +
-You will see what the DBBC has set the BBC freqs to (compare with the .prc file). Output looks like this: +
-<​code>​<​time>/<​bbc name>/<​Frequency (MHz)>/<​Conditioning module in use>,<​Bandwidth (MHz)>,​...</​code>​ +
-the Frequency, Conditioning module label and bandwidth should agree with the listing in ''​dbbcsx8''​ (or ''​dbbcsx4''​).  +
- +
-== Check the maser == +
-  * [[operations:​maser_check|Hydrogen Maser check]] +
- +
-== Mark5 Recorder setup == +
-Next the Mark5 recorder needs to be prepared for the observations.  +
- +
-If you need to swap disk modules: +
-  * Notes on [[operations:​documentation.ivs.mark5_v2_swap_modules|swapping between Mark5 disk modules]] +
- +
-Now check that the times are all OK:+
  
 +At this point it's probably worth verifying that the recorder has stopped:
 <​code>​mk5=dot?</​code>​ <​code>​mk5=dot?</​code>​
 +If the response includes the text ''​FHG_off''​ then the recorder is off. However, if you see ''​FHG_on''​ then you're still recording, so stop the recorder with <​code>​disk_record=off</​code>​
  
-look for ''​syncerr_eq_0'',​ and that the final number is less than ~5 ms. +If not running already, [[operations:​documentation:​start_monica|Start MONICA]] and choose ​the pre-set ​''​aust_check'' ​profile from the GUI for the antenna. e.g. for hobart ​<​code>​Navigator -> hb -> aust_check</​code> ​This will plot antenna coordinates, tracking errors ​and motor currents against time.
-If you see something like ''​syncerr_gt_3''​ then ''​fmset''​ may need to be run to synchronise the clocks. To do this, open a terminal on pcfs[hb|ke|yg], and type +
-<​code>​fmset</​code>​ +
-When the interface appears, type +
-<​code>​s</​code>​ +
-then confirm with a  +
-<​code>​y</​code>​ +
-Following this, the clocks should all be synchronised and agree with UT. Press the [Esc] key to exit ''​fmset''​. ​Then check the times again: +
- +
-<​code>​mk5=dot?</​code>​ +
- +
-== Clock and delay check ==  +
-  * [[operations:​documentation.clock_and_delay_check|Checking clock offsets and delays]] +
- +
-== Test recording == +
-Test recording notes are given below. <note important>​Note though that you should only test the module you'll be recording to (i.e. skip the last section where it describes how to test the second module), and //unless you are absolutely sure//, don't clear the disk module:</​note>​ +
-  * [[operations:​documentation.test_recording_fs9.10.5|Making a test recording]] +
- +
-== System Temperature (Tsys) check == +
-  * [[operations:​documentation.test_tsys|Tsys measurement]] check +
- +
-== Antenna checks == +
- +
-**Skip** these.  +
->  * [[operations:​documentation.test_antenna|Antenna checks]] +
- +
-== Weather checks == +
-  * [[operations:​documentation.weather_check|Weather conditions]] +
- +
-== Fringe check == +
- +
-**Skip** this unless you have at least an hour to spare! +
-  * [[operation:​fringe_checking|Fringe checking procedure]] +
- +
-== Send a Station Ready email == +
-  * [[operations:​documentation.station_ready_email|Sending a station ready email to IVS]]+
  
-== Skype == +Now the checks can begin: 
-  ​* Start Skype and begin a chat session with all participating sites plus the on-call person+  - Send the antenna to (Az, El) = (1, 87) deg <​code>​test1</​code>​ 
 +  ​- Wait for it to get there (look for ''​flagr/​antenna,​acquired''​ or watch the system monitor display). While you're waiting, Check the generator status:  
 +    - At Hobart ​and Katherine you can check the status of the backup generator via the System Monitor interface. The default mode is "In Auto, off" which means it's ready to turn on if there'​s a power failure but it's currently switched off. If it's in any other state, query it with the On Call person. 
 +  - Check that the RF and IF signal paths are correct and the attenuators are at their nominal levels. Refer to the notes on [[operations:​rf_if_check_9.10.5|configuring the RF and IF signal paths]] 
 +  - Check the DBBC is properly configured: <​code>​iread</​code>​ You will see what the current Conditioning module settings are. Output format is: <​code><​time>/<​Module label>/<​IF input number>,<​Auto or Manual gain control>,<​Nyquist filter number>,<​Target power level>,<​attenuation>,<​actual power level></​code>​ Check there’s agreement with what appears in the ''​ifdsx''​ definition in the procedure file. The actual power level should agree pretty well with the target level. The attenuation number can be anywhere between 0 (none) and 63 (maximum). If you see it at 0 or 63, it means the Conditioning module is having trouble getting the power to the right level. You may want to adjust the attenuators (see [[operations:​rf_if_check_9.10.5|configuring the RF and IF signal paths]]) to get them back in range.  
 +  - Then type: <​code>​bread</​code>​ You will see what the DBBC has set the BBC freqs to (compare with the .prc file). Output looks like this: <​code><​time>/<​bbc name>/<​Frequency (MHz)>/<​Conditioning module in use>,<​Bandwidth (MHz)>,​...</​code>​ the Frequency, Conditioning module label and bandwidth should agree with the listing in ''​dbbcsxd''​. 
 +  - Check the maser. See the[[operations:​maser_check|Hydrogen Maser check]] notes 
 +  - check that the clock offsets are in range <​code>​clkoff 
 +maserdelay</​code>​ and check the difference reported in the log monitor 
 +  - Make sure Skype is running to allow a chat session with all participating sites plus the on-call person 
 +  - Use the "Audio test" button on the log monitor to check the alarms are audible. 
 +  - Measure the Tsys while at a high elevation <​code>​systemp12</​code>​ 
 +  - [[operations:​documentation.test_antenna#​antenna_pointing_check|Check the pointing]] 
 +  - There are a series of Az/El slews to move the antenna over it's full range. Type each of these commands into eRemoteCtrl in the order they appear. //Wait for eRemoteCtrl to report ''​flagr/​antenna,​acquired''​ before continuing to the next//: <​code>​ 
 +test1 
 +test2 
 +test3 
 +test4 
 +test5 
 +test1</​code>​ These commands send the antenna to the following set of (Az, El) coordinates in turn : (1, 87), (1, 6), (1, 45), (-269, 45), (269, 45), (1,87). The idea here is to exercise the antenna and log the results se we can look for any changes over the days that may indicate mechanical problems. 
 +  - Lastly, export the data collected by MONICA from the MONICA GUI as follows: 
 +    - Select "​Export -> ASCII Data"​ 
 +    - When asked for a file name, give the current obs code and station name (e.g. a1308hb), the word "​check",​ and end with "​.txt"​. For example: <​code>​a1308hb_check.txt</​code>​ 
 +    - Select "​Export -> PNG image"​ 
 +    - As above but a "​.png"​ extension. e.g. <​code>​a1308hb_check.png</​code>​
  
 +Now you can re-start the schedule. Just use the schedule name without any line number suffix. e.g. <​code>​schedule=a1308hb
 +setupsx</​code>​ This will start the next scan that occurs at least 5 min in the future. Check that the antenna goes to the next source in the schedule that satisfies this criterion. ​
  
-===== Starting =====+Now make a note in the log that the schedule has been restarted following system checks. e.g. <​code>"​Schedule resumed following system checks</​code>​ Also, if there were any problems during the checks or anything else of note, now is a good time to do it. For example, if the weather was too cloudy to verify the pointing: <​code>"​Pointing check not successful, probably due to bad weather</​code>​
  
-The usual startup procedure can now be done: [[operations:​starting_monitoring_9.10.5|Starting and Monitoring]] 
  
 ===== Fringe Checks ===== ===== Fringe Checks =====
  
 On 1921-293 every day. Procedure? On 1921-293 every day. Procedure?
/home/www/auscope/opswiki/data/pages/operations/documentation_ops2_austral15day.txt · Last modified: 2014/11/25 03:36 by Jim Lovell

Page Tools

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki