====== CONT14: Notes for operators ======

<note important>Please read these notes well before your first shift. If there is //anything// you don't understand, please ask Jim, Jamie, Warren or Liza.</note>


  * [[#schedule_overview_and_module_usage|Observing schedule and module usage overview]]
  * [[#26m_antenna|The Hobart 26m antenna]]
  * [[#schedule_and_procedure_files|Schedule and procedure files]]
  * [[#monitoring_observations|Monitoring observations]]
  * [[#schedule_changeover|Schedule changeover procedure]]
  * [[#observatory_checking|Observatory checking procedure]]

===== Schedule Overview and Module Usage =====

Please read the [[operations:documentation_cont14_overview|Overview Page]] for general information on the CONT14 observations as well as session notes from IVS. The google calendar on the [[operations:schedule|schedules page]] describes experiment and shift times.

<note>Mark5 Nomenclature: "**Bank swap**" means changing where the data are recorded from one bank to the other. This does NOT mean modules are powered on/off or physically moved.
"**Module swap**" means that a module is powered down, removed, replaced with a new one and powered up.
</note>

The CONT14 schedules are named ''C14nn'' where ''(nn = 01, 02, 03, ... , 15)''. They all start at 00:00 UT and finish at 23:57 UT so there is only a 3 minute gap, just enough time to change schedule files and banks (if needed). Every day, an hour is set aside for detailed system checks at each site. They are staggered so that only one antenna is out of the array at a time. System check times for our sites are as follows:

^  Site  ^  Check time (UT)  ^  Check time (Hobart)  ^
| Yg    | 03.00 - 04.00 | 13.00 - 14.00 |
| Hb    | 04.00 - 05.00 | 14.00 - 15.00 |
| Ho    | 05.00 - 06.00 | 15.00 - 16.00 |
| Ke (Mon - Sat)   | 23.00 - 24.00 | 09.00 - 10.00 |
| Ke (Sundays only) | 00.00 - 01.00 | 10.00 - 11.00 |

We are using a mixture of module sizes at each station. They are organised so that bank swaps occur at the same times at every site. Module swaps will occur during System Check times and are assigned to schedules as follows:

^ Module ^ Schedule ^
| 1 | C1401 |
| 2 | C1402, C1403 |
| 3 | C1404, C1405, C1406, C1407 |
| 4 | C1408, C1409, C1410 |
| 5 | C1411, C1412, C1413, C1414 |
| 6 | C1415 |

Each site also has two spare modules which will be used in case of a failure.

//Note that it is vitally important that all disk activity stops on the Mark5 recorder before a module is swapped. If this is not done, there's a good chance of losing all the data on the module. Please be **very** careful. More on this later.//

All data are being shipped to Hobart for eTransfer to the Bonn correlator. The last schedule on each module is classified as a "Rapid" experiment which needs to get to Bonn ASAP. The local site staff have been asked to ship off each module as soon as it has been removed from the Mark5.

The sections below describe how operations differ from normal. 

===== 26m Antenna =====

The Hobart 26m antenna (Ho) is also participating in CONT14. The telescope is observing in a 'tag along' mode which means it doesn't record as much data as the other telescopes, so module sizes and the number of scans is significantly less. Data are recorded through the (analog) Mark-IV rack and onto the Mark5A recorder ''mkv.phys.utas.edu.au''. The telescope can be controlled and monitored via eRemoteCtrl and menu items on ''ops2'' exist to launch this and the Log Monitor, so it //looks// similar to the 12m antennas. However, the equipment, particularly the Mark-IV rack has limited remote control and a power failure (for example) will require a trip to the observatory to reset the equipment.

The other difference is that the S/X receiver on the 26m is cryogenic and can warm up. It's a good idea to keep an eye on the cryo temperatures and warn Brett if they start to increase. More on this and other things to keep an eye on are described in [[operations:documentation.eRemoteControl_checklist_midob26|Description of checklist parameters for the 26m telescope]].

Also please note that the VNC session that is running the Field System software can be found by typing <code>vncviewer newsmerd:1</code>

===== Schedule and procedure files =====

All schedule and procedure files for the period from C1401 to C1415 should already be prepared and ready to use before your shift.

<note important>Note that for CONT14 the procedure for setting up the Mark5 and DBBC is called ''setup8f'', NOT ''setupsx''.</note>


===== Monitoring observations =====

Please carry out the regular two-hourly checks as normal during the observations.
  * [[operations:documentation.eRemoteControl_checklist_midob|Description of checklist parameters for AuScope 12m telescopes]]
  * [[operations:documentation.eRemoteControl_checklist_midob26|Description of checklist parameters for the 26m telescope]]

Also please keep the [[operations:handover|Handover notes]] page up to date.
===== Schedule Changeover =====

<note>The Schedule Changeover procedure is the same for all telescopes except Katherine where the daily System Checkout coincides either with the start or end of a schedule.</note>


==== All Stations Except Katherine ====

There is only a 3 minute gap between the CONT14 schedules (23:57 to 00:00 UT). The end of each schedule has been modified to include commands to swap banks if necessary and then start the next schedule, so it should happen automatically. However, please pay careful attention when this happens. Note that it takes about 1 minute to swap banks as the commands include 30 second pauses either side of the bank swap command. The commands being sent are as follows (where "nn" is the schedule number and "xx" is the two-letter station code):    
  - When no bank swap is needed: <code>schedule=c14nnxx,#1</code>
  - When a bank swap is needed: <code>!+30s
mk5=bank_set=inc
!+30s
schedule=c14nnxx,#1</code>

Please make sure:
  * The new schedule starts. Look in the "Status Monitor" section of eRemoteCtrl and check that SCHED is set to the correct file
  * Type the command <code>mk5=bank_set?</code> and check that the [[operations:documentation_cont14_timetable|correct module]] is selected. The first one in the list is the active bank and the number between the slashes is the module capacity in GB. For example: <code>17:28:12.83/mk5/!bank_set? 0 : B : HOB+0025/6000/1024 : A : NYAL-019/2000/1024 ;</code> says the module in Bank B is active, has a VSN of HOB+0025 and is 6000 GB (6 TB) in capacity.
If for some reason the new schedule hasn't started or the wrong bank is selected then you will need to issue the ''bank_set'' and ''schedule'' commands by hand.

Once the new schedule has started:

  * [[operations:startmessage|Send a start message to IVS]]. 
  * Send off the log files from the previous experiment to IVS. Refer to the page [[operations:documentation.ivs.commentsfile|Prepare the comments file and transfer the logs]] for details. Note that all modules for CONT14 experiments from Yarragadee and Katherine are shipped to Hobart, and all data are eTransfered to Bonn for correlation.

==== Schedule Changeover at Katherine ====

At Katherine the System Checkout period occurs in the hour before a schedule change on every day except Sunday, when it occurs in the hour after a schedule change. In either case, the procedure is to carry out the System Checkout first (see below), then make sure the [[operations:documentation_cont14_timetable|correct module bank]] is selected, then start the new schedule by hand.

  - Do the [[operations:cont14_observatory_check|System Checkout]]
  - Type the command <code>mk5=bank_set?</code> and check that the [[operations:documentation_cont14_timetable|correct module]] is selected. The first one in the list is the active bank and the number between the slashes is the module capacity in GB. For example: <code>17:28:12.83/mk5/!bank_set? 0 : B : HOB+0025/6000/1024 : A : NYAL-019/2000/1024 ;</code> says the module in Bank B is active, has a VSN of HOB+0025 and is 6000 GB (6 TB) in capacity. If for some reason the the wrong bank is selected then you will need to issue these commands by hand:
      - <code>mk5=bank_set=inc</code> Wait about 15 sec
      - <code>mk5=bank_set?</code> 
  - Start the schedule. The command to send is as follows (where "nn" is the schedule number and "xx" is the two-letter station code):    
    - <code>schedule=c14nnxx,#1</code>
  - Please make sure the new schedule starts. Look in the "Status Monitor" section of eRemoteCtrl and check that SCHED is set to the correct file and the time next to NEXT is in the (near) future.

Once the new schedule has started:

  * [[operations:startmessage|Send a start message to IVS]]. 
  * Send off the log files from the previous experiment to IVS. Refer to the page [[operations:documentation.ivs.commentsfile|Prepare the comments file and transfer the logs]] for details. Note that all modules for CONT14 experiments from Katherine are shipped to Hobart, and all data are eTransfer-ed to Bonn for correlation.
===== Observatory Checking =====

Note that the observatory checking procedure requires you to interrupt the schedule and miss some scans. This allows us to skip checks if necessary but also means if the checks take less than the allocated hour we can be observing again as soon as possible.

Note that Observatory checks coincide with schedule changeovers at Katherine. See the notes above for the procedure at Katherine, but for all other sites, please read on...

Please follow this [[operations:cont14_observatory_check|Observatory System Checkout]] Procedure

Now you can re-start the schedule. Just use the schedule name without any line number suffix. e.g. <code>schedule=c1408hb
setup8f</code> This will start the next scan that occurs at least 5 min in the future. Check that the antenna goes to the next source in the schedule that satisfies this criterion. 

Now make a note in the log that the schedule has been restarted following system checks. e.g. <code>"Schedule resumed following system checks</code> Also, if there were any problems during the checks or anything else of note, now is a good time to do it. For example, if the weather was too cloudy to verify the pointing: <code>"Pointing check not successful, probably due to bad weather</code>

===== Some Additional Ho 26m Notes  =====
It is useful to be able to graphically view the 20K and 70K receiver temperatures, as well as the pressure vary with time.

On Ops2 start by: 
<code>>> ssh oper@hobart
>> cd /usr2/log/
>> logpl</code>
This will open up a GUI called: **LogPlotter 2**.

Now open a log file by clicking **File** --> **Open Log** in the header of the GUI. In the new window that pops up, input the log name (e.g. c1403ho.log) and open the file.

Go to **Plotting** and select
<code>rx-9-70k
rx-9-20k
rx-9-pres</code>

The plots do no update automatically, so restart the process with **File** --> **Open Log** when an update is required.