User Tools

Site Tools


operations:documentation_ops2_cont14

This wiki is not maintained! Do not use this when setting up AuScope experiments!

This is an old revision of the document!


CONT14: Notes for operators

Please read these notes well before your first shift. If there is anything you don't understand, please ask Jim, Jamie, Warren or Liza.

Schedule Overview and Module Usage

Please read the Overview Page for information on the observations, start and stop times etc. The google calendar on the schedules page describes experiment and shift times.

Mark5 Nomenclature: “Bank swap” means changing where the data are recorded from one bank to the other. This does NOT mean modules are powered on/off or physically moved. “Module swap” means that a module is powered down, removed, replaced with a new one and powered up.

The CONT14 schedules are named C14nn where (nn = 01, 02, 03, … , 15). They all start at 00:00 UT and finish at 23:57 UT so there is only a 3 minute gap, just enough time to change schedule files and modules (if needed). Every day, an hour is set aside for detailed system checks at each site. They are staggered so that only one antenna is out of the array at a time. System check times for our sites are as follows:

Site Check time (UT) Check time (Hobart)
Yg 03.00 - 04.00 13.00 - 14.00
Hb 04.00 - 05.00 14.00 - 15.00
Ho 05.00 - 06.00 15.00 - 16.00
Ke (Mon - Sat) 23.00 - 24.00 09.00 - 10.00
Ke (Sundays only) 00.00 - 01.00 10.00 - 11.00

We are using a mixture of module sizes at each station. They are organised so that bank swaps occur at the same times at every site. Module swaps will occur during System Check times and are assigned to schedules as follows:

Module Schedule
1 C1401
2 C1402, C1403
3 C1404, C1405, C1406, C1407
4 C1408, C1409, C1410
5 C1411, C1412, C1413, C1414
6 C1415

Each site also has two spare modules which will be used in case of a failure.

Note that it is vitally important that all disk activity stops on the Mark5 recorder before a module is swapped. If this is not done, there's a good chance of losing all the data on the module. Please be very careful. More on this later.

All data are being shipped to Hobart for eTransfer to the Bonn correlator. The last schedule on each module is classified as a “Rapid” experiment which needs to get to Bonn ASAP. The local site staff have been asked to ship off each module as soon as it has been removed from the Mark5.

The sections below describe how operations differ from normal.

26m Antenna

The Hobart 26m antenna (Ho) is also participating in CONT14. The telescope is observing in a 'tag along' mode which means it doesn't record as much data as the other telescopes, so module sizes and the number of scans is significantly less. Data are recorded through the (analog) Mark-IV rack and onto the Mark5A recorder mkv.phys.utas.edu.au. The telescope can be controlled and monitored via eRemoteCtrl and menu items on ops2 exist to launch this and the Log Monitor, so it looks similar to the 12m antennas. However, the equipment, particularly the Mark-IV rack has limited remote control and a power failure (for example) will require a trip to the observatory to reset the equipment.

The other difference is that the S/X receiver on the 26m is cryogenic and can warm up. It's a good idea to keep an eye on the cryo temperatures and warn Brett if they start to increase. More on this and other things to keep an eye on are described in Description of checklist parameters for the 26m telescope.

Schedule and procedure files

All schedule and procedure files for the period from C1401 to C1415 should already be prepared and ready to use before your shift.

Note that for CONT14 the procedure for setting up the Mark5 and DBBC is called setup8f, NOT setupsx.

Monitoring observations

Please carry out the regular two-hourly checks as normal during the observations.

Schedule Changeover

The Schedule Changeover procedure is the same for all telescopes except Katherine where the daily System Checkout coincides either with the start or end of a schedule.

All Stations Except Katherine

There is only a 3 minute gap between the CONT14 schedules (23:57 to 00:00 UT). The end of each schedule has been modified to include commands to swap banks if necessary and then start the next schedule, so it should happen automatically. However, please pay careful attention when this happens. Note that it takes about 1 minute to swap banks as the commands include 30 second pauses either side of the bank swap command. The commands being sent are as follows (where “nn” is the schedule number and “xx” is the two-letter station code):

  1. When no bank swap is needed:
    schedule=c14nnxx,#1
  2. When a bank swap is needed:
    !+30s
    mk5=bank_set=inc
    !+30s
    schedule=c14nnxx,#1

Please make sure:

  • The new schedule starts. Look in the “Status Monitor” section of eRemoteCtrl and check that SCHED is set to the correct file
  • Type the command
    mk5=bank_set?

    and check that the correct module is selected. The first one in the list is the active bank and the number between the slashes is the module capacity in GB. For example:

    17:28:12.83/mk5/!bank_set? 0 : B : HOB+0025/6000/1024 : A : NYAL-019/2000/1024 ;

    says the module in Bank B is active, has a VSN of HOB+0025 and is 6000 GB (6 TB) in capacity.

If for some reason the new schedule hasn't started or the wrong bank is selected then you will need to issue these commands by hand.

Once the new schedule has started:

Schedule Changeover at Katherine

At Katherine the System Checkout period occurs in the hour before a schedule change on every day except Sunday, when it occurs in the hour after a schedule change. In either case, the procedure is to carry out the System Checkout first (see below), then make sure the correct module bank is selected, then start the new schedule by hand.

  1. Do the System Checkout
  2. Type the command
    mk5=bank_set?

    and check that the correct module is selected. The first one in the list is the active bank and the number between the slashes is the module capacity in GB. For example:

    17:28:12.83/mk5/!bank_set? 0 : B : HOB+0025/6000/1024 : A : NYAL-019/2000/1024 ;

    says the module in Bank B is active, has a VSN of HOB+0025 and is 6000 GB (6 TB) in capacity.

If for some reason the the wrong bank is selected then you will need to issue these commands by hand:

  1. mk5=bank_set=inc
  2. wait about 15 sec
  3. mk5=bank_set?
  4. Start the schedule. The command to send is as follows (where “nn” is the schedule number and “xx” is the two-letter station code):
  5. schedule=c14nnxx,#1

Please make sure the new schedule starts. Look in the “Status Monitor” section of eRemoteCtrl and check that SCHED is set to the correct file and the time next to NEXT is in the (near) future.

Once the new schedule has started:

Observatory Checking

Note that the observatory checking procedure requires you to interrupt the schedule and miss some scans. This allows us to skip checks if necessary but also means if the checks take less than the allocated hour we can be observing again as soon as possible.

Firstly, check the schedule summary printout and find out which is the first scan to finish after the start of the check period (check times for each antenna are given at the top of this page). The idea is to stop the schedule while no data are being recorded. For example, if the check time is 19:00 UT, then in the schedule below, it could be halted after 19:05:45 UT but before 19:07:43 UT:

                                         Start     Start     Stop                      Record
 Scan      Line#  Source   Az El Cable  Record      Data      Data      Dur    Gbyte
date = 2013NOV21  DOY = 325
.
.
.
 325-1854   140 1610-771 170  6 CCW    18:53:58  18:54:08  18:58:33    4:25    35.0  *    
 325-1859   159 1255-316 119 19 CCW    18:59:02  18:59:12  19:05:45    6:33    43.8  *    
 325-1907a  178 0131-522 218  7 CCW    19:07:43  19:07:53  19:20:57   13:04    56.7  *    
.
.
.
  • Now, wait for the scan gap and look for a disk_record=off in the log to confirm that the command to stop recording has been sent, and then type:
halt
  • wait about 10 sec and verify that the recording has stopped with a
    mk5=dot?

    command. Look for FHG=off in the response. If you see FHG=on then the Mark5 is recording and you should stop it with a

    disk_record=off

    . Repeat this step if necessary.

  • Now make a note in the log that the schedule is being stopped for system checks. e.g.
    "Halting schedule for system checks
  • If this System Check period is one allocated for a Module Swap, do it next in coordination with the local person on-site:

If not running already, Start MONICA and choose the pre-set aust_check profile from the GUI for the antenna. e.g. for hobart

Navigator -> hb -> aust_check

This will plot antenna coordinates, tracking errors and motor currents against time.

Now the checks can begin. Note that the procedure is different for the 12m antennas and Hobart 26m:

12m antenna Checkout Procedure

  1. Send the antenna to (Az, El) = (1, 87) deg
    test1
  2. Wait for it to get there (look for flagr/antenna,acquired or watch the system monitor display). While you're waiting, Check the generator status:
    1. At Hobart and Katherine you can check the status of the backup generator via the System Monitor interface. The default mode is “In Auto, off” which means it's ready to turn on if there's a power failure but it's currently switched off. If it's in any other state, query it with the On Call person.
  3. Check that the RF and IF signal paths are correct and the attenuators are at their nominal levels. Refer to the notes on configuring the RF and IF signal paths
  4. Check the DBBC is properly configured:
    iread

    You will see what the current Conditioning module settings are. Output format is:

    <time>/<Module label>/<IF input number>,<Auto or Manual gain control>,<Nyquist filter number>,<Target power level>,<attenuation>,<actual power level>

    Check there’s agreement with what appears in the ifd8f definition in the procedure file. The actual power level should agree pretty well with the target level. The attenuation number can be anywhere between 0 (none) and 63 (maximum). If you see it at 0 or 63, it means the Conditioning module is having trouble getting the power to the right level. You may want to adjust the attenuators (see configuring the RF and IF signal paths) to get them back in range.

  5. Then type:
    bread

    You will see what the DBBC has set the BBC freqs to (compare with the .prc file). Output looks like this:

    <time>/<bbc name>/<Frequency (MHz)>/<Conditioning module in use>,<Bandwidth (MHz)>,...

    the Frequency, Conditioning module label and bandwidth should agree with the listing in dbbc8f8.

  6. Check the maser. See theHydrogen Maser check notes
  7. check that the clock offsets are in range
    clkoff
    maserdelay

    and check the difference reported in the log monitor

  8. Make sure Skype is running to allow a chat session with all participating sites plus the on-call person
  9. Use the “Audio test” button on the log monitor to check the alarms are audible.
  10. Measure the Tsys while at a high elevation
    systemp12
  11. There are a series of Az/El slews to move the antenna over it's full range. Type each of these commands into eRemoteCtrl in the order they appear. Wait for eRemoteCtrl to report flagr/antenna,acquired before continuing to the next:
    test1
    test2
    test3
    test4
    test5
    test1

    These commands send the antenna to the following set of (Az, El) coordinates in turn : (1, 87), (1, 6), (1, 45), (-269, 45), (269, 45), (1,87). The idea here is to exercise the antenna and log the results se we can look for any changes over the days that may indicate mechanical problems.

  12. Lastly, export the data collected by MONICA from the MONICA GUI as follows:
    1. Select “Export → ASCII Data”
    2. When asked for a file name, give the current obs code and station name (e.g. c1405hb), the word “check”, and end with “.txt”. For example:
      c1405hb_check.txt
    3. Select “Export → PNG image”
    4. As above but a “.png” extension. e.g.
      c1405hb_check.png

Now you can re-start the schedule. Just use the schedule name without any line number suffix. e.g.

schedule=c1408hb
setup8f

This will start the next scan that occurs at least 5 min in the future. Check that the antenna goes to the next source in the schedule that satisfies this criterion.

Now make a note in the log that the schedule has been restarted following system checks. e.g.

"Schedule resumed following system checks

Also, if there were any problems during the checks or anything else of note, now is a good time to do it. For example, if the weather was too cloudy to verify the pointing:

"Pointing check not successful, probably due to bad weather
/home/www/auscope/opswiki/data/attic/operations/documentation_ops2_cont14.1398838875.txt.gz · Last modified: 2014/04/30 06:21 by Jim Lovell