User Tools

Site Tools


operations:documentation_ops2_pcfs9.10.5plus

This wiki is not maintained! Do not use this when setting up AuScope experiments!

This is an old revision of the document!


Operator Documentation for observations

Off-site experiment monitoring notes offsite

Background

IVS Observation Instructions

Below are the links to a step-by-step procedure for making IVS observations on the 12m and 26mtelescopes. The current version of e-RemoteCtrl has a built-in checklist which can now be used during the setup procedure. So the first task is to start the Observing and monitoring software at each site, then e-RemoteCtrl.

Monitoring Observations on Parkes 64-m

Observations on the 12m telescopes from the Control Room

These are notes on IVS observations from ops8, the six-monitor PC in the AuScope VLBI Operations room in Hobart.

If you are unfamiliar with the computers and software used at the sites and in the control room, please read this before continuing:

Before the Observations

Prepare the schedule file

* Note - this script will now ask if there is a SAR satellite pass to be observed during the experiment (or immediately afterwards). Please check the Google calendar before answering! If there is a track, then the stations SNP file will be automatically edited to accommodate the scan & you will be given a list of shortened or skipped scans - record these in the handover notes!

  • During normal telescope operation you can do all telescopes with one command via
    schedules <experiment name> <st1> <st2> ... <st4> 

    or

    schedules $(nobs -p)

    after checking that this will produce the correct result by typing

    nobs -p

    into the terminal. Note that this will assume that there are no SAR tracks during the schedule to be drudged.

  • On occasions, the Hobart data are written to disk between scans for e-transfer to Tsukuba for ultra-rapid EOP observations. Some additional changes to the schedule files are required in this case:
  • The status of the Ultra-rapid experiments is published in real-time on the Tsukuba Ultra-rapid web page.

Timing PC Software

Check that the software on the timing PCs is running:

Field System Software

monitoring windows for ops8

  • To quickly open the necessary windows on ops8 for monitoring the telescopes (e-RemoteCtrl, Log Monitor, System Monitor, delaylive, tsys monit3) first make sure these are already closed for each telescope you plan to open them for by entering
    windex

    into a terminal (and manually closing any stragglers). Then enter

    windows hb ke yg

    feel free to omit any telescopes not in use.

  • Following this enter
    windowsweb

    to open browser windows to the VLBI webcams, Autocorrelation spectra, and Auscope Handover page.

  • You will need to login to each e-RemoteCtrl window, start recording the log, and open that log in the Log Monitor as instructed below.

System Monitor

Monica

Mark5 Recorder setup

Next the Mark5 recorder needs to be prepared for the observations.

e-Remote Ctrl

The checklist should now be visible. The items in the list are described below and can be ticked off as you go

  • For your convenience you may wish to use Multi Command Input – moprin (written by Dave). The link to moprin is located in the upper left screen (alternatively open a terminal on ops8 and type 'moprin'), together with the other helpful tools like Tsys displays and backup control – ssh-based remote operator input. With moprin you can send the same command to all three telescop operator input simultaneously. Useful for pointing, ckloff, maserdelay, terminate at the end of an experiment and so on. Also you can incert the telescope name by typing $s instead. For example, if you wish to set up relevant log names at each station, you can type
    log=crf86$s

    You can use it for setting up procedures and starting schedules as well. For more information (e.g. how to change the stations, type 'help' into the moprin terminal.

The Before Observation Checklist

The RF and IF signal paths, and DBBC are now configured through the Field System.

RF and IF signal path
DBBC Configuration

The Digital Base-Band Converter (DBBC) takes the IF signals from the telescope and digitises them for the Mark5 recorder. The DBBC needs to be properly configured prior to an observation. Check that the DBBC is healthy and the server software is running:

To configure the DBBC with the correct IF and frequency settings, use the following command

setupsx

This could be setup8f for the 512 R1, or setup01 for Hart schedules 'crds—', some 'aua—' setupsx calls ifdsx which sets up the DBBC Conditioning module inputs, filters and power levels, then dbbcsx8 (or dbbcsx4 for a 4 MHz DBBC mode) which sets the BBC frequencies. Then if you type:

iread

You will see what the Conditioning module settings are. Output format is:

<time>/<Module label>/<IF input number>,<Auto or Manual gain control>,<Nyquist filter number>,<Target power level>,<attenuation>,<actual power level>

Check there’s agreement with what appears in ifdsx, as well as the values listed here. The actual power level should agree pretty well with the target level. The attenuation number can be anywhere between 0 (none) and 63 (maximum). If you see it at 0 or 63, it means the Conditioning module is having trouble getting the power to the right level. See the Current Issues page for information on how to do this.

Then type:

bread

You will see what the DBBC has set the BBC freqs to (compare with the .prc file). Output looks like this:

<time>/<bbc name>/<Frequency (MHz)>/<Conditioning module in use>,<Bandwidth (MHz)>,...

the Frequency, Conditioning module label and bandwidth should agree with the listing in dbbcsx8 (or dbbcsx4).

Check the maser

Check that a suitable module is in the Mark 5 recorder.

If there is no specific module allocated to the experiment, the relevant column on the spreadsheet will be left blank.

Clock and delay check
Test recording
System Temperature (Tsys) check
Antenna checks
Weather checks
Fringe check
Skype
  • Start Skype and begin a chat session with all participating sites plus the on-call person

Send a Station Ready email

Lastly, after submitting the checklist form, send a ready message to IVS:

Starting and Monitoring

After the Observations

eVLBI observations

Fringe Checking

Alarms and Errors documentation

New content! Please add! Here is an ever growing list of common errors and alarms that you may come across whilst observing. Some are diagnostics and others are plain alarms. If you come across more while observing please add them to this page and note the data and time of the occurrence and your fix. Or, if you couldn't resolve it leave it blank so observers know that it is a problem that was encountered and this would be a good time to call an on-call person List of alarms and fixes.

Observations on the 26m telescope from the Control Room

Note: these notes are based on the 26m setup notes from the ra-wiki and are still in a terse, draft form - Jim Lovell

Before Setup:

1. Before you drudg, start the correct fs-mk5hb (fs-auscope?) for IVS which runs a script to select the correct backend (mk5 and DBBC). Drudg schedule for the experiment (if it has not been done already). Drudg the schedule using slogit.auscope.sh on ops-serv2 as for the other stations. Having the backend selected correctly will result in the correct proc file being generated.

From ops8, confirm that the sum file has been copied across to /vlbobs/ivs/sched/. Otherwise, ssh into hobart (ssh oper@hobart) and copy the .sum file with the command

scp /tmp/sched.tmp observer@ops-serv2:/vlbobs/ivs/sched/aug019ho.sum

.

If the .sum file was not written:

a) before the experiment started: drudge the schedule manually. On Hobart, go to the sched directory and type drudg r4705.skd. Then type Ho for station Hobart and then 3 to create the prc file and 5 to write the sum file. The sum file is written in the ./tmp directory as sched.tmp.

b) after the experiment has started: in order not to overwrite the prc file, copy the schedule file to the tmp directory and drudg it there.

2. Check to see if Jim P (or anyone else!) is observing.

CHECK THE CALENDAR!

Go to the 26m Live Page for the 26m and see if the telescope is tracking. Jim P puts a sticky note on the screen 4 of the newsmerd:1 VNC session where the Field System is saying he is observing also.

Please wait until the other person's experiment has finished until you set up.

If it is urgent, please see the notes on his instructions page on how to cancel the Vela observations:

http://ra-wiki.phys.utas.edu.au/index.php?n=MtPleasant26m.PulsarObserving

DO NOT SHUT DOWN THE FIELD SYSTEM. The field system is used to monitor the S-X temperatures, and if you shut down the field system, it will stop the temperature monitoring which is essential for the observations

Setup of Telescope:

Step 1. Open a VNC viewer to newsmerd

vncviewer newsmerd:1

The password is the usual one. (hobart26 FS runs on computer named hobart). If this is not accessible ssh to observer@newsmerd then restart the vnc with

 vncserver :1 -shared & 

Step 2. Check that the field system is running on newsmerd. Usually, the Geodesy window will be Desktop 4. If it is not running, please start it with fs-mk5hb to ensure that the geodetic backend is selected.

Step 3. Check receiver position

IS THE S-X RECIEVER ON AXIS? MAKE SURE IT IS!

Check the receiver position with the command rxp. It should be -256, +248, +18 (x,y,z).

Be sure the telescope is stowed BEFORE changing the reciever

To change it go to Desktop 3, bottom right window and enter '1' (OTTER interface). If the OTTER interface isn't open, then open a new terminal and type ./OTTER_interface.sh. Changing the receiver takes a couple of minutes (check on the live page http://ra-state.phys.utas.edu.au/ until the antenna stops moving - 5 sec update rate).

Check that the receiver is actually moving, and wait until it reaches position before moving the telescope (can take a few minutes if moving from L-Band).

Stuck Receiver

If the receiver is stuck, first try a reset in the OTTER interface on the newsmerd vnc desktop 3 (type r and hit enter).

Sometimes the OTTER interface crashes/fails to communicate with the reciever. If the GUI fails to start or returns error messages, please check that it is running (with ps -ef | grep OTTER as observer@rx26m). You can restart the server with sudo /etc/init.d/OTTER restart if there are any problems. The server is being continuous polled by MoniCA with the data being logged in the hobart database. More information on the OTTER server can be found hereOTTER

Step 4. Check the procedure file The procedure file will be in /usr2/proc and called <experiment name>ho.prc. It should not need editing, and should be similar to recent .prc files using the same mode\

Step 5. Mark 5 setup:

The 26m now uses uses mk5hb (oper@mk5hb) for recording.

Log in to oper@mk5hb and confirm that DIMino is running.

ps auxww | grep -i DIM

If DIMino is running, it will return an output like this:

root     10832  5.2  0.5  40564 12984 ?        Rl   May08 2666:16 DIMino -m0
oper     32071  0.0  0.0   2852   544 pts/2    D+   04:14   0:00 grep -i DIM

If the first line in this output does not appear, DIMino is not running. Run it with

DIMino -m0 &

Check the VSN of the module using

tstDIMino
vsn?

Check if there is any data on the module by using the command DirList. If there is (and there is supposed to be data on there), note the end pointer (as in how much data has been recorded so far on that module) in the Handover Notes

NOTE: If there is no free space on a module in the Mark 5, call your on-call person for assistance in changing modules.

Step 6. Start eRemoteCtrl.

It is availalbe from the drop down menus as it is for the 12m telescopes.

Step 7. Start the Log Monitor as for the 12m telescopes (from the drop down menu). Start a new log file and begin recording to it.

Step 8. Load the experiment procedure file (if you haven't already)

proc=rd1234ho

note the 'ho' on the end for Hobart 26m.

Step 9. Run the setup of the experiment using

 setupsx
  1. Run the setup. The command for an R1 experiment is setupsx but look in the experiment procedure library printout to find the correct setup procedure name. Watch for warnings.
  2. Check all the values reported by iread and bread are nominal.
  3. Check times are ok with fmset on oper@hobart.
  4. Set the cal to be controlled by PCFS.
    observer@newsmerd: calu -m pcfs -r rakbus
  5. Check that caltsys works. Caltsys should return results between 50 and 70K
  6. Check that weather works. wth
  7. Check that fmout-gps works. clkoff. Note that the readings are collected from pcfshb at this stage. If clkoff returns an error, or fixed readings, check that the FS on pcfshb is ruuning & restart it if necessary.
  8. Check that the Mark5 has a blank module big enough for the experiment. Compare the Drudg summary of the schedule with the label on the front of the disk module.
  9. Run disk_pos to check the disk module is empty. Result should be 0/0 otherwise the disk module contains data.
  10. Record some data. disk_record=on
  11. Observe green disk activity lights on module
  12. Stop recording. disk_record=off
  13. Check the recording. with checkmk5
  14. Observe the time and details in the log window for the test recording, and have a look at the autocorrelation spectrum and bit statistics screens.
  15. Start the schedule. schedule=r1186ho,#1
  16. Watch PCFS log for errors.
  17. Check the antenna is on source. onsource

Lastly, send the start message.

Once the schedule starts, watch the log and everything closely for the first scan or so.

Manual Re-starting the drives on the 26-m

Sometimes the drives need to be manually re-started. Documentation on how to do this can be found here.

Monitoring Observations on the 26-m

  • Please note that e-RemoteControl does not adequately show the recorded/free space on a module in mkv. To see how full is the module, please use commands mk5=bank_set (module capacity) and disk_pos (write pointer position).
  • Parameters of the check-list during observation for 26-m are explained here.

After 26m Observations

DO NOT SHUT DOWN THE FIELD SYSTEM. The field system is used to monitor the S-X temperatures, and if you shut down the field system, it will stop the temperature monitoring which is essential for the observations

Send the end message

Park the dish with

 source=stow 

in the Field System

Monitoring Observations on Parkes 64-m

User accounts and email addresses

Gmail accounts have been set up for registration with skype and other services. The usual Observer password applies.

Katherine

gmail: keobserver@gmail.com

Skype: keobserver

Yarragadee

gmail: ygobserver@gmail.com

skype: ygobserver

Hobart 12m

gmail: hbobserver@gmail.com

skype: hbobserver

/home/www/auscope/opswiki/data/attic/operations/documentation_ops2_pcfs9.10.5plus.1585187109.txt.gz · Last modified: 2020/03/26 01:45 by Jamie McCallum