====== Starting and Monitoring IVS sessions ====== Make sure that the field system software is running in the pcfs VNC session so that econtrol can connect to it. From ops4 in the operations room, start econtrol: Applications -> AuScope [Hobart|Katherine|Yarragadee] -> econtrol Click on the boxes next to “StatusMonitor" and "Logging and Operator Input”. This should trigger the program to connect to pcfs[hb|ke|yg] and you’ll see a status window and log window. If you don't see any information updating in the status window, type [ctrl]-[shift]-e to start the server process on the field system. You can send PCFS commands and comments from the bottom of the log window. Note you won’t see the log window update until the next scheduled command is sent. Set up the log monitor to provide warnings and alarms: * [[operations:documentation.logmonitor| setting up the log monitor]]. (As a backup, pmSTALM can be used but it checks for fewer error conditions. This is described [[operations:documentation.pmstalm| here]].) For Hobart and Katherine, start the schedule with e.g. schedule=r4447hb if there is longer than 5 minutes until the nominal start of the schedule. For Yarragadee, ALWAYS start the schedule with schedule=r4447yg,#1. This is necessary because the snp file has been edited to deal with the transmission stows which confuses the fs. Unless: you try to start late within five minutes of the scheduled start and get errors like; //m5 -900 no scans//;// m5 -900 not while recording or playing//;// m5 -900 can't get device info//. The mk5 is looking for the previous scheduled scan to check, which doesn't exist, and can't start the next scheduled scan recording. In this case start schedule with start line number schedule=r4447hb,#1 Otherwise: do NOT specify a start line number. [[operations:startmessage|Then send a start message to IVS]]. During the experiment, the following checks can be made. Please log them using the new Checklist GUI: * [[software:auscope_checklist|Using the Checklist GUI]] Please go through the checklist as often as you like but **at least once every 2 hours**. A description of the checklist parameters and what to look out for is available here: * [[operations:checklist_parameters|Description of checklist parameters]] The **system monitor** provides a useful summary of the drives & other Monica parameters. You can run it on ops4 or ops5 with the command ''monitor_system.pl'' or from the drop-down menu on the desktop "Applications -> AuScope -> Monitor System". Run it once for each site. You can launch clock displays too from the "Applications -> AuScope" menu. It's best to only have the Katherine and Yarragadee VNC sessions running when running through the checklist and to rely on econtrol, the system monitor and the log monitor the rest of the time. You should use the VNC sessions to check that the autocorrelation spectra are OK though. [[http://auscope.phys.utas.edu.au/webcams_etc.html|This web page]] provides a one-page summary of the webcams and weather radar. The PC in the "lounge" (ops6) runs the same PCFS log monitor that ops4 does. To start it, double-click on the "Log monitor" icon on the desktop and then open the same log file that econtrol is writing to. Run it once for each station. ===== Other things to watch out for ===== ==== Yarragadee stows due to USN transmissions ==== ==== Wind stows ==== ==== Recovering from power failures ==== source=disable to reinitialise the connection. ==== Re-starting observations, or starting late ==== Starting a schedule file with no additional arguments will start the observations according to the schedule, with the first observation beginning no earlier than 5 minutes from now. This is usually the best option. If you want to specify a particular part of the file to start in then you can do it as follows (taken from the manual): Syntax: schedule=name,start,#lines Response: schedule/name,line Settable parameters: name Name of schedule file to be started. If no directory path is specified, /usr2/sched assumed. If no extension is specified, .snp is assumed. Any currently-executing schedule file is closed, and the new schedule file is opened. If the new file cannot be opened, there will be no schedule active. When a valid schedule is started, a cont command may be necessary. start Place in the schedule to begin executing. May be one of the following: null to start with the observation beginning no earlier than 5 minutes from now. #line for a line number in the file, should be a source command. time to start with the observation beginning no earlier than this time. time is in standard SNAP format. #lines Number of lines to execute before automatically halting. Default is the remainder of the schedule. Monitor-only parameters: line The line number to be executed next. Comments: If the schedule is started successfully, a log file having the same name as the schedule is automatically started, and the procedure file having the same name as the schedule is automatically established as the schedule procedure library. Any previously time-scheduled procedures from this library are cancelled. If a # of lines is specified, an automatic halt will be issued after execution of these lines. The schedule may then be continued using the cont command. ===== Common problems ===== ==== Formatter to FS time offset ==== You might get a ERROR sc -13 setcl: formatter to FS time difference 0.5 seconds or greater to fix this do a: sy=run setcl offset Note this error is likely to reappear regularly. Note also that the error message ?ERROR sc -18 setcl: program is already running, try "run setcl" instead. has been seen recently when the command is issued from a terminal window. The problem has not been seen when the command is entered into the oprin window. If you do get this error when entering the command into the oprin window, please tell Jim. ==== FS time is out by several seconds ==== The origin of this problem is presently unknown but the FS time can get seriously out of step. To fix this, **while not recording** start the ''fmset'' program from an ''oper@pcfshb'' terminal and issue the "+" and "-" commands, then quit from fmset (ESC). Restart fmset and the FS time should now be correct. You may need to resync the mark5B pps after this procedure. ==== clkoff reading is drifting or far from the maser-GPS offset ==== This usually is caused by the DBBC. First, go around the back of rack 14 and move the cable from the DOTMON output of the Mark5B to the "1 PPS Mon" output of the DBBC (left hand side, sixth SMA from the top). If the same offset is seen on the counter, then the problem in in the DBBC. A temporary fix can be achieved with ''pps_sync'' in DBBC Control but this did not reliably fix the problem on 13/10/10. Instead, try reconfiguring the DBBC with ''reconf'' - this will take ~two minutes in total and you will need to re-issue the ''dbbcifa=...'' commands, and resynch the mark5B with ''fmset''. ==== PCFS log window reports problem with ReadPower.sh ==== This occurs when communication with the power sensor (a USB device) in the IF rack is lost. The power sensor is required for System Temperature (Tsys) measurements. The solution is to cycle power to the sensor by unplugging it's USB connection into the Field System PC and then plugging it back in again. If you are not at the site, and cannot contact anyone on-site to fix it, you can disable the Tsys measurements as follows: On pcfshb: pfmed pfmed: pf,station pfmed: ed,systemp12 An editor will start. Comment out the command by putting a double-quote at the start of the line. It should then look like this: "sy=/usr2/oper/systemp12rcp.sh & Now exit the editor, and pfmed: exit Lastly, please make a note in the log that Tsys measurements have been disabled. It is possible to remotely reset the power sensor at all three telescopes. You should first follow the procedure outlined above, then kill any remaining ''systemp12rcp.sh'' or ''ReadPower.sh'' processes running on pcfs[hb|ke|yg] (use ''ps -ef | grep ReadPower'' to identify the process IDs). Become root with ''su'' and issue the command /etc/init.d/AgilentU2000 restart It will run a series of procedures to toggle the power and then try to re-establish communications. It may take two tries to get it fully working - when it is ok, you should get a blithely cheery message to this effect, and be wished good luck. When you receive this message, wait for a break in the recording and test the power sensor by running ''/home/oper/systemp12rcp.sh''. All being well, there should be no timeouts although the measured power is likely to be nonsensical (there will be bogus values written into the data from the previous timeouts). If it fails with timeouts, persevere with the ''/etc/init.d/AgilentU2000 restart'' procedure. Once you have it working, repeat the pfmed process and remove the comment from the systemp12 procedure. ==== If econtrol gets closed during an observation ==== Recording continues as econtrol is a front-end viewer for the field system, so don't panic :) When you restart econtrol from the menu it may be unable to load the telescope information (the drop-down menu boxes), and the terminal from which econtrol runs produces "Can't open interface" type errors. If this happens, in the econtrol window (the green one, not the terminal) press ''Control+shift+e'', and then try to open one of the drop down boxes again - this time the icon in the bottom right corner should go from red through 'connecting' to green, the information will now load, and observing can continue as normal. When econtrol is back, check that there is a green bar above the red dot, second icon to the right of the text entry field. This indicates that a log file is being recorded. If there is not a green bar on this button, press the button and specify an appropriate filename in ''/vlbobs/ivs/logs''. Then in the Log Monitor, choose File > Open Log File and select the new file. Make a note that there are two log files. ==== If econtrol can't connect ==== Recording continues as econtrol is a front-end viewer for the field system, so don't panic :) If the econtrol program can't connect even after repeated ''Ctrl+Shift+e'' commands, you should check to make sure that the econtrol daemon is running on the pcfs machine. Log in to pcfshb and run ''ps -ef | grep econtrold''. There should be two entries in the list. If it's not running then start it with ''/usr2/econtrol/bin/econtrold''. If it is running at first but you still can't connect, try killing the econtrold processes and restarting it.