====== Operator Documentation for observations ====== ====== Background ====== * Background reading for new operators * [[http://auscope.phys.utas.edu.au|About the AuScope VLBI Project]] * {{:operations:schuh_behrend_2012.pdf|VLBI: A fascinating technique for geodesy and astrometry}} by Schuh and Behrend (2012) * [[http://link.springer.com/article/10.1007/s00190-016-0950-5|International VLBI Service for Geodesy and Astrometry]] by Nothnagel, Arts Behrend & Malkin, 2016 * {{:operations:lovell_etal.pdf|The AuScope Geodetic VLBI Array}} by Lovell et al. * {{:operations:auscope_operator_training.pdf|Operator training documentation}} (PDF). Includes project background, info on scheduling and rosters, overview of the telescopes and related hardware. * [[operations:acronyms|Acronym dictionary]] * [[operations:observerduties|Observer duties and shift times]] * More background reading * There's lots of useful information in the [[http://ivscc.gsfc.nasa.gov/meetings/tow2013/#Notebook|7th Technical Operations Workshop Notebook]]. A couple of items of particular interest: * {{:operations:vlbi_and_science_tow2013.pdf|Science Applications of Geodetic VLBI}} by John Gipson (2013) * {{:operations:bach-calibration.pdf|Pointing and Single Dish Amplitude Calibration Theory}} by Uwe Bach * [[operations:maps|Maps]] ===== IVS Observation Instructions ===== **[[operations:documentation_ops2_pcfs9.10.5plus:offsite|Off-site experiment monitoring notes]]** Below are the links to a step-by-step procedure for making IVS observations on the 12m and 26mtelescopes. The current version of e-RemoteCtrl has a built-in checklist which can now be used during the setup procedure. So the first task is to start the Observing and monitoring software at each site, then e-RemoteCtrl. * [[#[Observations on the 12m telescopes from the Control Room]|Observations with 12-m telescopes]] * [[Revised procedure using scripts]|Setup using scripts]] * [[operations:documentation.eremotecontrol_checklist_midob]|12-m checklist parameter description]] * [[#[Observations on the 26m telescope from the Control Room]|Observations with 26-m telescope]] * [[operations:documentation.eremotecontrol_checklist_midob26]|26-m checklist parameter description]] * [[[operations:AUM-hb|New procedures for the Mixed-Mode (AUM) experiments with Hobart12 (Hb) and Katherine (Ke) telescopes]] ==== Monitoring Observations on Parkes 64-m ==== [[operations:documentation.parkes_monitoring|Parkes monitoring and set-up instructions]] ===== Observations on the 12m telescopes from the Control Room ===== These are notes on IVS observations from ops8, the six-monitor PC in the AuScope VLBI Operations room in Hobart. If you are unfamiliar with the computers and software used at the sites and in the control room, please read this before continuing: * [[[[operations:documentation:computers_software|Introductory notes on computers and software]] ==== Before the Observations ==== === Prepare the schedule file === * [[operations:documentation.ivs.drudg_9.10.5|Preparing the schedule and procedure files]] * Open a terminal and **ssh into observer@ops-serv2** and for each station run: ~/slogit.auscope.sh * Sometimes ~/ is not needed before the slogit command. * Note - this script will now ask if there is a [[operations:SAR|SAR satellite pass]] to be observed during the experiment (or immediately afterwards). Please check the Google calendar before answering! If there is a track, then the stations SNP file will be automatically edited to accommodate the scan & you will be given a list of shortened or skipped scans - record these in the handover notes! * During normal telescope operation you can do all telescopes with one command via schedules ... or schedules $(nobs -p) after checking that this will produce the correct result by typing nobs -p into the terminal. Note that this will assume that there are no SAR tracks during the schedule to be drudged. * On occasions, the Hobart data are written to disk between scans for e-transfer to Tsukuba for ultra-rapid EOP observations. Some additional changes to the schedule files are required in this case: * [[operations:documentation.ivs.drudg_ureop_9.10.5|Additional schedule preparation for Hb ultra-rapid EOP experiments]] * The status of the Ultra-rapid experiments is published in real-time on the [[http://www.spacegeodesy.go.jp/vlbi/dUT1/|Tsukuba Ultra-rapid web page]]. === Timing PC Software === Check that the software on the timing PCs is running: * [[operations:documentation.ivs.timingsw|Software on the timing PC]] === Field System Software === * [[operations:starting_fs|Starting the PC Field System Software]] * [[operations:documentation:fs_antenna_9.10.5|Some notes on antenna control and monitoring]] using the new version of the Field system === monitoring windows for ops8 === * To quickly open the necessary windows on ops8 for monitoring the telescopes (e-RemoteCtrl, Log Monitor, System Monitor, delaylive, tsys monit3) first make sure these are already closed for each telescope you plan to open them for by entering windex into a terminal (and manually closing any stragglers). Then enter windows hb ke yg feel free to omit any telescopes not in use. * Following this enter windowsweb to open browser windows to the VLBI webcams, Autocorrelation spectra, and Auscope Handover page. * You will need to login to each e-RemoteCtrl window, start recording the log, and open that log in the Log Monitor as instructed below. === System Monitor === * [[operations:starting_sysmon|Start the System Monitor]] === Monica === * [[operations:starting_monica|Start MONICA]] == Mark5 Recorder setup == Next the Mark5 recorder needs to be prepared for the observations. * [[operations:documentation.ivs.mark5_v2|Mark5B recorder setup]] === e-Remote Ctrl === * [[operations:starting_eRemoteCtrl|Start e-Remote Ctrl]] * [[operations:eRemoteCtrl_newPCFSLog|Open a new PCFS log file]] * Next, [[operations:documentation.logmonitor_eremotectrl|set up the log monitor]] to provide warnings and alarms * Now start the Checklist by loading the procedure file for the experiment, then ''exper_initi''. e.g for R1556 at Hobart: proc=r1556hb exper_initi The checklist should now be visible. The items in the list are described below and can be ticked off as you go * For your convenience you may wish to use Multi Command Input -- moprin (written by Dave). The link to moprin is located in the upper left screen (alternatively open a terminal on ops8 and type 'moprin'), together with the other helpful tools like Tsys displays and backup control -- ssh-based remote operator input. With moprin you can send the same command to all three telescop operator input simultaneously. Useful for pointing, //ckloff//, //maserdelay//, //terminate// at the end of an experiment and so on. Also you can incert the telescope name by typing //$s// instead. For example, if you wish to set up relevant log names at each station, you can type log=crf86$s You can use it for setting up procedures and starting schedules as well. For more information (e.g. how to change the stations, type 'help' into the moprin terminal. === The Before Observation Checklist === The RF and IF signal paths, and DBBC are now configured through the Field System. == RF and IF signal path == * [[operations:rf_if_check_9.10.5|Configuring the RF and IF signal paths]] * if Hb reports 12,0,10,10 and you want 0,0,10,10 like it says on the above page, type "ifpic=,,0,0,10,10," into eremote (Lucas). == DBBC Configuration == The Digital Base-Band Converter (DBBC) takes the IF signals from the telescope and digitises them for the Mark5 recorder. The DBBC needs to be properly configured prior to an observation. Check that the DBBC is healthy and the server software is running: * [[operations:documentation.DBBCSetupIVS_9.10.5|DBBC Setup]] To configure the DBBC with the correct IF and frequency settings, use the following command setupsx This could be ''setup8f'' for the 512 R1, or ''setup01'' for Hart schedules 'crds---', some 'aua---' ''setupsx'' calls ''ifdsx'' which sets up the DBBC Conditioning module inputs, filters and power levels, then ''dbbcsx8'' (or ''dbbcsx4'' for a 4 MHz DBBC mode) which sets the BBC frequencies. Then if you type: iread You will see what the Conditioning module settings are. Output format is: Check there’s agreement with what appears in ifdsx, as well as the values listed [[operations:documentation.dbbcsetupivs| here]]. The actual power level should agree pretty well with the target level. The attenuation number can be anywhere between 0 (none) and 63 (maximum). If you see it at 0 or 63, it means the Conditioning module is having trouble getting the power to the right level. See the [[operations:currentissues|Current Issues]] page for information on how to do this. Then type: bread You will see what the DBBC has set the BBC freqs to (compare with the .prc file). Output looks like this: the Frequency, Conditioning module label and bandwidth should agree with the listing in ''dbbcsx8'' (or ''dbbcsx4''). == Check the maser == * [[operations:maser_check|Hydrogen Maser check]] Check that a suitable module is in the Mark 5 recorder. * [[operations:documentation.ivs.mark5_module_selection|Which module do I use?]] * [[http://131.217.63.180:8000|Module and Experiment Library]] If there is no specific module allocated to the experiment, the relevant column on the spreadsheet will be left blank. == Clock and delay check == * [[operations:documentation.clock_and_delay_check|Checking clock offsets and delays]] == Test recording == * [[operations:documentation.test_recording_fs9.10.5|Making a test recording]] == System Temperature (Tsys) check == * [[operations:documentation.test_tsys|Tsys measurement]] check * Note that Ke is now using a different continuous Tsys measuring procedure. These notes on [[operations:documentation.contcal|Continuous calibration]] explain what's going on. == Antenna checks == * [[operations:documentation.test_antenna|Antenna checks]] == Weather checks == * [[operations:documentation.weather_check|Weather conditions]] == Fringe check == * [[operation:fringe_checking|Fringe checking procedure]] == Skype == * Start Skype and begin a chat session with all participating sites plus the on-call person === Send a Station Ready email === Lastly, **after submitting the checklist form**, send a ready message to IVS: * [[operations:documentation.station_ready_email|Sending a station ready email to IVS]] ==== Starting and Monitoring ==== * [[operations:starting_monitoring_9.10.5|Starting and monitoring the observations]] ==== After the Observations ==== At the end of an observation: * [[operations:documentation.ivs.finishing_v2|Finishing up]] * [[operations:documentation.ivs.newstart_v3_fs9.10.5|Following on with another experiment]] * [[operations:documentation.ivs.Logs|Processing logs and shipping data]] ===== eVLBI observations ===== * [[operations:documentation.eVLBI|eVLBI]] ===== Fringe Checking ===== * [[operations:documentation.fringe.schedfile|Schedule file preparation]] * [[operations:documentation.fringe.observing|Observing]] * [[operations:documentation.fringe.correlation|Correlation]] ====== Alarms and Errors documentation ====== New content! Please add! Here is an ever growing list of common errors and alarms that you may come across whilst observing. Some are diagnostics and others are plain alarms. If you come across more while observing please add them to this page and note the data and time of the occurrence and your fix. Or, if you couldn't resolve it leave it blank so observers know that it is a problem that was encountered and this would be a good time to call an on-call person [[operations:documentation.errors|List of alarms and fixes]]. ====== Observations on the 26m telescope from the Control Room ====== Note: these notes are based on the 26m setup notes from [[http://ra-wiki.phys.utas.edu.au/index.php?n=MtPleasant26m.IVSSetup|the ra-wiki]] and are still in a terse, draft form - Jim Lovell ==Before Setup:== **1. Before you drudg, start the correct fs-mk5hb (fs-auscope?) for IVS which runs a script to select the correct backend (mk5 and DBBC). Drudg schedule for the experiment (if it has not been done already)**. Drudg the schedule using ''slogit.auscope.sh'' on ops-serv2 as for the other stations. Having the backend selected correctly will result in the correct proc file being generated. From ops8, confirm that the sum file has been copied across to ''/vlbobs/ivs/sched/''. Otherwise, ssh into hobart (''ssh oper@hobart'') and copy the .sum file with the command scp /tmp/sched.tmp observer@ops-serv2:/vlbobs/ivs/sched/aug019ho.sum. __If the .sum file was not written:__ a) before the experiment started: drudge the schedule manually. On Hobart, go to the sched directory and type ''drudg r4705.skd''. Then type ''Ho'' for station Hobart and then ''3'' to create the prc file and ''5'' to write the sum file. The sum file is written in the ./tmp directory as sched.tmp. b) after the experiment has started: in order not to overwrite the prc file, copy the schedule file to the tmp directory and drudg it there. **2. Check to see if Jim P (or anyone else!) is observing. ** CHECK THE CALENDAR! Go to the [[http://ra-state.phys.utas.edu.au/|26m Live Page]] for the 26m and see if the telescope is tracking. Jim P puts a sticky note on the screen 4 of the ''newsmerd:1'' VNC session where the Field System is saying he is observing also. Please wait until the other person's experiment has finished until you set up. If it is urgent, please see the notes on his instructions page on how to cancel the Vela observations: http://ra-wiki.phys.utas.edu.au/index.php?n=MtPleasant26m.PulsarObserving DO NOT SHUT DOWN THE FIELD SYSTEM. The field system is used to monitor the S-X temperatures, and if you shut down the field system, it will stop the temperature monitoring which is essential for the observations ==Setup of Telescope:== **Step 1.** Open a VNC viewer to newsmerd vncviewer newsmerd:1 The password is the usual one. (hobart26 FS runs on computer named hobart). If this is not accessible ssh to observer@newsmerd then restart the vnc with vncserver :1 -shared & **Step 2.** Check that the field system is running on newsmerd. Usually, the Geodesy window will be Desktop 4. If it is not running, please start it with ''fs-mk5hb'' to ensure that the geodetic backend is selected. **Step 3.** Check receiver position IS THE S-X RECIEVER ON AXIS? MAKE SURE IT IS! Check the receiver position with the command ''rxp''. **It should be -256, +248, +18 (x,y,z)**. **Be sure the telescope is stowed BEFORE changing the reciever** To change it go to Desktop 3, bottom right window and enter '1' (OTTER interface). If the OTTER interface isn't open, then open a new terminal and type ''./OTTER_interface.sh''. Changing the receiver takes a couple of minutes (check on the live page http://ra-state.phys.utas.edu.au/ until the antenna stops moving - 5 sec update rate). Check that the receiver is actually moving, and wait until it reaches position before moving the telescope (can take a few minutes if moving from L-Band). **Stuck Receiver** If the receiver is stuck, first try a reset in the OTTER interface on the newsmerd vnc desktop 3 (type ''r'' and hit enter). Sometimes the OTTER interface crashes/fails to communicate with the reciever. If the GUI fails to start or returns error messages, please check that it is running (with ps -ef | grep OTTER as observer@rx26m). You can restart the server with sudo /etc/init.d/OTTER restart if there are any problems. The server is being continuous polled by MoniCA with the data being logged in the hobart database. More information on the OTTER server can be found here[[http://ra-wiki.phys.utas.edu.au/index.php?n=MtPleasant26m.OTTER|OTTER]] **Step 4. Check the procedure file** The procedure file will be in ''/usr2/proc'' and called ''ho.prc''. It should not need editing, and should be similar to recent .prc files using the same mode\ **Step 5.** Mark 5 setup: The 26m now uses uses ''mk5hb'' (''oper@mk5hb'') for recording. Log in to oper@mk5hb and confirm that DIMino is running. ps auxww | grep -i DIM If DIMino is running, it will return an output like this: root 10832 5.2 0.5 40564 12984 ? Rl May08 2666:16 DIMino -m0 oper 32071 0.0 0.0 2852 544 pts/2 D+ 04:14 0:00 grep -i DIM If the first line in this output does not appear, DIMino is not running. Run it with DIMino -m0 & Check the VSN of the module using tstDIMino vsn? Check if there is any data on the module by using the command DirList. If there is (and there is supposed to be data on there), note the end pointer (as in how much data has been recorded so far on that module) in the [[handover|Handover Notes]] NOTE: If there is no free space on a module in the Mark 5, call your on-call person for assistance in changing modules. **Step 6.** Start eRemoteCtrl. It is availalbe from the drop down menus as it is for the 12m telescopes. **Step 7.** Start the Log Monitor as for the 12m telescopes (from the drop down menu). Start a new log file and begin recording to it. **Step 8.** Load the experiment procedure file (if you haven't already) proc=rd1234ho note the 'ho' on the end for Hobart 26m. **Step 9.** Run the setup of the experiment using setupsx - Run the setup. The command for an R1 experiment is ''setupsx'' but look in the experiment procedure library printout to find the correct setup procedure name. Watch for warnings. - Check all the values reported by ''iread'' and ''bread'' are nominal. - Check times are ok with ''fmset'' on oper@hobart. - Set the cal to be controlled by PCFS. observer@newsmerd: calu -m pcfs -r rakbus - Check that ''caltsys'' works. Caltsys should return results between 50 and 70K - Check that weather works. ''wth'' - Check that fmout-gps works. ''clkoff''. Note that the readings are collected from pcfshb at this stage. If clkoff returns an error, or fixed readings, check that the FS on pcfshb is ruuning & restart it if necessary. - Check that the Mark5 has a blank module big enough for the experiment. Compare the Drudg summary of the schedule with the label on the front of the disk module. - Run ''disk_pos'' to check the disk module is empty. Result should be 0/0 otherwise the disk module contains data. - Record some data. ''disk_record=on'' - Observe green disk activity lights on module - Stop recording. ''disk_record=off'' - Check the recording. with ''checkmk5'' - Observe the time and details in the log window for the test recording, and have a look at the autocorrelation spectrum and bit statistics screens. - Start the schedule. ''schedule=r1186ho,#1'' - Watch PCFS log for errors. - Check the antenna is on source. ''onsource'' * [[software:monica_26m_cryo|open the monica client to monitor the He pressure]] Lastly, [[operations:startmessage|send the start message]]. \\ Once the schedule starts, watch the log and everything closely for the first scan or so. ==== Manual Re-starting the drives on the 26-m ==== Sometimes the drives need to be manually re-started. Documentation on how to do this can be found [[operations:26mdrives|here]]. ==== Monitoring Observations on the 26-m ==== * Please note that e-RemoteControl is now longer supported and will not connect to the field system! * Instead, please use ''stream_log ho'' to create a local file for the log monitor. * Checklists now have to be manually triggered with ''checklistHo $exper'' (where ''$exper'' is the experiment name) - to launch the start or end checklists, please use ''checklistHo $exper start'' or ''checklistHo $exper stop'' repectively. * Parameters of the check-list during observation for 26-m are [[operations:documentation.eremotecontrol_checklist_midob26|explained here]]. ==== After 26m Observations ==== DO NOT SHUT DOWN THE FIELD SYSTEM. The field system is used to monitor the S-X temperatures, and if you shut down the field system, it will stop the temperature monitoring which is essential for the observations Send the [[operations:documentation.ivs.commentsfile|end message]] Park the dish with source=stow in the Field System ==== Monitoring Observations on Parkes 64-m ==== [[operations:documentation.parkes_monitoring|Setting up and monitoring Parkes instructions.]] ====== User accounts and email addresses ====== Gmail accounts have been set up for registration with skype and other services. The usual Observer password applies. ===== Katherine ===== gmail: keobserver@gmail.com Skype: keobserver ===== Yarragadee ===== gmail: ygobserver@gmail.com skype: ygobserver ===== Hobart 12m ===== gmail: hbobserver@gmail.com skype: hbobserver