User Tools
Sidebar
This wiki is not maintained! Do not use this when setting up AuScope experiments!
Handover notes
Please use this page to write up any notes on anything that needs to be passed on to the next observer or should go into the end of experiment message.
Notes should be placed below the horizontal line as a new blog entry.
Continuing problems should be copied to the Current Issues page.
**Template for Handover notes:**
Be sure to add any specific notes about the experiment if required, for example “Special back end configuration for x reason”. Put as much detail as you can - the more the better.
====== Experiment Name ====== **Hobart 12m** Disk VSN: abc-123 | Data volume at beginning: 2.34 GB * 1300UT a comment from me (user) **Katherine 12m** Disk VSN: | Data volume at beginning: GB * **Yarragadee 12m** Disk VSN: | Data volume at beginning: GB * 1300UT Experiment started okay (JMc).
Format for entering a comment It is important to stick to this formula - there is a good reason we need this information, so please include it!
Time stamp in UT * Comment * (Your Name)
For Example:
20:00UT: Wind stows, Missed scans 012-3456 to 012-3457 (Jim)
20:50UT: Mark5 lost connection, had to restart Mark5 and reconfigure DBBC, called Jamie on call to fix, missed scans 012-3456 to 012-4567 (Jim)
Again - as much detail as you can. It is important to remember that these notes will be sent to the correlator, so please make sure they make sense to others.
R1763
Hobart
Module: HART+106, start pos: 5018.461 GB
- 17:00 UT - Experiment started OK. (Tiege)
Katherine
Module: HOB+0083, start pos 5322.734 GB
- 17:00 UT - Experiment started OK. (Tiege)
Yarragadee
Module: HOB+1003, start pos: 4436.751 GB
- 17:00 UT - Experiment started OK. (Tiege)
AUG031
Important pre-notes
Experiment AUG031 isn't the first one of this name this year. The drudge (via slogit) initially got the OLD version from day 094 instead of 297. So make sure to check the UT day in the sum file. I think Jamie fixed it by manually re-drudging? (please edit this Jamie)
“ ERROR ch -4 Device v3 timed-out on response from MATCN ” means that something is wrong with the mk4 VCs and the mk4 rack (white one with 'VC' lights) needs to be restarted manually at Mt P. This error appeared after setupsx when changing the 26m back to the mk4 from the mk5/dbbc. The rack can be reset by turning off, waiting then turning on again and waiting (red lights everywhere until you setupsx again). The on/off switch is at the top of the rack. If this doesn't fix the matcn error then you needs to call Brett.
Hobart 26m
Disk module HOB+1000, initially recorded 4302.249 GBs.
- Experiment started ok (Lucas).
- Note that time-out errors appearing in log are ignorable/benign (Jamie/Lucas)
- Experiment interrupted for RadioAstron 22:40/297 (Warren)
- Experiment restarted 00:14/298 (Warren)
Hobart 12m
Disk module HART+106, 0.705GBS initially recorded.
- Experiment started ok. (Lucas)
Katherine
Disk module HOB+0083, 3.205 GBs initially recorded.
- Experiment started ok. (Lucas)
- fs locked up near 23:15/297, back on source at 00:12/298 after reboots (Warren)
Yarragadee
Disk module HOB+1003, 0.769 initially recorded.
- Experiment started ok. (Lucas)
- fs locked up near 23:00/297, back on source at 00:12/298 after reboots (Warren)
HOB006
When the hydraulics at Ho failed the following error was in the log monitor but was not setting off an alarm so Lucas wasn't woken up for it. It is not listed in the alarm_actions file
WARNING: error fl -1 previous source in this schedule not reached before new source was commanded
Hobart 26m
disk USN-0085, total data pre-recorded 1115.510 GBs
- Experiment started late, GPIB device continually timed out whilst many errors ensued with control and equip files (Lucas).
- Turns out experiment was being incorrectly drudged due to unique setup (mk5/dbbc). Jamie manually re-drudged and setupxx worked correctly. First good scan 295-2040 (Lucas).
- Still getting delay difference changing due to GPIB device timeout.
- Update, as of 295-2052 after repeated “counter” commands it has fixed itself. Monitoring. (Lucas).
- I think there is a hydraulics fault. The Live page keeps saying to and keep getting command 'previous source in sched not reached before new source was commanded' (Lucas).
- Hydraulics fault was caused by circuit break as the motor wasn't covered and water got in. Collaborative effort of Eric and Brett solved the problem. (Lucas).
- GPIB device continues to time-out, however delay difference is still stable and < 0.3. Counters are being monitored on rack camera to make sure that severe drifts don't occur whilst Eric investigates. (Lucas).
- 23:50 UT - schedule resumed, lost about 500 GB. First scan 295-2351. GPIB still a problem though, reporting massive delay differences every scan. Clkoff values looks reasonable. (Ellen)
- 01:04 UT - Eric has reset the GPIB device and it is still a problem. (Ellen)
- 01:11 UT - Tsys values all overflowing again, restarted CAL again and everything is fine. The CAL has dropped out like this halfway through an experiment before. (Ellen)
- 01:22 UT - Jamie has turned the GPIB off so that we are no longer measuring a delay difference because it's wrong anyway. Keeping an eye on dbbc=pps_delay instead. (Ellen)
- 02:17 UT - CAL needed resetting again. And several other times. (Ellen)
- 07:15 UT - CAL reset again. (Ross)
Hobart 12m
disk HOB+0098, total data pre-recorded 10024.782GBs
- Experiment Started OK (Lucas).
- Schedule halted whilst 26m was down. All scans missed between 295-1911 and 295-2016 - ~125 GB missing (Lucas).
R4762
Hobart
Module: HOB+0098, start pos: 8114.178 GB
- 18:30 UT - Antenna stuck, experiment will start slightly late. (Tiege)
- 18:38 UT - Experiment started, however wind stowed immediately. (Tiege)
- 18:42 UT - Unstowed, First valid scan for the experiment is 294-1846c (Tiege)
Katherine
PLEASE NOTE: LBA experiment was recorded onto the wrong disk according to Martin (he phoned Bryn earlier today), which mean there was not enough room for the R4 on FIG-0004 (apparently the intended disk for this experiment). Therefore, the R4 has had to go on USN-0067 which contains ~890 GB of LBA data.
Module: USN-0067, start pos 888.075 GB
- 18:30 UT - Experiment started OK. (Tiege)
- 20:34 UT - “ALARM: error st -999 tcp/ip connection error”, fixed within 15 seconds using antenna=open and antenna=operate, happened between scans so no scans were affected. (Tiege)
- 20:37 UT - Straight after fixing the above error I got “ALARM: error s5 -104 rfpcn: time-out, connection closed”, have restarted Rxmon to fix. (Tiege)
- 20:48 UT - 294-2049 will be first good scan after the errors were fixed. Scans affected by both errors are 294-2037 to 294-2045a inclusive. (Tiege)
- 20:57 UT - Received “ALARM: error st -999 tcp/ip connection error” again, used the same fix as previously, no scans affected. (Tiege)
- 21:08 UT - “ALARM: error s5 -104 rfpcn: time-out, connection closed” error again, this time I restarted Monica, scan affected is 294-2107. (Tiege)
- 21:14 UT - “ALARM: error s5 -104 rfpcn: time-out, connection closed” error again, tried restarting Rxmon again this time, scan affected is 294-2114a. Restarting of Rxmon and/or Monica are the solutions listed in the wiki, however, they haven't seemed to help much here. It seems when these errors do crop up in observations, they tend to reoccur. Jim will be taking over soon and I will mention the problem to him so he can check it out and hopefully determine whether the problem is something more sinister. (Tiege)
- 06:28 UT - “ALARM: error s5 -104 rfpcn: time-out, connection closed” error again, with Jesse's help fixed by halting the schedule and stopping then restarting DIM which seemed to be running but wasn't really. Missed scans … (Liz)
Yarragadee
Module: USN+0014, start pos 1185.734 GB end pos 1420.317 GB: TOTAL: 234.583 GB
Module: HAY-0062, start pos 0 GB Changed at 21:30 UT, first good scan 294-2145
- 18:30 UT - Experiment started OK. (Tiege)
- 20:49 UT - Restarted Rxmon to fix “error s5 -23 rfpcn: error opening, rfpic probably not running”, seems to be fine. (Tiege)
- 21:30 UT - Randall from Yg just called up, I had forgotten to change the module for Yg after the LBA experiments, I have now switched to the other disk (HAY-0062). This means ~234 GB of R4762 data is on USN+0014 (from disk_pos 1185.734 GB through to 1420.317 GB). First good scan on the new module will be 294-2145. (Tiege)
- System monitor for Yg was not running because one of the Monica servers (antmon) had crashed
t2113
Hobart 12m:
Disk VSN: HOB+0098 (7099.712 GB)
- 17:30 UT - Experiment started okay (Ross)
Katherine 12m:
Disk VSN: HOB+0122 (14343.809 GB) (Part 1/2, together with r1762)
USN-0067 (0.0 GB) (Part 2/2)
* 17:30 UT - Experiment started okay (Ross)
* 2325 UT - Experiment halted to swap banks to allow R1 shipment. New module is USN-0067/4000/1024 (Initial disk_pos 0.0) (JMc)
* 2337 UT - Experiment resumed, first good data. All data between 2325 and 2336 unusable. (JMc)
* 0050 UT - MET sensor disconnected - GA upgrading GNSS receiver & new interface is required. (JMc)
Yarragadee 12m:
Disk VSN: BKG-0015 (0.000 GB)
- 17:30 UT - Experiment started okay (Ross)
R1762
Fringes found at channel 08 in hb-ke baseline, 08, 13 baseline hb-yg and none at ke-yg.
The watchdog timeouts on the log monitors no longer trigger an alarm. The message appears in the log monitor dialogue but doesn't do anything else (not even send an sms). (seen in both Hb and Yg). Addition, no alarm sets off a tone, even when “acknowledge alarm” becomes an option.
Hobart
Module: HOB+0098, start pos: 3996.785 GB
iread power levels ifc and ifd are oscillating even though the attenuation isn't maxed. Jamie is investigating, although possibly just interference that can't be controlled.
Power dropping out a lot. Antenna needs re-connecting each time.
- 17:00 UT - Clock drifted for second time, missed scans 291-1702 and 1705 to reconfig DBBC. (Ellen)
- 19:50 UT - Clock drifted from about -2.4 to -3.2 us. Rebooted DBBC, no scans missed. (Ellen)
- 12:15 UT - S RCP attenuator incorrectly set, presumably for whole experiment. HAS NOW BEEN SET TO 0dB. (Ross)
Katherine
Module: HOB+0122, start pos 11405.548 GB
- 17:00 UT - Experiment started ok. (Ellen)
Yarragadee
Module: HOB+0126, start pos 9487.412 GB
Something happened to the fishlens camera between 8pm and 1am when I came back in - it's getting a very bad signal.
- 17:00 UT - Experiment started ok. (Ellen)
