User Tools
Sidebar
This wiki is not maintained! Do not use this when setting up AuScope experiments!
Handover notes
Please use this page to write up any notes on anything that needs to be passed on to the next observer or should go into the end of experiment message.
Notes should be placed below the horizontal line as a new blog entry.
Continuing problems should be copied to the Current Issues page.
**Template for Handover notes:**
Be sure to add any specific notes about the experiment if required, for example “Special back end configuration for x reason”. Put as much detail as you can - the more the better.
====== Experiment Name ====== **Hobart 12m** Disk VSN: abc-123 | Data volume at beginning: 2.34 GB * 1300UT a comment from me (user) **Katherine 12m** Disk VSN: | Data volume at beginning: GB * **Yarragadee 12m** Disk VSN: | Data volume at beginning: GB * 1300UT Experiment started okay (JMc).
Format for entering a comment It is important to stick to this formula - there is a good reason we need this information, so please include it!
Time stamp in UT * Comment * (Your Name)
For Example:
20:00UT: Wind stows, Missed scans 012-3456 to 012-3457 (Jim)
20:50UT: Mark5 lost connection, had to restart Mark5 and reconfigure DBBC, called Jamie on call to fix, missed scans 012-3456 to 012-4567 (Jim)
Again - as much detail as you can. It is important to remember that these notes will be sent to the correlator, so please make sure they make sense to others.
DS288
General Notes
All telescopes give WARNING: error bo -212 can't change experiment procedure library when an experiment procedure is executing when starting up dynamic schedule
Still regular slewing during recording at all telescopes, only a couple of seconds at the beginning though. (Ellen)
Halting the schedule to fix problems appears to have no effect (at least on Hb, Ke), the schedule continues trying to record. (Ellen)
Continuing to get “ALARM: error m5 -900 not while recording or playing” errors frequently at Ke throughout my shift, these have been reported by Bryn and Ellen as occurring all day, they don't seem to correspond to any actual problem with the telescope. (Tiege)
NOTE: I have changed this alarm m5 -900 to BEEP instead of ALARM in /usr2/control.ke/alarm_actions.txt so that Lucas can get some sleep between checklists
Hobart 12m
Disk VSN: GSFC+011
Data volume at beginning = 1829.6 GB
- 23:00 UT started recording on hb, halt didn't stop the schedule, so there will be 15 mins of recordings on hb. (Bryn)
- 23:25 UT enabled dynamic schedule (Bryn)
- 23:46 UT First good scan 288-2347 (Bryn)
- 01:23 UT Wind stow, missed scans 289-0122 to 289-0128
- 02:05 UT Wind stow, missed scans 289-0205 to 289-0209
- 03:30 UT Wind stow, missed scans 289-0330 to 289-0335
- 03:50 UT Wind stow, missed scans 289-0350 to 289-0411
- 04:35 UT Wind stow, missed scans 289-0435 to 289-0623
- 05:01 UT elevation drives not released when wind stow released. Immediately went into wind stow again - will check elevation drives when next released from wind stow. (Bryn)
- 06:21 UT Wind stow released, drives working as normal. (Bryn)
- 06:26 UT Wind stow, missed scans 289-0626 to 289-0632
- 06:45 UT Wind stow, missed scans 289-0648 to 289-0650
- 07:19 UT Wind stow, missed scans 289-0718 to 289-0808
- 07:30 UT Wind stow, but telescope not slewing to stow position - problem with the drives? (Bryn)
- 07:52 restarted fs on Jamie's suggestion, telescope wouldn't release from stow. (Bryn)
- 16:19 UT Wind stow, missed scans 289-1614 to 289-1640 (Ross).
- 17:13 UT Wind stow, missed scans 289-1711 to 289-1718 (Ross).
- 05:22 UT - Clock jump, massive delay differences for 2 scans. Reporting no controller clock. Drives turned themselves off but the wind isn't very high and they didn't try to stow. Halting schedule to reset DBBC and restart drives. Also putting in new module, disk pos 11482 GB. (Ellen)
- 05:50 UT - Won't connect to the mark 5. Jamie is investigating. (Ellen)
- First scan back 290-0604, on new module HOB+0098. (Ellen)
- 07:30 UT - Lost connection to the antenna controller (I think), telescope was stuck and could not be fixed with the standard open or operate commands, needed to use the HMI interface on timehb and do a drive reset and operate toggle to regain connection. At the same time DBBC connection dropped out, had to reconfigure and resync the dbbc to fix it. Possibly another power glitch at Mt Pleasant has caused this? Everything seems fine now and the dynamic scheduler has been restarted. (Tiege)
- 08:30 UT - Everything back up and running, first good scan 290-0834 (Tiege)
Katherine
Disk VSN: HOB+0122
Data volume at beginning = 0 GB
- 23:25 UT enabled dynamic schedule (Bryn)
- 23:46 UT First good scan 288-2347 (Bryn)
- 01:11 UT slewing errors followed by a 2016.289.01:12:18 ALARM: error m5 -900 not while recording or playing
- 01:33 UT same errors as 01:11 UT, seems to be whenever schedule is updated (Bryn)
- ALARM: error m5 -900 not while recording or playing occurring frequently. UT 20:41 ; 22:13 ; 00:45 ; 03:33 (Ellen)
- 00:19 UT - FS crashed while recording, was picked up after 20 minutes from watchdog. Next good scan 290-0057. (Ellen)
- 05:38 UT - Antenna lost connection and no amount of resetting and rebooting has fixed it. Reporting that antenna is stuck. Jamie is investigating. (Ellen)
- First scan back 290-0547. But Rxmon not running until 290-0605. (Ellen)
- 13:10 UT - ALARM. field system crashed after 12:56 UT, last recorded scan 290-1256.
- Field system restarted and schedule restarted, first scan 290-1322 (Lucas)
Yarragadee
Disk VSN: HOB+0126
Data volume at beginning = 0 GB
- 23:25 UT Yg being taken out of schedule for radioastron in 30 mins (Bryn)
- 03:10 UT Yg finished radioastron, back on dynamic scheduling. First good scan 289-0320(Bryn)
r4761
Hobart
Disk: GSFC+011, Position: 1719.226 GBs.
- Experiment Started OK (Lucas).
Katherine
Disk: UVLBI-07, Position: 0.000 GBs.
- Experiment Started OK (Lucas).
Yarragadee
Disk: NRAO-179, Position: 0.000 GBs.
- Experiment Started OK (Lucas).
AOV012
Hobart 12m:
Disk VSN: HOB+0102, start 6819.09 GB
- 1730UT Experiment started okay (JS)
- 1945UT SystemClock1 errors. Attempting to diagnose (JS)
- 1950UT Looks like the generator has died at hb, Brett is going out to refuel (JS)
- Affected scans: 286-1941 to 286-2307a. ie first good scan was 286-2311
- 2311UT Experiment restarted (Liz)
2016.287.03:28:26 Scan name = 287-0027 2016.287.03:28:26 ALARM: error m5 -900 already off
2016.287.03:28:26 WARNING: error m5 -906 mark5 return code 6: inconsistent or conflicting request 2016.287.03:28:26 Total data recorded = -7041.000 GB 2016.287.03:28:26 Scan name = 287-0032 2016.287.03:28:26 ALARM: error m5 -900 no scans
2016.287.03:28:26 WARNING: error m5 -906 mark5 return code 6: inconsistent or conflicting request
- 11:40 UT Autocorrelations look a bit askew in sub-bands 7 and 9. I'll monitor them over the next few scans and configure the dbbc if they persist (Lucas)
Katherine 12m:
Disk VSN: HART+104, start 7040.11 GB
Disk VSN: HOB+0015, start 1533.39 GB
- 1730UT Experiment started okay (JS)
- 2117UT Katherine switched back to bank A (HOB+0015) since bank B was at capacity.
- 0030UT Katherine switched back to bank B (XAO-1013) for bank A (HOB+0015) to be able to be shipped etc for r1761. (Liz)
- Missed scans 287-0027 and 287-0029 (Liz)
- 0328UT alarm sounded ALARM: error m5 -900 already off, WARNING: error m5 -906 mark5 return code 6: inconsistent or conflicting request, ALARM: error m5 -900 no scans, and then fixed itself. On closer inspection, log monitor seems to be printing information about scans from the last 2 hours in quick succession over 0321-0331UT. All outputs before and after this bracket match the schedule file. Seems to be fine now, and eremote was unaffected so appears to be a strange log monitor artefact rather than anything wrong with the experiment. Will monitor for now. (Liz)
- 0509UT Happened again. Alarm at 0457UT and 0503UT. Spoke to Jamie. Log file was reading from a tmp file which he thinks was probably the cause. Made new log file on /vlbobs/~ seems to be something was26 GB
2016.287.06:05:10 Scan name = 287-0605b interfering with the tmp file and log monitor was having to rewrite experiment history (Liz).
- 11:40 UT Autocorrelations look a bit askew in sub-bands 1, 3 and 8. I'll monitor them over the next few scans and configure the dbbc if they persist (Lucas)
yarragadee 12m:
Disk VSN: HOB+0119, start 9513.18 GB
- 1720UT No response from MK5 needs to be rebooted, can't get trhough to yg telephone!! (JS)
- 1740UT reset mk5 via ip switch (JS)
- 1752UT Onsource for 286-1752a (JS)
- missed scans: 286-1730 to 286-1745
- 2345UT For some time yg has had formatter to fs time difference 0.5s or greater as mk5=dot? response nearly 1 sec out. According to Jamie fix with fmset after this experiment end. (Liz)
r1761
Hobart 12m:
Disk VSN: GSFC+011
- 17:00UT Experiment started okay (Liz)
- 18:42UT computer acu time difference exceeded 0.25 seconds, for about 2 mins. From previous notes, assume network issue. Fixed itself. (Liz)
- 06:00UT windstow, missed scans 286-0558a through 286-0606 (Ross)
- 07:50UT band of 11 of autocorrelation spectra looks weird (Ross)
- 11:10UT antenna control issue, fixed by rebooting control system (Ross)
- 1237UT DBBC reset to fix clkoff issues no scans interrupted (JS)
Katherine 12m:
Disk VSN: HOB+0015
- 17:00UT Experiment started okay (Liz)
- 07:50UT band of 11 of autocorrelation spectra looks weird (Ross)
yarragadee 12m:
Disk VSN: USN-0017
- 17:00UT Experiment started okay (Liz)
aua013
Hobart 12m:
Disk VSN: HOB+0102
Data volume at beginning: 1225.5 gb
- 1915UT during setsup ifc power levels were below target, 27K when expecting 32k.
- 1930UT Experiment started OK. (AK)
- 0056UT Wind stow, missed scans 283-0055 to 283-0110a (Bryn)
- 0443UT Wind stow, missed scans 283-0441 to 283-0456 (Bryn)
- 0504UT Wind stow, missed scans 283-0503 to 283-0517
- 0523UT Wind stow, missed scans 283-0523 to 283-0529
- 0557UT Wind stow, missed scans 283-0558 onwards. (Bryn)
- 0817UT, antenna still wind stowed from stow previous (Lucas).
- Antenna unstowed at 0826UT, first scan 283-0824 (partial) (Lucas).
- Kept tracking 283-0824, had to re-start schedule. Missed all scans though to 283-0905 on-inclusive (Lucas)
- 10:53:01 UT, scan 283-1053 wind stow.
- 10:58:53 UT, auto-stow released. First scan 283-1058.
- 10:59:56 UT, scan 283-1058 wind stow.
- 11:05:06 UT, auto-stow released, first scan 283-1108.
- 11:20:00 UT Temporarily got a GPIB time-out when maserdelay called, set off alarm for Yg and Hb. Counters were fine and within 0.3 us difference. Monitoring. (Lucas)
- 11:54 UT - Wind stow, during scan 283-1153. (Tiege)
- 12:00 UT - Auto-stow released, first good scan 283-1203 (Tiege)
Katherine 12m:
Disk VSN: HART+104
Data volume at beginning: 1067.0 gb
- 1930UT Experiment started OK. (AK)
Yarragadee 12m:
Disk VSN: HOB+0119
Data volume at beginning: 3744.30 gb
- 1930UT Experiment started OK. (AK)
- 2242UT Disk pos lagging by 18 GB (Bryn)
- 11:20:00 UT Temporarily got a GPIB time-out when maserdelay called, set off alarm for Yg and Hb. Counters were fine and within 0.3 us difference. Monitoring. (Lucas)
aug030
Hobart 26m:
Disk VSN: ?
Data volume at beginning = ~1000GB
- 21:47UT Jamie started the experiment after getting correct schedule file (Bryn)
- 08:22 UT - ALL SCANS BEFORE THIS HAVE ERRONEOUS SYSTEM TEMPERATURES. THE CAL WAS NOT FIRING AT ALL PREVIOUS TO THIS. (Ellen)
Hobart 12m:
Disk VSN: HOB+0072
Data volume at beginning: ~5220 gb
- 21:35UT Jamie started the experiment after redistributing correct schedule file (Bryn)
- 09:00 UT - Disc pos has fallen behind by 25 GB in last 1.5 hours. (Ellen)
Katherine 12m:
Disk VSN: MED-0038
Data volume at beginning: 0 gb
- 21:35UT Jamie started the experiment after redistributing correct schedule file (Bryn)
- 09:05 UT - Disc pos has fallen behind by 10 GB in last 1.5 hours. (Ellen)
Yarragadee 12m:
Disk VSN:HOB+0119
Data volume at beginning: 0 gb
- 21:35UT Jamie started the experiment after redistributing correct schedule file (Bryn)
- 22:17UT missed scan 281-2219 - Mark 5 error -900 can't read while data transfer in process (Bryn)
- 07:30 UT - Disc pos is ~40 GB behind. No record has been kept in the logs of when this started happening. (Ellen)
- 09:05 UT - Disc pos has fallen behind by 10 GB in last 1.5 hours. (Ellen)
