User Tools

Site Tools


operations:documentation.errors

This wiki is not maintained! Do not use this when setting up AuScope experiments!

This is an old revision of the document!


Alarm and error documentation page

This page is dedicated to presenting new and current observers with the myriad (or miriad ;) ) of problems that the observer may face. And more importantly, the successful solutions to them.

If you come across an error whilst observing, no matter how small, please add:

  • The alarm (without data/time) that came up and alerted you to the problem or (if there was no alarm);
  • The diagnostic that presented the problem to you;

A real-life example:

Auto-stow sequencing (no alarm) | ALARM: error fl -1 previous source in this schedule not reached before new source was commanded

I noticed the wind speed had dropped to 7.5km/h but the antenna hadn't recovered. I halted the schedule and restarted the drives. This did not fix the problem however alarms sounded every scan 'ALARM: error fl -1 previous source in this schedule not reached before new source was commanded' and started trying to record data whilst stowed. Solution was to:

      halt
      disk-record=off
      antenna=off
      terminate fs
      restart
      antenna=open
      antenna=operate
      proc=r4774hb
      schedule=r4774hb
      

Final note: logs noted a 'Auto-stow release failed' before this when all other wind stows ended with 'Auto-stow released succeeded'

Antenna stuck (no alarm) | ALARM: error fl -1 previous source in this schedule not reached before new source was commanded

  Power outage, antenna got stuck after the system recovered:
      halt
      disk_record=off
      antenna=off
      terminate
      in timeke, HMI, 'reset drives', wait until both comms stati are green
      fs
      proc=mv004ke
      setup01
      antenna=open
      antenna=operate
      source=stow

And it moved. Then I restarted the schedule. Generator status was 'in auto-off' so I assumed the power came on without any problems.

Antenna stuck | Warning: error st -27 computer acu time difference exceeded 0.25 seconds, see value above.

Normally this error can be ignored, but there was an additional error with this - WARNING: error st -10 rampepochtime not -1 when preparing to load, check antenna is in “operate”. Error presented as drive status not ready, with drives green in HMI. HMI was unresponsive - could not turn drives off. Antenna=operate and Antenna=open both return Antenna OK, but don't work. The clock on the right of HMI interface on cnsshb (CURRENT TIME UT1) was not updating.

Solution to issue is using HMI and selecting the reboot central button under resets. This reset the clock connection to the antenna, allowing the drives to communicate again.

rfpcn errors (time-outs) etc.

ALARM: error s5 -104 rfpcn: time-out, connection closed

If you receive a persistent “rfpcn: error opening, rfpic probably not running, see above for error” report, or notice that the recording is notably behind the summary file and the becklog grows, you might want to restart Rxmon.

Log in to pcfs as root and perform the following command:

pcfsyg:~#  su
pcfsyg:~#  /etc/init.d/Monica.Rxmon stop
pcfsyg:~#  ps -ef | grep Rxmon
pcfsyg:~#  /etc/init.d/Monica.Rxmon start

If the command worked, you will see the parameters listing.

The “ /etc/init.d/Monica.Rxmon start” command may not work as “ERROR on binding: Address already in use”. Just wait a minute and repeat an attempt. If it still doesn't work restart Monica:

pcfsyg:~#  /etc/init.d/Monica.monica stop
pcfsyg:~#  /etc/init.d/Monica.monica start

ALARM: Large difference between formatter and maser delays. Check for stability of new offset

First scan of the schedule this came up. The telescope checks the maserdelay and clkoff before every scan and if it is >0.5us then this alarm comes up. I immediately checked the maser first with

maserke

In terminal and it was all green. After that I went to “ke” (pcfske) and checked

fmset

And saw there was a large delay between the dbbc and pcfs. I halted the schedule:

halt, disk_record=off (in erememote)

and typed into fmset “s”, “y”, “s”, “y” to resync. After the resync it still wasn't the same so I opened a terminal and typed:

“vncviewer dbbcke”

In the dbbc, I closed the 'DBBC Control v104_2.exe' window and reopened the program from the desktop. It asked me to reconfigure and “y”. When it was finished I typed

“dbbc=pps_sync”

Into eremote and resynced in fmset again. After this it was stable so I restarted the schedule with

“cont”

/home/www/auscope/opswiki/data/attic/operations/documentation.errors.1488820938.txt.gz · Last modified: 2017/03/06 17:22 by Bryn Emptage