The AUM experiments are designed to investigate and hopefully demonstrate the compatibility of the new wideband VGOS receiver with the existing S/X system. This is important for maintaining a global geodetic network during the transition to a fully global VGOS array.

The Hobart12 and Katherine antennas now have a functional wideband system, with reasonable sensitivity. As of January 2020, we are still observing using the same mode as in the AUM series. This uses the “VGOS”-mode DBBC3 software, recording channels with an (excessive) bandwidth of 32 MHz which will be dealt with in the correlation stage. Once the new DBBC3 firmware is confirmed as usable for our systems, we will move across to using that. As such, these instructions may be updated - please check the page carefully each time you are starting or monitoring an AUA/AUM experiment.

The major differences for observing an AUM experiment rather than a standard S/X experiment are that there is limited/no FS support for the hardware and that we need to run two DBBCs simultaneously to record the S- and X-band data. The recorder has also changed from the mark5B to a flexbuff which operates in a very different fashion. Preliminary scripts have been written to monitor the relevant parameters but again, these are likely to be changed and improved as needed.

=== Before the Experiment === [Updated by Lim 2021March04]

• Preparing the schedule file: Do not use the existing slogit scripts for Hb and Ke, but instead download the skd file directly to pcfs[hb|ke|yg] and run drudg to generate the snp and sum files and put them where they're needed.

cd /usr2/sched
wget ftp://ivs.bkg.bund.de/pub/vlbi/ivsdata/aux/2021/aum028/aum028.skd
drudg aum028.skd
   hb (station)
   11 (show/set equipment type)
   1 16 1 1  (none, flexbuff[mark5b], none, none)
   3  (make snap file)
   5  (print summary)
   0  (exit)
scp /tmp/sched.tmp observer@ops-serv2:/vlbobs/ivs/sched/aum028hb.sum (tmp summary file might have a different file name! i.e. DR.tmp)

The next step is to create the procedure file with the appropriate commands. The easiest way is to copy the procedure file from a previous experiment using the same mode. For the 32-MHz mode (In use as of March 2020), the SI experiment setup can be used as the template. e.g:

cd /usr2/proc/
cp si.prc aum018hb.prc  (Hb, Ke)

because Yg now use mark5B instead of flexbuff, use slogit to drudg as usual.

• Terminate the FS and restart it with “fs-dummy”. (To ensure that the Field system will not connect to the DBBC2 and block the configuration scripts)
• Log in to DBBC3 (vncviewer dbbc3hb) and make sure that the DBBC3 Control DDC_V_v124.exe program is running, and that the final line of output reads Waiting for connection).
• Next, check the DBBC2 (vncviewer dbbcho for Hb and vncviewer dbbcke for Ke) and check that the correct control program is running (DBBC2 Control DDC v105E_2.exe for Hb, DBBC2 Control DDV v106E_120118.exe for Ke) and displays Waiting for connection)
• If the server program is not running, please start it from the link on the desktop (and be prepared for a 15-20 minute wait for the DBBC3 after answering “y” to the reconfiguration question).
• If it does not read waiting for connection, it should give the IP address of the machine that's currently talking to it (e.g if it reads Command from 131.217.63.201: then the connection is from 131.217.63.201). You can lookup up the name of this machine using nslookup 131.217.63.201 - in this example it is the hobart machine that is talking with the DBBC. If a PCFS machine is connected, the best option is to terminate the FS on that machine and restart with a version that will not conflict. For the Hobart example, running fs-auscope will avoid this problem while fs-flexbuff will block communications.
• From a terminal on the pcfs machine, run the command ./bin/aum.query.summ.py. If the script stalls (and does not return to the command prompt), then it's likely that either one or both of the DBBC control programs are not running, or that something else is talking to them.
• Once you can run ~/bin/aum.query.summ.py successfully, you should be fine to configure the backend at Hobart 12 with ./bin/aum.setup.lim.py [exp_name]. NB - this script will force a synchronisation of the BBCs and is equivalent to running setupsx and fmset in sequence. Do not run the setup script during the experiment!
• As of March 2021, only dbbc3ke is available at Katherine, please setup using ./bin/aum.setup.dbbc3only.py instead.
• Load the procedures with proc=aum018hb and configure the recorder with setupvgos
• Check the ~/bin/aum.query.py returns “pps_delay/ [1]: 43 ns, [2] 39 ns, [3] 39 ns, [4] 43 ns, [5] 43 ns, [6] 43 ns, [7] 0 ns, [8] 0 ns”. (If not, the dbbc3 may need to be reconfigured or even restarted)
• Check that ~/bin/aum.query.summ.py returns “Delays: OK” and “Samplers: Ok” (if the IF or BBC power levels are reported as bad, this should be re-checked after a few minutes but they might be out of range when freshly set up).
• You can make a test recording to check for synchrnoisation problems between the different DBBCs with ~/bin/sxtimetest.sh which will print out the decoded time of the start of the test recording. All three values should agree and match the time reported in the FS log for when the “record=on” command was issued. If the times differ between the different datastreams, you will need to reconfigure the relevant DBBCs. If they are all in agreement but differ from the FS log, please note this difference in the log but proceed on to the next step.
• Set the log file for the experiment with log=aum018hb
• Load the schedule file with schedule=aum018hb,#1

• Yarragadee should be set up normally, using the DBBC2 (v104_2.exe) and mk5yg - Please make sure this is configured by starting the Field System with fs.

A new script has been created which summarises the DBBC3/DBBC2 status. A version is installed on the pcfs machines as ~oper/bin/aum.query.summ.py and should be running in the VNC session. The Hobart and Katherine menus on ops8 also have a new entry called AUM Monitor which will start a version on ops, running in an xterm and refreshing every 60s. The output is below:

2020.056.05:00:06
##################
DBBC3 - Delays: OK
DBBC2 - Delays: OK
##################
DBBC3 - Sampler 3: OK
DBBC3 - Sampler 4: OK
#####################

DBBC3 IFC: OK
DBBC3 IFD: OK
DBBC3	Freq	BW	Power	DBBC3	Freq	BW	Power
#############################################################
BBC17	2204.99	32	OK	BBC25	2204.99	32	OK
BBC18	2244.99	32	OK	BBC26	2244.99	32	OK
BBC19	2344.99	32	OK	BBC27	2344.99	32	OK
BBC20	2504.99	32	OK	BBC28	2504.99	32	OK
BBC21	2724.99	32	OK	BBC29	2724.99	32	OK
BBC22	2844.99	32	OK	BBC30	2844.99	32	OK
BBC23	2884.99	32	OK	BBC31	2884.99	32	OK
BBC24	2924.99	32	OK	BBC32	2924.99	32	OK

DBBC2 IFA: Bad
DBBC2 IFB: Bad
DBBC2	Freq	BW	Power	DBBC2	Freq	BW	Power
#############################################################
BBC01	300.99	32	OK	BBC09	300.99	32	OK
BBC03	332.99	32	OK	BBC11	332.99	32	OK
BBC05	364.99	32	OK	BBC13	364.99	32	OK
BBC07	348.99	32	OK	BBC15	348.99	32	OK

When carrying out a check, please run this script and check the outputs. Any errors related to the delays or DBBC3 samplers are critical are should be followed up (phone Jamie). Errors related to the IF or BBC levels should be noted - if these are consistently bad (in three consecutive readings), please contact Jamie to follow this up.

Taking the existing e-remote control checklists as a template, here is a listing of the significant differences. Most of the information is gathered by the ~/oper/bin/aum.query.py script. This can be called from the FS with sy=bin/aum.query.py with the outputs printed to screen and also logged to a file called e.g: /usr2/log/aua060hb.dbbc.log. It's strongly recommended to have a terminal watching this (tail -100f /usr2/log/aua060hb.dbbc.log) as it can be difficult to read from the FS display. An example of the output is given at the end of this page.

* RF and IF paths configured/DBBC settings: The RF/IF/DBBC configuration has been compiled into a single script on the pcfs machine called ~/aum.setup.py. Running this should set up all the essential settings for both the X-band recorded through the DBBC3 and S-band through the DBBC2 but the script should not be run during an experiment to avoid unnecessary clock breaks.

• Check that the dbbcif[abcd] input, filter and power target level readings in the aua060hb.dbbc.log are correct. For Hobart12, these should be

  dbbcifa=4,**11**,agc,2,**47680**, 48000
  dbbcifb=4,**27**,agc,2,**47843**,48000
  dbbcifc=2,32,agc,2,**40073**,40000
  dbbcifd=2,23,agc,2,**40070**,40000

and for Katherine:

dbbcifa=2,**38**,agc,2,**42008**,42000
dbbcifb=2,**37**,agc,2,**41669**,42000
dbbcifc=2,**29**,agc,2,**48178**,48000
dbbcifd=2,**23**,agc,2,**47928**,48000

NB - values with asterisks may vary depending on the input power level. Just make sure that the IF power readings is within ~1000 counts of the target.

• DBBC server: If the aum.query.py script cannot communicate with the DBBC servers, you can restart the software running through the VNC sessions as described in the setup procedure. Again, please note that this will introduce a clock break so make sure that there's no other issue like another machine having connected to the DBBC server.
• Maser ok: Check the Hobart26 maser as per usual
• Mark5: Not used. Instead log in to the Flexbuff recorder (with ssh observer@flexbuffhb) and confirm that the time is correct (with ntpq -np) and that jive5ab is running (with ps -ef | grep jive)
• The clock offsets can be (partially) read through the FS. maserdelay reads the maser-GPS difference as usual while dbbc3delay logs the DBBC3-GPS delay as gps-fmout for Katherine. dbbc2delay (at Katherine) will log the DBBC2-GPS difference. For Hobart12, the DBBC3-maser difference is being printed in a terminal on in the pcfshb VNC session - this should be ~0.1 microseconds. If the display is not present, or not updating try running cat /dev/ttyS0 as root on flexbuffhb.

• Test Recordings: To make a test recording, simply run disk_record=on, disk_record=off and checkmk5 in the operator input as per usual.
• Check for any integer second offsets in the recorded data by running ./bin/sxtimetest.sh on pcfshb. This will make a short test recording and then present the decoded time of the three files - all three entries should agree to the integer second.
• No autocorrelation script has been implemented yet.
• There is no systemp measurewment implemented yet.
• There is no pointing/SEFD check implemented as yet
• No fringe check implemented as yet.

• Antenna checks as per usual
• Check data is being recorded in three streams (mk5=datastream? 0:3:sxy:xx:xy) and 3 files the same size per scan on flexbuff<st> vbs_ls -lh aum019_* | tail
• LO check unavailable
• No autocorrelations
• Check the delays with maserdelay and dbbc3delay (Ke only).
• Check the Hb and Ke masers as usual.
• Check the weather data is being logged & add in sky conditions as usual.
• No Tsys checks.

Most of the additional monitoring of the DBBC3 is currently logged to a separate log file (/usr2/log/aua060hb.dbbc.log), updated every scan when the sy=bin/aum.query.py | tee … script is run. This is part of the midob procedure. I've annotated the output below:

2020.028.12:07:03
###############################
Checking dbbc3hb
###############################
dbbcifc/ 2,32,agc,2,40019,40000;
dbbcifd/ 2,23,agc,2,39843,40000;

This first block is the equivalent of iread - check the inputs and filters are set to 2 and that the power level matches the target

dbbc017/ 2204.990000,a,32,1,agc,64,73,14921,14957,0,0;
dbbc018/ 2244.990000,a,32,1,agc,75,80,14837,14828,0,0;
dbbc019/ 2344.990000,a,32,1,agc,100,121,14883,14875,0,0;
dbbc020/ 2504.990000,a,32,1,agc,151,133,14822,14915,0,0;
dbbc021/ 2724.990000,a,32,1,agc,135,148,15015,14956,0,0;
dbbc022/ 2844.990000,a,32,1,agc,230,233,14936,14889,0,0;
dbbc023/ 2884.990000,a,32,1,agc,227,243,14899,14933,0,0;
dbbc024/ 2924.990000,a,32,1,agc,255,255,14777,14163,0,0;
dbbc025/ 2204.990000,a,32,1,agc,60,64,15830,14184,0,0;
dbbc026/ 2244.990000,a,32,1,agc,65,74,14735,15623,0,0;
dbbc027/ 2344.990000,a,32,1,agc,92,105,14711,15240,0,0;
dbbc028/ 2504.990000,a,32,1,agc,147,114,14965,15007,0,0;
dbbc029/ 2724.990000,a,32,1,agc,122,130,14927,15052,0,0;
dbbc030/ 2844.990000,a,32,1,agc,182,211,14940,15090,0,0;
dbbc031/ 2884.990000,a,32,1,agc,228,220,14919,14975,0,0;
dbbc032/ 2924.990000,a,32,1,agc,209,255,14828,14455,0,0;

This is the quivalent of bread confirm all the values are within ~1000 of the 16000 target and that the frequencies match those listed above.

pps_delay/ [1]: 43 ns, [2] 39 ns, [3] 39 ns, [4] 39 ns, [5] 43 ns, [6] 39 ns, [7] 0 ns, [8] 0 ns;

This is the translated equivalent of dbbc=pps_delay, separated by processing board. Check that all the listed values are either 43 or 39 ns, and are stable.

core3hstats/ 
Core3H[3] Power:
Sampler 0: 44362700
Sampler 1: 44956375
Sampler 2: 44344937
Sampler 3: 43982701

The Core3H power readings from each sampler should all agree to within ~3-5%. This is the first of two samplers in use.

Core3H[3] Bstat.
Sampler 0:
11:       10 0.06%
10:     7822 50.06%
01:     7785 49.82%
00:        6 0.04%

Sampler 1:
11:       11 0.07%
10:     7771 49.73%
01:     7833 50.13%
00:        8 0.05%

Sampler 2:
11:       21 0.13%
10:     7986 51.11%
01:     7610 48.70%
00:        6 0.04%

Sampler 3:
11:        9 0.06%
10:     7757 49.64%
01:     7850 50.24%
00:        6 0.04%

The Bstats can be ignored at this time - they're displayed because it's complicated to hide them.

Core3H[3] Corr.
Sampler 0-1: 183273215
Sampler 1-2: 186151539
Sampler 2-3: 183984793

All inter-sampler correlation values should be above ~170000000.

;
core3hstats/ 
Core3H[4] Power:
Sampler 0: 64799981
Sampler 1: 64621395
Sampler 2: 65964916
Sampler 3: 64445221

The Core3H power readings from each sampler should all agree to within ~3-5%. This is the first of two samplers in use.

Core3H[4] Bstat.
Sampler 0:
11:       50 0.32%
10:     7707 49.32%
01:     7807 49.96%
00:       60 0.38%

Sampler 1:
11:       59 0.38%
10:     7778 49.78%
01:     7743 49.56%
00:       42 0.27%

Sampler 2:
11:       57 0.36%
10:     7804 49.95%
01:     7719 49.40%
00:       43 0.28%

Sampler 3:
11:       53 0.34%
10:     7639 48.89%
01:     7887 50.48%
00:       44 0.28%

The Bstats can be ignored at this time - they're displayed because it's complicated to hide them.

Core3H[4] Corr.
Sampler 0-1: 187136057
Sampler 1-2: 190993079
Sampler 2-3: 186760591

All inter-sampler correlation values should be above ~170000000.

;
###############################
Checking dbbcho
###############################
dbbcifa/4,9,agc,2,47129,48000,1
dbbcifb/4,24,agc,2,48036,48000,1

This is the equivalent of iread. All inputs for dbbcho should be 1, all filters 2 and all target power levels 48000

dbbc01/300.990000,a,32,1,agc,72,136,17885,17699,0,0
dbbc03/332.990000,a,32,1,agc,63,63,17873,17835,0,0
dbbc05/364.990000,a,32,1,agc,53,62,17617,17971,0,0
dbbc07/348.990000,a,32,1,agc,57,61,18100,17633,0,0
dbbc09/300.990000,b,32,1,agc,56,166,17989,17571,0,0
dbbc11/332.990000,b,32,1,agc,63,54,18342,18083,0,0
dbbc13/364.990000,b,32,1,agc,91,64,18088,17794,0,0
dbbc15/348.990000,b,32,1,agc,88,55,17788,18248,0,0

This is the equivalent of bread. Note that only odd-numbered bbcs are used. As this is S-band, power levels may vary significantly so a range of 10000-25000 is acceptable (but notable - please put an entr in the handover notes)

pps_delay/61971

The pps_delay should be stable at 61971, to within 1 count of so.

fila10g/sysstat

System status:
 Selected input      : vsi1
 Input sample rate   : 64000000 Hz
 VSI input swapped   : no
 VSI input bitmask   : 0x0000FFFF
 VSI input width     : 16 bit
 PPS count           : 23451

 TVG mode            : vsi-h

 MK5B timesync       : yes
 VDIF timesync       : yes
 GPS receiver        : installed

 Output              : started
 Output 0 format     : vdif
 Output 0 dest.      : 192.168.1.13:46227
 Output 1 format     : vdif
 Output 1 dest.      : none
 Ethernet ARPs       : off

 Selected VSI output : vsi1-2

FiLa10G %

This last block is the Fila10G status. Please confirm that these settings match the output when performing the checks.