The AUM experiments are designed to investigate and hopefully demonstrate the compatibility of the new wideband VGOS receiver with the existing S/X system. This is important for maintaining a global geodetic network during the transition to a fully global VGOS array.
The Hobart12 and Katherine antennas now have a functional wideband system, with reasonable sensitivity. As of January 2020, we are still observing using the same mode as in the AUM series. This uses the “VGOS”-mode DBBC3 software, recording channels with an (excessive) bandwidth of 32 MHz which will be dealt with in the correlation stage. Once the new DBBC3 firmware is confirmed as usable for our systems, we will move across to using that. As such, these instructions may be updated - please check the page carefully each time you are starting or monitoring an AUA/AUM experiment.
The major differences for observing an AUM experiment rather than a standard S/X experiment are that there is limited/no FS support for the hardware and that we need to run two DBBCs simultaneously to record the S- and X-band data. The recorder has also changed from the mark5B to a flexbuff which operates in a very different fashion. Preliminary scripts have been written to monitor the relevant parameters but again, these are likely to be changed and improved as needed.
=== Before the Experiment === [Updated by Lim 2021March04]
cd /usr2/sched wget ftp://ivs.bkg.bund.de/pub/vlbi/ivsdata/aux/2021/aum028/aum028.skd drudg aum028.skd hb (station) 11 (show/set equipment type) 1 16 1 1 (none, flexbuff[mark5b], none, none) 3 (make snap file) 5 (print summary) 0 (exit) scp /tmp/sched.tmp observer@ops-serv2:/vlbobs/ivs/sched/aum028hb.sum (tmp summary file might have a different file name! i.e. DR.tmp)
The next step is to create the procedure file with the appropriate commands. The easiest way is to copy the procedure file from a previous experiment using the same mode. For the 32-MHz mode (In use as of March 2020), the SI experiment setup can be used as the template. e.g:
cd /usr2/proc/ cp si.prc aum018hb.prc (Hb, Ke)
because Yg now use mark5B instead of flexbuff, use slogit to drudg as usual.
vncviewer dbbc3hb) and make sure that the DBBC3 Control DDC_V_v124.exe program is running, and that the final line of output reads Waiting for connection). vncviewer dbbcho for Hb and vncviewer dbbcke for Ke) and check that the correct control program is running (DBBC2 Control DDC v105E_2.exe for Hb, DBBC2 Control DDV v106E_120118.exe for Ke) and displays Waiting for connection)waiting for connection, it should give the IP address of the machine that's currently talking to it (e.g if it reads Command from 131.217.63.201: then the connection is from 131.217.63.201). You can lookup up the name of this machine using nslookup 131.217.63.201 - in this example it is the hobart machine that is talking with the DBBC. If a PCFS machine is connected, the best option is to terminate the FS on that machine and restart with a version that will not conflict. For the Hobart example, running fs-auscope will avoid this problem while fs-flexbuff will block communications. ./bin/aum.query.summ.py. If the script stalls (and does not return to the command prompt), then it's likely that either one or both of the DBBC control programs are not running, or that something else is talking to them. ~/bin/aum.query.summ.py successfully, you should be fine to configure the backend at Hobart 12 with ./bin/aum.setup.lim.py [exp_name]. NB - this script will force a synchronisation of the BBCs and is equivalent to running setupsx and fmset in sequence. Do not run the setup script during the experiment!./bin/aum.setup.dbbc3only.py instead.proc=aum018hb and configure the recorder with setupvgos~/bin/aum.query.py returns “pps_delay/ [1]: 43 ns, [2] 39 ns, [3] 39 ns, [4] 43 ns, [5] 43 ns, [6] 43 ns, [7] 0 ns, [8] 0 ns”. (If not, the dbbc3 may need to be reconfigured or even restarted)~/bin/aum.query.summ.py returns “Delays: OK” and “Samplers: Ok” (if the IF or BBC power levels are reported as bad, this should be re-checked after a few minutes but they might be out of range when freshly set up). ~/bin/sxtimetest.sh which will print out the decoded time of the start of the test recording. All three values should agree and match the time reported in the FS log for when the “record=on” command was issued. If the times differ between the different datastreams, you will need to reconfigure the relevant DBBCs. If they are all in agreement but differ from the FS log, please note this difference in the log but proceed on to the next step. log=aum018hbschedule=aum018hb,#1fs.
A new script has been created which summarises the DBBC3/DBBC2 status. A version is installed on the pcfs machines as ~oper/bin/aum.query.summ.py and should be running in the VNC session. The Hobart and Katherine menus on ops8 also have a new entry called AUM Monitor which will start a version on ops, running in an xterm and refreshing every 60s. The output is below:
2020.056.05:00:06 ################## DBBC3 - Delays: OK DBBC2 - Delays: OK ################## DBBC3 - Sampler 3: OK DBBC3 - Sampler 4: OK ##################### DBBC3 IFC: OK DBBC3 IFD: OK DBBC3 Freq BW Power DBBC3 Freq BW Power ############################################################# BBC17 2204.99 32 OK BBC25 2204.99 32 OK BBC18 2244.99 32 OK BBC26 2244.99 32 OK BBC19 2344.99 32 OK BBC27 2344.99 32 OK BBC20 2504.99 32 OK BBC28 2504.99 32 OK BBC21 2724.99 32 OK BBC29 2724.99 32 OK BBC22 2844.99 32 OK BBC30 2844.99 32 OK BBC23 2884.99 32 OK BBC31 2884.99 32 OK BBC24 2924.99 32 OK BBC32 2924.99 32 OK DBBC2 IFA: Bad DBBC2 IFB: Bad DBBC2 Freq BW Power DBBC2 Freq BW Power ############################################################# BBC01 300.99 32 OK BBC09 300.99 32 OK BBC03 332.99 32 OK BBC11 332.99 32 OK BBC05 364.99 32 OK BBC13 364.99 32 OK BBC07 348.99 32 OK BBC15 348.99 32 OK
When carrying out a check, please run this script and check the outputs. Any errors related to the delays or DBBC3 samplers are critical are should be followed up (phone Jamie). Errors related to the IF or BBC levels should be noted - if these are consistently bad (in three consecutive readings), please contact Jamie to follow this up.
Taking the existing e-remote control checklists as a template, here is a listing of the significant differences. Most of the information is gathered by the ~/oper/bin/aum.query.py script. This can be called from the FS with sy=bin/aum.query.py with the outputs printed to screen and also logged to a file called e.g: /usr2/log/aua060hb.dbbc.log. It's strongly recommended to have a terminal watching this (tail -100f /usr2/log/aua060hb.dbbc.log) as it can be difficult to read from the FS display. An example of the output is given at the end of this page.
* RF and IF paths configured/DBBC settings: The RF/IF/DBBC configuration has been compiled into a single script on the pcfs machine called ~/aum.setup.py. Running this should set up all the essential settings for both the X-band recorded through the DBBC3 and S-band through the DBBC2 but the script should not be run during an experiment to avoid unnecessary clock breaks.
aua060hb.dbbc.log are correct. For Hobart12, these should be dbbcifa=4,**11**,agc,2,**47680**, 48000 dbbcifb=4,**27**,agc,2,**47843**,48000 dbbcifc=2,32,agc,2,**40073**,40000 dbbcifd=2,23,agc,2,**40070**,40000
and for Katherine:
dbbcifa=2,**38**,agc,2,**42008**,42000 dbbcifb=2,**37**,agc,2,**41669**,42000 dbbcifc=2,**29**,agc,2,**48178**,48000 dbbcifd=2,**23**,agc,2,**47928**,48000
NB - values with asterisks may vary depending on the input power level. Just make sure that the IF power readings is within ~1000 counts of the target.
ssh observer@flexbuffhb) and confirm that the time is correct (with ntpq -np) and that jive5ab is running (with ps -ef | grep jive)maserdelay reads the maser-GPS difference as usual while dbbc3delay logs the DBBC3-GPS delay as gps-fmout for Katherine. dbbc2delay (at Katherine) will log the DBBC2-GPS difference. For Hobart12, the DBBC3-maser difference is being printed in a terminal on in the pcfshb VNC session - this should be ~0.1 microseconds. If the display is not present, or not updating try running cat /dev/ttyS0 as root on flexbuffhb. disk_record=on, disk_record=off and checkmk5 in the operator input as per usual. ./bin/sxtimetest.sh on pcfshb. This will make a short test recording and then present the decoded time of the three files - all three entries should agree to the integer second.maserdelay and dbbc3delay (Ke only).
Most of the additional monitoring of the DBBC3 is currently logged to a separate log file (/usr2/log/aua060hb.dbbc.log), updated every scan when the sy=bin/aum.query.py | tee … script is run. This is part of the midob procedure. I've annotated the output below:
2020.028.12:07:03 ############################### Checking dbbc3hb ############################### dbbcifc/ 2,32,agc,2,40019,40000; dbbcifd/ 2,23,agc,2,39843,40000;
This first block is the equivalent of iread - check the inputs and filters are set to 2 and that the power level matches the target
dbbc017/ 2204.990000,a,32,1,agc,64,73,14921,14957,0,0; dbbc018/ 2244.990000,a,32,1,agc,75,80,14837,14828,0,0; dbbc019/ 2344.990000,a,32,1,agc,100,121,14883,14875,0,0; dbbc020/ 2504.990000,a,32,1,agc,151,133,14822,14915,0,0; dbbc021/ 2724.990000,a,32,1,agc,135,148,15015,14956,0,0; dbbc022/ 2844.990000,a,32,1,agc,230,233,14936,14889,0,0; dbbc023/ 2884.990000,a,32,1,agc,227,243,14899,14933,0,0; dbbc024/ 2924.990000,a,32,1,agc,255,255,14777,14163,0,0; dbbc025/ 2204.990000,a,32,1,agc,60,64,15830,14184,0,0; dbbc026/ 2244.990000,a,32,1,agc,65,74,14735,15623,0,0; dbbc027/ 2344.990000,a,32,1,agc,92,105,14711,15240,0,0; dbbc028/ 2504.990000,a,32,1,agc,147,114,14965,15007,0,0; dbbc029/ 2724.990000,a,32,1,agc,122,130,14927,15052,0,0; dbbc030/ 2844.990000,a,32,1,agc,182,211,14940,15090,0,0; dbbc031/ 2884.990000,a,32,1,agc,228,220,14919,14975,0,0; dbbc032/ 2924.990000,a,32,1,agc,209,255,14828,14455,0,0;
This is the quivalent of bread confirm all the values are within ~1000 of the 16000 target and that the frequencies match those listed above.
pps_delay/ [1]: 43 ns, [2] 39 ns, [3] 39 ns, [4] 39 ns, [5] 43 ns, [6] 39 ns, [7] 0 ns, [8] 0 ns;
This is the translated equivalent of dbbc=pps_delay, separated by processing board. Check that all the listed values are either 43 or 39 ns, and are stable.
core3hstats/ Core3H[3] Power: Sampler 0: 44362700 Sampler 1: 44956375 Sampler 2: 44344937 Sampler 3: 43982701
The Core3H power readings from each sampler should all agree to within ~3-5%. This is the first of two samplers in use.
Core3H[3] Bstat. Sampler 0: 11: 10 0.06% 10: 7822 50.06% 01: 7785 49.82% 00: 6 0.04% Sampler 1: 11: 11 0.07% 10: 7771 49.73% 01: 7833 50.13% 00: 8 0.05% Sampler 2: 11: 21 0.13% 10: 7986 51.11% 01: 7610 48.70% 00: 6 0.04% Sampler 3: 11: 9 0.06% 10: 7757 49.64% 01: 7850 50.24% 00: 6 0.04%
The Bstats can be ignored at this time - they're displayed because it's complicated to hide them.
Core3H[3] Corr. Sampler 0-1: 183273215 Sampler 1-2: 186151539 Sampler 2-3: 183984793
All inter-sampler correlation values should be above ~170000000.
; core3hstats/ Core3H[4] Power: Sampler 0: 64799981 Sampler 1: 64621395 Sampler 2: 65964916 Sampler 3: 64445221
The Core3H power readings from each sampler should all agree to within ~3-5%. This is the first of two samplers in use.
Core3H[4] Bstat. Sampler 0: 11: 50 0.32% 10: 7707 49.32% 01: 7807 49.96% 00: 60 0.38% Sampler 1: 11: 59 0.38% 10: 7778 49.78% 01: 7743 49.56% 00: 42 0.27% Sampler 2: 11: 57 0.36% 10: 7804 49.95% 01: 7719 49.40% 00: 43 0.28% Sampler 3: 11: 53 0.34% 10: 7639 48.89% 01: 7887 50.48% 00: 44 0.28%
The Bstats can be ignored at this time - they're displayed because it's complicated to hide them.
Core3H[4] Corr. Sampler 0-1: 187136057 Sampler 1-2: 190993079 Sampler 2-3: 186760591
All inter-sampler correlation values should be above ~170000000.
; ############################### Checking dbbcho ############################### dbbcifa/4,9,agc,2,47129,48000,1 dbbcifb/4,24,agc,2,48036,48000,1
This is the equivalent of iread. All inputs for dbbcho should be 1, all filters 2 and all target power levels 48000
dbbc01/300.990000,a,32,1,agc,72,136,17885,17699,0,0 dbbc03/332.990000,a,32,1,agc,63,63,17873,17835,0,0 dbbc05/364.990000,a,32,1,agc,53,62,17617,17971,0,0 dbbc07/348.990000,a,32,1,agc,57,61,18100,17633,0,0 dbbc09/300.990000,b,32,1,agc,56,166,17989,17571,0,0 dbbc11/332.990000,b,32,1,agc,63,54,18342,18083,0,0 dbbc13/364.990000,b,32,1,agc,91,64,18088,17794,0,0 dbbc15/348.990000,b,32,1,agc,88,55,17788,18248,0,0
This is the equivalent of bread. Note that only odd-numbered bbcs are used. As this is S-band, power levels may vary significantly so a range of 10000-25000 is acceptable (but notable - please put an entr in the handover notes)
pps_delay/61971
The pps_delay should be stable at 61971, to within 1 count of so.
fila10g/sysstat System status: Selected input : vsi1 Input sample rate : 64000000 Hz VSI input swapped : no VSI input bitmask : 0x0000FFFF VSI input width : 16 bit PPS count : 23451 TVG mode : vsi-h MK5B timesync : yes VDIF timesync : yes GPS receiver : installed Output : started Output 0 format : vdif Output 0 dest. : 192.168.1.13:46227 Output 1 format : vdif Output 1 dest. : none Ethernet ARPs : off Selected VSI output : vsi1-2 FiLa10G %
This last block is the Fila10G status. Please confirm that these settings match the output when performing the checks.