====== Some background ====== The AUM experiments are designed to investigate and hopefully demonstrate the compatibility of the new wideband VGOS receiver with the existing S/X system. This is important for maintaining a global geodetic network during the transition to a fully global VGOS array. The Hobart12 and Katherine antennas now have a functional wideband system, with reasonable sensitivity. As of January 2020, we are still observing using the same mode as in the AUM series. This uses the "VGOS"-mode DBBC3 software, recording channels with an (excessive) bandwidth of 32 MHz which will be dealt with in the correlation stage. Once the new DBBC3 firmware is confirmed as usable for our systems, we will move across to using that. As such, these instructions may be updated - please check the page carefully each time you are starting or monitoring an AUA/AUM experiment. ===== Major Differences ===== The major differences for observing an AUM experiment rather than a standard S/X experiment are that there is limited/no FS support for the hardware and that we need to run two DBBCs simultaneously to record the S- and X-band data. The recorder has also changed from the mark5B to a flexbuff which operates in a very different fashion. Preliminary scripts have been written to monitor the relevant parameters but again, these are likely to be changed and improved as needed. === Before the Experiment === [Updated by Lim 2021March04] * Preparing the schedule file: **Do not use the existing slogit scripts** for **Hb** and **Ke**, but instead download the skd file directly to pcfs[hb|ke|yg] and run drudg to generate the snp and sum files and put them where they're needed. cd /usr2/sched wget ftp://ivs.bkg.bund.de/pub/vlbi/ivsdata/aux/2021/aum028/aum028.skd drudg aum028.skd hb (station) 11 (show/set equipment type) 1 16 1 1 (none, flexbuff[mark5b], none, none) 3 (make snap file) 5 (print summary) 0 (exit) scp /tmp/sched.tmp observer@ops-serv2:/vlbobs/ivs/sched/aum028hb.sum (tmp summary file might have a different file name! i.e. DR.tmp) The next step is to create the procedure file with the appropriate commands. The easiest way is to copy the procedure file from a previous experiment using the same mode. For the 32-MHz mode (In use as of March 2020), the SI experiment setup can be used as the template. e.g: cd /usr2/proc/ cp si.prc aum018hb.prc (Hb, Ke) **because Yg now use mark5B instead of flexbuff, use slogit to drudg as usual.** === Setting up for Hb, Ke === * Terminate the FS and restart it with "fs-dummy". (To ensure that the Field system will not connect to the DBBC2 and block the configuration scripts) * Log in to **DBBC3** (''vncviewer dbbc3hb'') and make sure that the ''DBBC3 Control DDC_V_v124.exe'' program is running, and that the final line of output reads ''Waiting for connection''). * Next, check the **DBBC2** (''vncviewer dbbcho'' for Hb and ''vncviewer dbbcke'' for Ke) and check that the correct control program is running (''DBBC2 Control DDC v105E_2.exe'' for Hb, ''DBBC2 Control DDV v106E_120118.exe'' for Ke) and displays ''Waiting for connection'') * If the server program is not running, please start it from the link on the desktop (and be prepared for a 15-20 minute wait for the DBBC3 after answering "y" to the reconfiguration question). * If it does not read ''waiting for connection'', it should give the IP address of the machine that's currently talking to it (e.g if it reads ''Command from 131.217.63.201:'' then the connection is from 131.217.63.201). You can lookup up the name of this machine using ''nslookup 131.217.63.201'' - in this example it is the ''hobart'' machine that is talking with the DBBC. If a PCFS machine is connected, the best option is to terminate the FS on that machine and restart with a version that will not conflict. For the Hobart example, running ''fs-auscope'' will avoid this problem while ''fs-flexbuff'' will block communications. * From a terminal on the pcfs machine, run the command ''./bin/aum.query.summ.py''. If the script stalls (and does not return to the command prompt), then it's likely that either one or both of the DBBC control programs are not running, or that something else is talking to them. * Once you can run ''~/bin/aum.query.summ.py'' successfully, you should be fine to configure the backend at Hobart 12 with ''./bin/aum.setup.lim.py [exp_name]''. NB - this script will force a synchronisation of the BBCs and is equivalent to running ''setupsx'' and ''fmset'' in sequence. **Do not run the setup script during the experiment!** * **As of March 2021, only dbbc3ke is available at Katherine, please setup using ''./bin/aum.setup.dbbc3only.py'' instead.** * Load the procedures with ''proc=aum018hb'' and configure the recorder with ''setupvgos'' * Check the ''~/bin/aum.query.py'' returns "pps_delay/ [1]: 43 ns, [2] 39 ns, [3] 39 ns, [4] 43 ns, [5] 43 ns, [6] 43 ns, [7] 0 ns, [8] 0 ns". (If not, the dbbc3 may need to be reconfigured or even restarted) * Check that ''~/bin/aum.query.summ.py'' returns "Delays: OK" and "Samplers: Ok" (if the IF or BBC power levels are reported as bad, this should be re-checked after a few minutes but they might be out of range when freshly set up). * You can make a test recording to check for synchrnoisation problems between the different DBBCs with ''~/bin/sxtimetest.sh'' which will print out the decoded time of the start of the test recording. All three values should agree and match the time reported in the FS log for when the "record=on" command was issued. If the times differ between the different datastreams, you will need to reconfigure the relevant DBBCs. If they are all in agreement but differ from the FS log, please note this difference in the log but proceed on to the next step. * Set the log file for the experiment with ''log=aum018hb'' * Load the schedule file with ''schedule=aum018hb,#1'' == Setting up for Yg == * Yarragadee should be set up normally, using the DBBC2 (v104_2.exe) and mk5yg - Please make sure this is configured by starting the Field System with ''fs''. ==== Revised checklist for Hb, Ke==== A new script has been created which summarises the DBBC3/DBBC2 status. A version is installed on the pcfs machines as ''~oper/bin/aum.query.summ.py'' and should be running in the VNC session. The Hobart and Katherine menus on ''ops8'' also have a new entry called ''AUM Monitor'' which will start a version on ops, running in an xterm and refreshing every 60s. The output is below: 2020.056.05:00:06 ################## DBBC3 - Delays: OK DBBC2 - Delays: OK ################## DBBC3 - Sampler 3: OK DBBC3 - Sampler 4: OK ##################### DBBC3 IFC: OK DBBC3 IFD: OK DBBC3 Freq BW Power DBBC3 Freq BW Power ############################################################# BBC17 2204.99 32 OK BBC25 2204.99 32 OK BBC18 2244.99 32 OK BBC26 2244.99 32 OK BBC19 2344.99 32 OK BBC27 2344.99 32 OK BBC20 2504.99 32 OK BBC28 2504.99 32 OK BBC21 2724.99 32 OK BBC29 2724.99 32 OK BBC22 2844.99 32 OK BBC30 2844.99 32 OK BBC23 2884.99 32 OK BBC31 2884.99 32 OK BBC24 2924.99 32 OK BBC32 2924.99 32 OK DBBC2 IFA: Bad DBBC2 IFB: Bad DBBC2 Freq BW Power DBBC2 Freq BW Power ############################################################# BBC01 300.99 32 OK BBC09 300.99 32 OK BBC03 332.99 32 OK BBC11 332.99 32 OK BBC05 364.99 32 OK BBC13 364.99 32 OK BBC07 348.99 32 OK BBC15 348.99 32 OK When carrying out a check, please run this script and check the outputs. Any errors related to the delays or DBBC3 samplers are critical are should be followed up (phone Jamie). Errors related to the IF or BBC levels should be noted - if these are consistently bad (in three consecutive readings), please contact Jamie to follow this up. === Checklists === Taking the existing e-remote control checklists as a template, here is a listing of the significant differences. Most of the information is gathered by the ''~/oper/bin/aum.query.py'' script. This can be called from the FS with ''sy=bin/aum.query.py'' with the outputs printed to screen and also logged to a file called e.g: ''/usr2/log/aua060hb.dbbc.log''. It's strongly recommended to have a terminal watching this (''tail -100f /usr2/log/aua060hb.dbbc.log'') as it can be difficult to read from the FS display. An example of the output is given at the end of this page. * RF and IF paths configured/DBBC settings: The RF/IF/DBBC configuration has been compiled into a single script on the pcfs machine called ''~/aum.setup.py''. Running this should set up all the essential settings for both the X-band recorded through the DBBC3 and S-band through the DBBC2 but the script should not be run during an experiment to avoid unnecessary clock breaks. * Check that the dbbcif[abcd] input, filter and power target level readings in the ''aua060hb.dbbc.log'' are correct. For Hobart12, these should be dbbcifa=4,**11**,agc,2,**47680**, 48000 dbbcifb=4,**27**,agc,2,**47843**,48000 dbbcifc=2,32,agc,2,**40073**,40000 dbbcifd=2,23,agc,2,**40070**,40000 and for Katherine: dbbcifa=2,**38**,agc,2,**42008**,42000 dbbcifb=2,**37**,agc,2,**41669**,42000 dbbcifc=2,**29**,agc,2,**48178**,48000 dbbcifd=2,**23**,agc,2,**47928**,48000 NB - values with asterisks may vary depending on the input power level. Just make sure that the IF power readings is within ~1000 counts of the target. * DBBC server: If the aum.query.py script cannot communicate with the DBBC servers, you can restart the software running through the VNC sessions as described in the setup procedure. Again, please note that this will introduce a clock break so make sure that there's no other issue like another machine having connected to the DBBC server. * Maser ok: Check the Hobart26 maser as per usual * Mark5: Not used. Instead log in to the Flexbuff recorder (with ''ssh observer@flexbuffhb'') and confirm that the time is correct (with ''ntpq -np'') and that jive5ab is running (with ''ps -ef | grep jive'') * The clock offsets can be (partially) read through the FS. ''maserdelay'' reads the maser-GPS difference as usual while ''dbbc3delay'' logs the DBBC3-GPS delay //as gps-fmout// for Katherine. dbbc2delay (at Katherine) will log the DBBC2-GPS difference. For Hobart12, the DBBC3-maser difference is being printed in a terminal on in the pcfshb VNC session - this should be ~0.1 microseconds. If the display is not present, or not updating try running ''cat /dev/ttyS0'' as root on flexbuffhb. * Test Recordings: To make a test recording, simply run ''disk_record=on'', ''disk_record=off'' and ''checkmk5'' in the operator input as per usual. * Check for any integer second offsets in the recorded data by running ''./bin/sxtimetest.sh'' on pcfshb. This will make a short test recording and then present the decoded time of the three files - all three entries should agree to the integer second. * No autocorrelation script has been implemented yet. * There is no systemp measurewment implemented yet. * There is no pointing/SEFD check implemented as yet * No fringe check implemented as yet. == Monitoring == * Antenna checks as per usual * Check data is being recorded in three streams (mk5=datastream? 0:3:sxy:xx:xy) and 3 files the same size per scan on flexbuff vbs_ls -lh aum019_* | tail * LO check unavailable * No autocorrelations * Check the delays with ''maserdelay'' and ''dbbc3delay'' (Ke only). * Check the Hb and Ke masers as usual. * Check the weather data is being logged & add in sky conditions as usual. * No Tsys checks. == Additional log output == Most of the additional monitoring of the DBBC3 is currently logged to a separate log file (''/usr2/log/aua060hb.dbbc.log''), updated every scan when the ''sy=bin/aum.query.py | tee ...'' script is run. This is part of the ''midob'' procedure. I've annotated the output below: 2020.028.12:07:03 ############################### Checking dbbc3hb ############################### dbbcifc/ 2,32,agc,2,40019,40000; dbbcifd/ 2,23,agc,2,39843,40000; This first block is the equivalent of ''iread'' - check the inputs and filters are set to ''2'' and that the power level matches the target dbbc017/ 2204.990000,a,32,1,agc,64,73,14921,14957,0,0; dbbc018/ 2244.990000,a,32,1,agc,75,80,14837,14828,0,0; dbbc019/ 2344.990000,a,32,1,agc,100,121,14883,14875,0,0; dbbc020/ 2504.990000,a,32,1,agc,151,133,14822,14915,0,0; dbbc021/ 2724.990000,a,32,1,agc,135,148,15015,14956,0,0; dbbc022/ 2844.990000,a,32,1,agc,230,233,14936,14889,0,0; dbbc023/ 2884.990000,a,32,1,agc,227,243,14899,14933,0,0; dbbc024/ 2924.990000,a,32,1,agc,255,255,14777,14163,0,0; dbbc025/ 2204.990000,a,32,1,agc,60,64,15830,14184,0,0; dbbc026/ 2244.990000,a,32,1,agc,65,74,14735,15623,0,0; dbbc027/ 2344.990000,a,32,1,agc,92,105,14711,15240,0,0; dbbc028/ 2504.990000,a,32,1,agc,147,114,14965,15007,0,0; dbbc029/ 2724.990000,a,32,1,agc,122,130,14927,15052,0,0; dbbc030/ 2844.990000,a,32,1,agc,182,211,14940,15090,0,0; dbbc031/ 2884.990000,a,32,1,agc,228,220,14919,14975,0,0; dbbc032/ 2924.990000,a,32,1,agc,209,255,14828,14455,0,0; This is the quivalent of ''bread'' confirm all the values are within ~1000 of the 16000 target and that the frequencies match those listed above. pps_delay/ [1]: 43 ns, [2] 39 ns, [3] 39 ns, [4] 39 ns, [5] 43 ns, [6] 39 ns, [7] 0 ns, [8] 0 ns; This is the translated equivalent of ''dbbc=pps_delay'', separated by processing board. Check that all the listed values are either 43 or 39 ns, and are stable. core3hstats/ Core3H[3] Power: Sampler 0: 44362700 Sampler 1: 44956375 Sampler 2: 44344937 Sampler 3: 43982701 The Core3H power readings from each sampler should all agree to within ~3-5%. This is the first of two samplers in use. Core3H[3] Bstat. Sampler 0: 11: 10 0.06% 10: 7822 50.06% 01: 7785 49.82% 00: 6 0.04% Sampler 1: 11: 11 0.07% 10: 7771 49.73% 01: 7833 50.13% 00: 8 0.05% Sampler 2: 11: 21 0.13% 10: 7986 51.11% 01: 7610 48.70% 00: 6 0.04% Sampler 3: 11: 9 0.06% 10: 7757 49.64% 01: 7850 50.24% 00: 6 0.04% The Bstats can be ignored at this time - they're displayed because it's complicated to hide them. Core3H[3] Corr. Sampler 0-1: 183273215 Sampler 1-2: 186151539 Sampler 2-3: 183984793 All inter-sampler correlation values should be above ~170000000. ; core3hstats/ Core3H[4] Power: Sampler 0: 64799981 Sampler 1: 64621395 Sampler 2: 65964916 Sampler 3: 64445221 The Core3H power readings from each sampler should all agree to within ~3-5%. This is the first of two samplers in use. Core3H[4] Bstat. Sampler 0: 11: 50 0.32% 10: 7707 49.32% 01: 7807 49.96% 00: 60 0.38% Sampler 1: 11: 59 0.38% 10: 7778 49.78% 01: 7743 49.56% 00: 42 0.27% Sampler 2: 11: 57 0.36% 10: 7804 49.95% 01: 7719 49.40% 00: 43 0.28% Sampler 3: 11: 53 0.34% 10: 7639 48.89% 01: 7887 50.48% 00: 44 0.28% The Bstats can be ignored at this time - they're displayed because it's complicated to hide them. Core3H[4] Corr. Sampler 0-1: 187136057 Sampler 1-2: 190993079 Sampler 2-3: 186760591 All inter-sampler correlation values should be above ~170000000. ; ############################### Checking dbbcho ############################### dbbcifa/4,9,agc,2,47129,48000,1 dbbcifb/4,24,agc,2,48036,48000,1 This is the equivalent of ''iread''. All inputs for dbbcho should be ''1'', all filters ''2'' and all target power levels ''48000'' dbbc01/300.990000,a,32,1,agc,72,136,17885,17699,0,0 dbbc03/332.990000,a,32,1,agc,63,63,17873,17835,0,0 dbbc05/364.990000,a,32,1,agc,53,62,17617,17971,0,0 dbbc07/348.990000,a,32,1,agc,57,61,18100,17633,0,0 dbbc09/300.990000,b,32,1,agc,56,166,17989,17571,0,0 dbbc11/332.990000,b,32,1,agc,63,54,18342,18083,0,0 dbbc13/364.990000,b,32,1,agc,91,64,18088,17794,0,0 dbbc15/348.990000,b,32,1,agc,88,55,17788,18248,0,0 This is the equivalent of ''bread''. Note that only odd-numbered bbcs are used. As this is S-band, power levels may vary significantly so a range of 10000-25000 is acceptable (but notable - please put an entr in the handover notes) pps_delay/61971 The pps_delay should be stable at 61971, to within 1 count of so. fila10g/sysstat System status: Selected input : vsi1 Input sample rate : 64000000 Hz VSI input swapped : no VSI input bitmask : 0x0000FFFF VSI input width : 16 bit PPS count : 23451 TVG mode : vsi-h MK5B timesync : yes VDIF timesync : yes GPS receiver : installed Output : started Output 0 format : vdif Output 0 dest. : 192.168.1.13:46227 Output 1 format : vdif Output 1 dest. : none Ethernet ARPs : off Selected VSI output : vsi1-2 FiLa10G % This last block is the Fila10G status. Please confirm that these settings match the output when performing the checks.