This wiki is not maintained! Do not use this when setting up AuScope experiments!

Differences

This shows you the differences between two versions of the page.

--- analysis:fourfit [2014/01/19 23:50]
Jamie McCallum
+++ analysis:fourfit [2017/06/06 01:26] (current)
Jamie McCallum
@@ Line 1: / Line 1: @@
+Fourfit is the software that performs the multi-band delay fitting to the correlator visibilities. At present, magilla is set up with the latest version and has scripts installed which remove some of the donkey-work in preparing properly fringe-fitted data sets. The following recipe will work for data sets correlated at Curtin.
+  * Download the following files from the Pawsey server to ~/Downloads:  AUG013.MARK4.tar.gz, aug013.vex and aug013.v2d
+  * Run "postcorr_prep.sh aug013" to copy the data across to /data/AUSTRAL/ extract them into the appropriate folders. It will also get you to check the ovex files station ID assignments and generate the apriori SNR file (delete all the leading text before the first two headers).
+  * After completing the initial fringe-fitting (using the default settings defined by /data/AUSTRAL/cf_aust), you will be prompted to enter a command like "cd /data/AUSTRAL/aug013/1344 ; export CORDATA=/data/AUSTRAL/aug013/ ; export DATADIR=/data/AUSTRAL/aug013/ ; aedit -f alist.ed.out". Do so, and you will be able to make a first inspection of the data.
+====== Aedit ======
+After running "edi dup snr", run "psplot" and check for either grey scans (uncorrelated data) or brown ones indicating non-detections. If found, make sure there's a sensible/known reason for them (module failure, antenna windstowed, etc). If there's no explanation, get in contact with the correlator ASAP.
+------------
 There are two machines set up to run fourfit at present - hex2 (through the hex2:1 VNC session) or ACECOMPUTE (using acecompute.antcrc.utas.edu.au:1). The two systems give slightly different results, thought to be due to the difference in the  32/64-bit compilers. The 32-bit version on hex2 occasionally gives unreliable estimates of the SNR, while the 64-bit version on ACECOMPUTE does not. Bonn also uses the 64-bit version, so ACECOMPUTE in strongly recommended for use in processing the data.
@@ Line 5: / Line 20: @@
 Make a directory and download the tarball from CURTIN into this directory [''mkdir AUST10 ; cd AUST10; wget -c ftp://cira.ivec.org/aust10/AUST10.MARK4.tar'']
-Untar the AUST10.MARK4.tar file to create the 1234 directory. Copy the ~/cf_aust template file into the 1234 directoey as cf_1234 [''cp ~/cf_aust ~/AUST10/1234/'']
+Untar the AUST10.MARK4.tar file to create the 1234 directory. Copy the ~/cf_aust template file into the 1234 directory as cf_1234 [''cp ~/cf_aust ~/AUST10/1234/'']
 Change to the 1234 directory and create a list of scans with ''cd ~/AUST10/1234/ ; ls -d 219-* 220* > /tmp/scans'' (Replacing the 219 and 220 with the appropriate DOYs for the experiment.
@@ Line 11: / Line 26: @@
 Now perform an initial fourfit on the data. You need to do this scanwise as there are often corrupted scans in the data set which will crash fourfit otherwise. The script below gives verbose output for each scan:
-''for i in $(cat /tmp/scans) ; do fourfit -m 1 -c cf_1234 $i ; done''
+   for i in $(cat /tmp/scans) ; do fourfit -m 1 -c cf_1234 $i ; done
 A more concise version is included below:
-''for i in $(cat /tmp/scans) ; do echo $i ; fourfit -c cf_1234 $i ; done''
+<code>
+parallel fourfit -c cf_1234 ::: $(ls -d 12*)
+</code>
 Once the fourfit processes are complete (15-30 minutes?), run alist with ''alist * '' and then edit the file with the following commands
-''sed -i 's/ 16383 / 1234 /g' alist.out
+   sed -i 's/ 16383 / 1234 /g' alist.out
-grep -v " LL " alist.out | grep -v " ii " | grep -v " xx " | grep -v " gg " | grep -v " WW " > alist.ed.out
+   grep -v  '0.000000 0.000  -1  -1' alist.out > alist.ed.out
-''
 Download the skd file and reprocess it into an ovex file. NB - ''nskd_ovex.pl'' needs to have internet access - you'll need to define the proxy settings for most machines.
@@ Line 28: / Line 45: @@
 Edit the ovex file & change the site IDs to match the Curtin ones
-eg Warkworth = w
+   Warkworth = W
    Hobart12 = L
    Hart15 = g
    Katherine = i
-   Yarragadee = x
+   Yarragadee = e
+   Hobart26 = H
+   Hart26m = x
 Define these two variables as the directories that contain the 1234 directory.
-export CORDATA=/u/rd_scope/AUST10/
+   export CORDATA=/u/rd_scope/AUST10/
-export DATADIR=/u/rd_scope/AUST10/
+   export DATADIR=/u/rd_scope/AUST10/
 Then run alist and make it ready for use with:
-''aedit -f alist.ed.out ''
+   aedit -f alist.ed.out
-''      edi dup snr ''
+         edi dup snr
+===== Phase Cal estimation =====
-====== Manual Phasecal ======
-If you have a station with no [or weak] phase-cal, you will need to do a manual pcal. To do this, pick a scan with good SNR (50+) in both bands - you can check this within aedit with
+To generate pcal estimates for all the stations with working pcals, run the follwing commands in aedit
-''axis snr ''\\
+   qc 1-9
-''baselines LW''\\
+   sort basel
-''device /xs''\\
+   sort freq
-''plot''\\
+   edi inp
+   param 72 54 55 22
+   pwrite additivephase.txt
+   exit
-[L == Hobart12 is being used as the reference station here, and W == Warkworth is the station lacking phasecal]
+Then run
-Once you've identified your scan, exit aedit
+   ~/phasedisp_c additivephase 7 30
+   fgrep Median phasedisp_c.out > med.txt
-Modify your cf_1234 file to include these sections
+Check med.txt for the baselines. If your chosen reference station is the first station in the baseline, then the sign is correct, otherwise reverse it. Paste the median values in the "pc_phases.." lines in the cf_1234 file.
-if station W and f_group X
+Once you've got these in the cf_1234 file, delete previous fourfit scans with
-   pc_mode manual
-   pc_phases ghijklmn 0 0 0 0 0 0 0 0
-if station W and f_group S
+   cd /data/AUSTRAL/AUST11/1234/
-   pc_mode manual
+   find . -name '??.X.*' -delete
-   pc_phases abcdef 0 0 0 0 0 0
+   find . -name '??.S.*' -delete
-With the modified the cf_1234 file, run fourfit on your selected scan [219-1239 in this example]
+and then reprocess all the scans with the correct pcal settings.
-fourfit -pt -m 1 -c ../cf_1234 -bLW 219-1239
+for i in $(cat /tmp/scans) ; do echo $i ; fourfit -c cf_1234 $i ; done
-In he debugging output of each plot will be the estimated pcal phases for that scan. If the baseline is reference->manual pcal target, you'll need to reverse the sign on these after pasting them into your control file.
+Then re-create the alist file with
-Once the cf_1234 file is updated, re-run fourfit and check that the phases are aligned on zero in all the individual IF channels for that scan. If not, check for a sign error? Also, check that other scans also show the IF phases aligned (although not necessarily on zero).  If the manual pcal stops working after a certain time, you can re-fringe fit and specify valid time ranges for the varrious pcal values with
+   alist *
+   sed -i 's/ 16383 / 1234 /g' alist.out
+   grep -v  '0.000000 0.000  -1  -1' alist.out > alist.ed.out
-if station O and f_group S and scan < 256-165000
-  pc_phases abcdef -4.0    2.3    2.8   -1.7    1.0   -0.4
-if station O and f_group S and scan  256-170000 to 258-165000
+Run aedit to check the results. "sum 2" should have a better tally of good scans and only a few H or G-codes.
-  pc_phases abcdef   -4.0 290.3 146.8  -1.7  73.0  -0.
-if station O and f_group S and scan > 258-165000
+NB - G-codes are caused by having a bad channel in the frequency group where it's sensitivity is < half of the average. For S-band, it's generally RFI that causes this but some stations have chronic poor performance in some LSB channels. If you detect a large number of G-codes, it may be worth dropping the channel. To do so, you need to edit the cd_1234 file once again and add a line like
-  pc_phases abcdef -4.0   2.3   2.8  -1.7   1.0  -0.4
-===== Phase Cal estimation =====
+   if station W and f_group X
+      freqs g h i j k l m n+
+This will flag out the X8L channel [dubbed n- in HOPS] in the Warkworth data from the fitting process. Re-run fourfit on the affected scans and quickly check the results [ e,g, remove previous fourfit results with the find command above & then run ''fourfit -pt -m 1 -c cf_1234 ddd-hhmm''. Check that the G-code error is gone & that the scan is good. If so, reprocess the entire data set with the updated cf file and resume.
-To generate pcal estimates for all the stations with working pcals, run the follwing commands in aedit
+====== Manual Phasecal ======
-''qc 1-9
+If you have a station with no [or weak] phase-cal, you will need to do a manual pcal. To do this, pick a scan with good SNR (50+) in both bands - you can check this within aedit with
-sort basel
-sort freq
-edi inp
-param 72 54 55 22
-pwrite additivephase.txt
-exit
-''
-Then run
-''~/phasedisp_c additivephase 7 30
+   axis snr
-fgrep Median phasedisp_c.out > med.txt
+   baselines LW
-''
+   device /xs
-Check med.txt for the baselines. If your chosen reference station is the first station in the baseline, then the sign is correct, otherwise reverse it. Paste the median values in the "pc_phases.." lines in the cf_1234 file.
+   plot
-Once you've got these in the cf_1234 file, delete previous fourfit scans with
+[L == Hobart12 is being used as the reference station here, and W == Warkworth is the station lacking phasecal]
-''cd /data/AUSTRAL/AUST11/1234/
+Once you've identified your scan, exit aedit
-find . -name '??.X.*' -delete
-find . -name '??.S.*' -delete
-''
-and then reprocess all the scans with the correct pcal settings.
-for i in $(cat /tmp/scans) ; do echo $i ; fourfit -c cf_1234 $i ; done
+Modify your cf_1234 file to include these sections
-Then re-create the alist file with
+   if station W and f_group X
+      pc_mode manual
+      pc_phases ghijklmn 0 0 0 0 0 0 0 0
-''alist *
+   if station W and f_group S
-sed -i 's/ 16383 / 1234 /g' alist.out
+      pc_mode manual
-grep -v " LL " alist.out | grep -v " ii " | grep -v " xx " | grep -v " gg " | grep -v " WW " > alist.ed.out
+      pc_phases abcdef 0 0 0 0 0 0
-''
-Run aedit to check the results. "sum 2" should have a better tally of good scans and only a few H or G-codes.
+With the modified the cf_1234 file, run fourfit on your selected scan [219-1239 in this example]
+fourfit -pt -m 1 -c ../cf_1234 -bLW 219-1239
+In he debugging output of each plot will be the estimated pcal phases for that scan. If the baseline is reference->manual pcal target, you'll need to reverse the sign on these after pasting them into your control file.
+Once the cf_1234 file is updated, re-run fourfit and check that the phases are aligned on zero in all the individual IF channels for that scan. If not, check for a sign error? Also, check that other scans also show the IF phases aligned (although not necessarily on zero).  If the manual pcal stops working after a certain time, you can re-fringe fit and specify valid time ranges for the varrious pcal values with
+   if station O and f_group S and scan < 256-165000
+     pc_phases abcdef -4.0    2.3    2.8   -1.7    1.0   -0.4
+   if station O and f_group S and scan  256-170000 to 258-165000
+     pc_phases abcdef   -4.0 290.3 146.8  -1.7  73.0  -0.
+   if station O and f_group S and scan > 258-165000
+     pc_phases abcdef -4.0   2.3   2.8  -1.7   1.0  -0.4
-NB - G-codes are caused by having a bad channel in the frequency group where it's sensitivity is < half of the average. For S-band, it's generally RFI that causes this but some stations have chronic poor performance in some LSB channels. If you detect a large number of G-codes, it may be worth dropping the channel. To do so, you need to edit the cd_1234 file once again and add a line like
-''if station W and f_group X
+===== =====
-  freqs g h i j k l m n+
-''
-This will flag out the X8L channel [dubbed n- in HOPS] in the Warkworth data from the fitting process. Re-run fourfit on the affected scans and quickly check the results [ e,g, remove previous fourfit results with the find command above & then run ''fourfit -pt -m 1 -c cf_1234 ddd-hhmm''. Check that the G-code error is gone & that the can is good. If so, reprocess the entire data set with the updated cf file and resume.
 Recreate the alist [with sed & grep -v as before], load it in aedit and check the "sum 2" or psplot outputs to make sure it's improved things.

AuScope VLBI Operations Wiki

User Tools

Site Tools

This wiki is not maintained! Do not use this when setting up AuScope experiments!

Differences

Page Tools