autoaips -- a tool to generate aips runfiles for a particular observation. Written for the VLBA parallax project. Walter Brisken, Socorro NM July 2002 Documentation version 1.0 Last modified 2002/11/22, SC. 1. Building and installation: A few standard unix programs will be required for installation and at autoaips runtime: strings, grep, wget, python. Any modern Linux installation will certainly provide these. No guarantee on Solaris. Get the latest version of autoaips: see the downloads section on http://www.astro.cornell.edu/~shami/psrvlb/ i. Download the .tar.gz file ii. Extract : tar zxvf autoaips-.tar.gz iii. cd autoaips iv. Take a look at the changelog in autoaips.c v. edit Makefile so that INSTDIR points to the desired installation directory. vi. make install Note that you need write permission for INSTDIR. Installation puts two files in INSTDIR: autoaips and ionex.py. If INSTDIR is not in your path (test by changing to a different directory and trying to run autoaips with no arguments) you need an environment variable IONEXPY pointing to ionex.py. For example, if INSTDIR is /home/aips/random: tcsh % setenv IONEXPY /home/aips/random/ionex.py bash $ IONEXPY=/home/aips/random/ionex.py It is probably more convenient to add the above command to your .tcshrc/.bash_profile. 2. Calibration strategy: Autoaips generates scripts (all of the form X*.001) for the following calibration strategy: 1. Load the data run fitld separately on each VLBA UVFITS file. Always concatenate the data sets. 2. Delete provided NX table and rerun INDXR. This ensures that no scans are split into 2 by gaps caused by tape reversal 3. Mergecal (divided into 3 TAMRG calls) to combine the GC, TY, and PC tables from each of the FITS files. 4. Run ACCOR to correct amplitudes based on autocorrelation values 5. Apply gain table to data (APCAL) 6. Apply JPL (or other?) ionosphere data 7. Flag data with commands specified in the .x.N file (N=epochnum) 8. Fringe fit the bandpass calibrator (and show fringe phases) 9. Image the calibrator and self cal for a couple loops 10. Re-fringe the bandpass calibrator using model from 9. 11. Self calibrate bandpass calibrator within the multi-source file 12. Run BPASS to calculate the bandpass response. 13. Show the results using POSSM. 14. Fring fit and self calibrate the phase calibrators A model of the calibrator is saved (.MODEL) The pulsar or in-beam calibrator is split out using this calibration. 3. How to run autoaips: AIPS uses environment variables to specify disk areas. You will need to use the same one when running autoaips. - First copy all .FITS files to an empty directory. - Set the environment variable to point to this directory. Remember it needs to be all capital letters. - Then copy the .key file used for the observation and the .x.N file used to generate this .key file into the same directory. - Run autoaips: autoaips [ [] ] where is the environment variable pointing to the directory, is one of psr ib1 ib2 ib3 ... sets the INDISK and OUTDISK parameters inside AIPS. All files for the given project will stay on this same disk. If is specified, a different environment variable (here specified) will be used as the FITS file directory. If is provided, the environment variable there specified will be used for the box files. - Console output with a leading "**" indicates errors or warnings. Note that autoaips must be run separately for each correlation pass and each requires its own directory. Autoaips _should_ be immune from file clobbering due to experiments with similar file or source names, as long as the experiment code differs. Note that ionex.py must be in your path (as above) to get the ionosphere models. If the data is already in the directory, this program is not needed. 4. Autoaips output: autoaips generates several files in the directory pointed to by : .tablog A file listing what each aips table is (a * means table modified, not created) XLOAD.001 The script that loads the FITS data XAPRIORI.001 The script that runs apriori calibrations XUVFLG.001 A replication of the flag commands in XAPRIORI.001 for convenience XBPASS.001 A script to do a good job of band pass calibration. XPHASE.001 Script for phase calibration. Single source files for the calibrators and target are made. XGEOD.001 An experimental geodesy calibration step XCOPY.001 Script to copy calibration tables from one correlation pass to another. Must first set AIPS variable CALXFR jplg???0.0?i One or more JPL ionex files autoaips.save Currently just caches the source name, to speed up (tremendously) future runs of autoaips in that directory. 5. How to run these scripts in AIPS: Before entering AIPS, make sure the same environment variable used in step 3 is still pointing to the data directory. then start AIPS. Once aips is going, type VERSION = '' This allows aips to find the X*.001 scripts. Then just type RUN XLOAD to get going. Once that is finished, keep following the instructions that are given. Note that when running, aips will want to write data back to the directory, specifically the boxfiles used for image cleaning and any save files. Thus, if the data was distributed on CDROM, you should copy the contents to your hard disk. The XLOAD script writes nothing back to the disk, so it can be run off the CDROM, and then the non-FITS files can be copied to the hard disk for later steps if disk space is limited. Alternatively, the path to the data on read-only media can be specified with the argument to autoaips. The phase referenced target name will be .SPLIT for the pulsar, where is the number of the phase calibrator used (the script will run for each phase calibrator). For the in-beam passes, the output file will be ..SPLIT, where is the in-beam number. 6. CLEANUP and UVCLEANUP Two procedures are defined in each RUNFILE: CLEANUP -- goes through the current INDISK and deletes all files with extension XCALIB, I----*, UVTMP, IBM0** and it RECATs the disk. These extensions should be unique enough not to clobber any other data on the disk. UVCLEANUP -- goes through and deletes all original (un-FXPOLed) multisource UV files. All files with extensions XPSR or XIB* will be deleted. These can be run safely at any time that AIPS is not running one of the autoaips RUNFILES. It is ill-advised to run UVCLEANUP between XLOAD and as this would require running XLOAD again. 7. Customization Most customization is done by adding to the .x file. Since customization and flagging will differ epoch-to-epoch, each epoch has a dedicated .x file, identified by the epoch number (.x.1, .x.2, etc). A few bits of customization can be done, both before and after running autoaips. By default, autoaips uses refant = 4 (usually Kitt Peak). This can be changed either before autoaips is run by inserting a line refant = into the .x.N file, or by changing a line (obvious which!) in X*.001. It is recommended to use the first approach as this is less likely to cause inconsistencies. The default bandpass calibrator is given by geod_cal_name1 in the .x.N file. Adding a line bpass_cal_name = to the .x.N file will force it to use that instead. Known flags can also be inserted into the .x.N file. AIPS task UVFLG is used to execute the flag. In the .x.N file place a command of the form: flag1 = bif=2;eif=2;bchan=12;echan=14;ante=10 flag2 = ante=5;basel=6 flag3 = bchan=1;echan=1 Make sure no spaces are in the flag expression. All flags are initialized with no selection before being applied (ie, flagging adverbs are not cumulative). 8. Calibration Copying For many of the pulsar observation there will be two or more data sets that differ only in the way the target source (be that the pulsar or an in-beam calibrator) is correlated. In these cases, when processing the second or third set of data, the fringing and bandpassing can be avoided by copying the BP, FG, SN, CL tables from an existing data set that is still loaded into AIPS. For example, if BC123C inbeam pass #1 is already on disk, and BC123C inbeam pass #2 needs processing, all that needs to be done is: VERSION = 'BC123CIB2' RUN XLOAD RUN XAPRIORI RUN XCOPY XCOPY needs the variable CALXFR set to the class name of the source data set. This variable defaults to 'IB1' as all cases where XCOPY is needed can be done relative to the first in-beam calibrator. XCOPY copies all needed tables to the destination data set (in this case in-beam set number 2) and splits out the target source with each of the phase calibrators present. XCOPY does the equivalent of XBPASS and XPHASE. XGEOD is only needed on one correlation pass, presumably the first. 9. Additional notes Sometimes an ionosphere file does not exist on the JPL server. In this case TECOR step should be run manually or not run at all and the CL version 4 should be copied to version 5. If using a different refant or disk, rather than modifying all of the X*.001 files, it is highly recommended that the scripts are recreated with autoaips. The refant parameter is in the .x.N file. 10. A complete example walkthrough (tcsh): ----------------------------------------- a. Set up environment variables. Set up ionex.py parallax<170>% setenv IONEXPY /home/aips/random/ionex.py Set up disk areas and data on CDROM parallax<178>% setenv PSRDATA /mnt/cdrom parallax<179>% setenv BC123AIB1 /home/parallax2/big/BC123Aib1 parallax<180>% setenv BC123APSR /home/parallax2/big/BC123Apsr Note that the last two directories must be writable. If not using CDROM: parallax<177>% setenv PSRDATAIB1 /home/parallax2/big/data/BC123Aib1 parallax<178>% setenv PSRDATAPSR /home/parallax2/big/data/BC123Apsr parallax<179>% setenv BC123AIB1 /home/parallax2/big/BC123Aib1 parallax<180>% setenv BC123APSR /home/parallax2/big/BC123Apsr b. Make the data files available: All .key and .x.N files are available online, at http://www.astro.cornell.edu/~shami/psrvlb/observe/ Copy the .x.N and the .key file into $BC123AIB1. At this point a file listing should show: parallax<182>% ls -l $BC123AIB1 -rw-r--r-- 1 wbrisken staff 1599 Sep 7 10:16 B1508+55.x.1 -rw-r--r-- 1 wbrisken staff 2732 Aug 20 16:26 bc123a.key Put the CDROM in and mount it if your machine does not automount: parallax<184>% mount /mnt/cdrom and make sure $PSRDATA (or $PSRDATAIB1) contains the needed fits files: parallax<186>% ls -l $PSRDATA -rw-r--r-- 1 wbrisken staff 47836800 Aug 20 15:34 32200001.FITS -rw-r--r-- 1 wbrisken staff 12297600 Aug 20 15:45 32210001.FITS -rw-r--r-- 1 wbrisken staff 8294400 Aug 20 15:55 32220001.FITS -rw-r--r-- 1 wbrisken staff 23788800 Aug 20 16:03 32230001.FITS -rw-r--r-- 1 wbrisken staff 61891200 Aug 20 16:13 32240001.FITS -rw-r--r-- 1 wbrisken staff 254937600 Aug 20 16:22 32250001.FITS c. Run autoaips Make sure that ionex.py is in your path (try running it with no arguments): parallax<188>% ionex.py Usage : %s { | } Range of doys does not work around a year boundary or if IONEXPY is defined: parallax<190>% $IONEXPY Usage : %s { | } Range of doys does not work around a year boundary Now it is time to finally run autoaips: autoaips [ [] ] Figure out which disk in aips you want to use and type (if data is on CD) parallax<192>% autoaips BC123AIB1 ib1 2 PSRDATA or if from the disk: parallax<192>% autoaips BC123AIB1 ib1 2 PSRDATAIB1 Note: The second argument, ib1, tells autoaips that is is in-beam calibrator #1 for this source. Other possibilities are ib2, ib3, ... for other in-beams, or psr if the pulsar. The third argument, 2, is the AIPS disk to use. Autoaips may take a little while the first time it runs while searching the FITS file for the name of the pulsar, and it will eventually finish after outputting lots of messages to the screen. The fourth, optional, argument specifies the location of the UVFITS data. It defaults to if not provided. A fifth, optional, argument, not used here, points to the directory where box files can be stored. This argument defaults to if not provided. Now if you look at the contents of the script directory: parallax<191>% ls -l $BC123AIB1 -rw-r--r-- 1 wbrisken staff 11 Sep 8 11:49 autoaips.save -rw-r--r-- 1 wbrisken staff 1599 Sep 7 10:16 B1508+55.x -rw-r--r-- 1 wbrisken staff 2732 Aug 20 16:26 bc123a.key -rw-r--r-- 1 wbrisken staff 1660 Jun 30 13:07 bc123a.log -rw-r--r-- 1 wbrisken staff 1081 Sep 26 13:47 bc123a.tablog -rw-rw-r-- 1 wbrisken staff 90 Sep 7 11:01 J1429+5406.BOX -rw-rw-r-- 1 wbrisken staff 90 Sep 7 11:00 J1436+6336.BOX -rw-rw-r-- 1 wbrisken staff 90 Sep 7 10:26 J1510+5702.BOX -rw-rw-r-- 1 wbrisken staff 60 Sep 7 10:58 J1545+5135.BOX -rw-rw-r-- 1 wbrisken staff 30 Sep 7 11:26 J1551+5806.BOX -rw-r--r-- 1 wbrisken staff 786810 Aug 20 16:26 jplg1770.02i -rw-r--r-- 1 wbrisken staff 786729 Aug 20 16:26 jplg1780.02i -rw-r--r-- 1 wbrisken staff 5478 Sep 26 13:47 XAPRIORI.001 -rw-r--r-- 1 wbrisken staff 12066 Sep 26 13:47 XBPASS.001 -rw-r--r-- 1 wbrisken staff 6165 Sep 26 13:47 XCOPY.001 -rw-r--r-- 1 wbrisken staff 1598 Sep 8 11:49 XFLAG.001 -rw-r--r-- 1 wbrisken staff 22566 Sep 26 13:47 XGEOD.001 -rw-r--r-- 1 wbrisken staff 3576 Sep 26 13:47 XLOAD.001 -rw-r--r-- 1 wbrisken staff 5664 Sep 26 13:47 XPHASE.001 -rw-r--r-- 1 wbrisken staff 2155 Sep 26 13:47 XUVFLG.001 d. Start aips with the right "version": It is convenient to start AIPS in a new xterm. You will need to set the environment variables again in that terminal: parallax<1>% setenv PSRDATA /mnt/cdrom parallax<2>% setenv BC123AIB1 /home/parallax2/big/BC123Aib1 parallax<3>% setenv BC123APSR /home/parallax2/big/BC123Apsr parallax<4>% aips Make sure you have about 1 GB free disk space before you begin. About half of that is temporary storage for FXPOL and can be deleted quickly. >free AIPS 1: Disk Volume name Total Full Free Timd Access AIPS 1: # blocks % blocks days AIPS 1: 1 /DATA/PARALLAX_1 35001508 92 2638092 365.0 Alluser AIPS 1: 2 /DATA/PARALLAX_2 35001508 92 2638092 365.0 Alluser In aips, you need to set the "version" adverb to set the script directory. >version = 'BC123AIB1' Now things are ready to reduce the inbeam data. e. Load the inbeam data: The XLOAD.001 script contains the commands needed to load the data. Make sure that if you are using a CDROM that it is installed and mounted. Then: >run xload This invokes the XLOAD.001 script. After a few minutes you are left with the UV data file in AIPS: >ucat AIPS 1: Catalog on disk 2 AIPS 1: Cat Usid Mapname Class Seq Pt Last access Stat AIPS 1: 1 2179 BC123H .XIB1 . 1 UV 25-SEP-2002 11:53:47 The 'X' in the Class suggests that this file is temporary and identifies it as scrap when the UVCLEANUP procedure is run. Don't delete it yet! Note that the autoaips scripts shouldn't screw up any other data that may be on the disks. It will run recat: if this is a problem, re-run autoaips (step c) with a different . f. Determine flags: XAPRIORI leaves AIPS in a nice state -- typing >go starts SPFLG with reasonable parameters for you. Make sure the TV is ready. A quick dose of autoaips philosophy: Instead of flagging the data with any flagging tool, it is most convenient to keep the flags stored in the .x.N file. This allows future data reduction, with different models, for example, to be completely automated. Antenna based flagging is encouraged as most errors are antenna based. Some exploration is needed sometimes to determine which antenna is responsible for some RFI. SPFLG is a good way to explore the data and look for antenna based errors, but feel free to use your own! When a problem is identified, make an entry in the .x.N file of the form: flag1 = ante=5,0;bif=2;eif=2;bchan=25;echan=25 flag2 = ante=5,0;bif=4;eif=4;bchan=5;echan=5 Note: No spaces after the flag string starts. Also, the commands after the = sign are interpreted directly in AIPS, so any UVFLG adverbs can be specified. Remember the ; between commands and remember that the selection is reset (effectively restore 0) before each flag command, so it will have no memory of previous flag commands. Usually only about 4 commands are needed for the pulsar data sets. Sometimes as many as 15 may be needed, but this makes flagging the other in-beam and pulsar correlations much easier. In fact, a good flagging of any one data set (inbeam or pulsar) should suffice. g. Apply flags: Once the .x.N file is modified, rerun autoaips (Step c) with the same parameters as before. This will update the XAPRIORI and XUVFLG scripts. Now just run the updated XUVFLG script: >run xuvflg h. Determine the bandpass calibration: >run xbpass You will need to interactively clean the bandpass calibrator at this stage. 4 iterations of calib and imagr are performed. Don't accidently "stop cleaning" instead of "continue clean" when you have no clean components yet -- this would mean deleteing the .IB1 file and rerunning XAPRIORI. Note that XUVFLG would not need to be rerun as the flag commands are also stored in XAPRIORI. At the end, the bandpasses will be displayed for you. If you wish, you can run CLEANUP now to remove some temporary files. It is not needed and has a very small chance of clobbering your other files, so do so with caution. Reread item 6 above for more detail. i. Perform the phase referencing: >run xphase This creates single source files for the in-beam calibrator and the phase calibrators. Note that an in-beam single-source file is made for each phase calibrator. The naming of these files obeys this rule: . .SPLIT At this point, imager and calib have reasonable parameters set so you can start doing the imaging/astrometry that you wish to do. Reducing a second (or third...) correlation is now easy. It is the same for a second in-beam or a pulsar, so lets assume here it is the pulsar... j. Set up autoaips again: Copy the .x.N and .key file (and optionally the jplg* files and autoaips.save to speed things up) to the pulsar script directory $BC123APSR Make sure the pulsar data is in the data directory (if a CD, replace the in-beam CD with the pulsar CD, otherwise you are probably using a different source directory than for the in-beam -- make sure the data is there. Then run autoaips just as before (from the autoaips xterm): parallax<195>% autoaips BC123APSR psr 2 PSRDATA or if from the disk: parallax<195>% autoaips BC123APSR psr 2 PSRDATAPSR Make sure to use psr as argument #2 if this is the pulsar or ib2 if it is a second in-beam calibrator k. Use aips with the new scripts: You need to change the aips version to find the pulsar scripts. >version = 'BC123APSR' Load the data >run xload This creates the .XPSR class file. Perform initial calibration >run xapriori Creating the .PSR class file. This also applies the flags that were stored in the .x file that was copied to the pulsar directory. l. Copy the bandpass and phase calibration tables: Instead of redoing the bandpass and phase calibration, it can be copied over from the first pass in one quick and easy step: >run xcopy After this is run, the pulsar single source file is created and you are ready to do science.