Next: Requirements list
Up: E-ALFA Software
Previous: SDFITS notes
Discussion on detection algorithms
- Martha's questions to Jon and Erwin (23Apr03):
I have some questions for you about your extant software and how you do
things now. I can't find much on the public HIPASS site of enough specificity
to answer my questions, and how universally the
Barnes et al. paper applies.
- * What software platform do you use for image processing of HIJASS/HIPASS?
Do you stick within aips++ or export and work in something else?
- * What language do you code the detection algorithms in? Are they standalone?
- * What data format (SDFITs? Measurement set?) are the data in when the detection
- * Are there differences in the above within the HIJASS/HIPASS collaborations?
- From Erwin (24Apr03):
Some info on HIPASS and the software we used for processing and
analysing the data - The HIJASS software is a variation on the same
theme I believe, but the HIJASS people will be able to tell you.
> I have some questions for you about your extant software and how you do
> things now. I can't find much on the public HIPASS site of enough specificity
> to answer my questions, and how universally the Barnes et al. paper applies.
In principle, the
Barnes et al describes the procedure for the
pipe-line processing that we used for standard HIPASS. The raw data
comes in as single dish fits files, and is then processed in AIPS++
using their measurement sets scheme. Most of this has been written in
GLISH, the scripting language for AIPS++. So all the hard work is done
in AIPS++. This was a conscious decision made at the time, which was
possible because of the large support and development effort for AIPS++
made at ATNF at the time.
The end result were 3D fits cubes, which were then used for all subsequent
analysis. All galaxy detection wasd done in the image domain (except for
the wavelets deccribed below)
> * What software platform do you use for image processing of HIJASS/HIPASS?
> Do you stick within aips++ or export and work in something else?
Most of the processing is done outside AIPS++, generally with whatever
tools were available at the time. Given the exploratory and possibly
pioneering nature of the whole analysis things sort of developed, and
things were done for historical reasons that would now obviously be done
much better with the hindsight we have. The first attempt at galaxy
detection software was a Miriad script written by Virginia Killborn,
which was essentially a peak detection algoritm that find all x-sigma
peaks, eliminated double detections, and extracted a total spectrum.
Other efforts were made with a wavelet algorithm, written in GLISH if I
remember correctly, as well as a detection algorithm based on connected
pixels, developed in cardiff (Jon or Robert Minchin will be able to tell
more about it). The wavelets turned out to be a bit of a dead end, as
the student in question left, and it wasnt possible to disentangle the
black box he left. There also was not much overlap between lists
produced by wavelets and peak detection algorithms, leading us to have
some doubt about the wavelet method. It would still be worth exploring though.
One of the main findings is that is is very very diffcult to develop an
algorithm that is both complete and reliable. With the conventional peak
detection methods it is possible to create an almost complete 5sigma
peakflux catalog, but completeness at high reliability is difficult at
the 3 sigma level. Partly this is caused by the sheer number of pixels
in an all sky survey, so that there are a significant number of false
noise detections (even at 5 sigma we are talking about 10s to hundreds
in HIPASS). These have to be filtered out by a human, or cross
correlation with other catalogs. recently Melbourne has made a
herculanean effort to catalog all detections in HIPASS down to 3 sigma,
which meant that tens of thousands of detections had to be individually
checked, and even now the reliability is not extrememly high. This is
not something we want to do with EALFA. Furthermore for HIPASS the
analysis is complicated due to the non-gaussianity of the noise (and I
have no doubt that EALFA noise will be non-gaussian as well). HIPASS has
a larger number of high-sigma noise peaks than you'd expect on the basis
of gaussian noise.
So to get back to your question, most of the detection algorithms that
are being used in Australia for HIPASS are essentially modified existing
algorithms from Miriad or aips or whatever was handy at the time.
Jon is in a better position to tell about the Cardiff ones.
Unfortunately not much was published about the Australian HIPASS galaxy
finders. There is a little bit of information about them in
Kilborn et al (2002 AJ 124 690), but not at a useful level of detail.
The efforts made by Cardiff are probably best documented of all in
some of Jons papers.
Virginia Kilborn may be in the best position to give a summary of
developments on her galaxy finder.
Jon, can we get Virginia to write a one-page description of her program?
- From Jon (24Apr03):
I think Erwin answered most of your questions. The one remaining is on
automated galaxy detection. I think the only thing published is the one we
Davies et al. 2001, MNRAS, 328, 1151. As Erwin said this initially
relies on a peak detection followed by convolution with a matched filter. Robert
has developed this further (so hopefully after reading this he will tell you
what he has done, but I don't think anything is published). As Erwin says the
noise is none Gaussian so it is difficult to work out realistic detection
levels. This is a problem because for an HI survey you do not want to rely on
optical identifications, so either you set a high (very) S:N level or you have
to do lots of HI follow ups. There were some very interesting things in the
'noise' in the HIPASS data which I didn't follow up. There is far more positive
going noise than negative (I multiplied the spectra by -1). My guess is that
strong sources can be detected in spectra at many 10s of arc min away from the
source, so there are many faint signals in spectra. A detailed look at the noise
characteristics of HIPASS would be an interesting project. I had a very
interesting chat yesterday with one of our people who works on the detection of
signals in gravity wave data. I think they are much further advanced than we
are. They use a large template bank and fourier deconvolution - I think this
looks very promising (stand alone and written in C I think). I am in the process
of arranging a meeting with his group to see if we can carry out a few trial
runs using HI data. I will keep you informed.
- From Robert (24Apr03):
As Erwin and Jon have said, the primary selection relies on peak-flux
detection in a single spectrum at a time. After this selection a
matched-filter is used to improve reliability and get a first guess at the
velocity widths. The whole cube is examined spectrum by spectrum and a
catalogue constructed. This catalogued is then purged to remove duplicate
sources, keeping those with the best match to the template. The current
version of the finder is coded in F77 and is fairly much a stand-alone
Next: Requirements list
Up: E-ALFA Software
Previous: SDFITS notes
This page created and maintained by
Last modified: Thu Apr 24 21:54:40 EDT 2003