Visit the main TOGS Homepage
Latest Work: Text files describing the Spring 2007 TOGS observations and quickfile quality checks.
Also see the TOGS diagnostics page.
We also refined a few details about the datacubes that will be the major product from the TOGS project. Our tiling scheme changed in a way that significantly reduces the number of cubes we'll have to make, and the cube properties themselves are being standardized as well. It's all coming together. Once the stg0 redo is finished, we'll try merging in some basketweave data again. I think it would be best to do this with gsr2.3 code, since the xing suite of 2.2 takes up too much disk space in the xing/ area. Tonnes of new data are coming in too, and I'm keeping up with that quite well, in fact I'm including the endfinder program as part of my daily data check, and in principle I could stg0 the data right away and make it part of the larger dataset almost immediately. I should try to aim to have that in place by the time TOGS comes back from its short break (around March 18).
This morning the Spring '07 semester of ALFALFA/TOGS began, so for the next two weeks there is some overlap between the end of Fall '06 and the start of Spring '07. It shouldn't be too hard to keep it all straight though. Over the next week we need to figure out how best to analyse the fall 06 data. It makes sense to do it at Arecibo, and try to merge it with fall 05 + A2050.
I'm back from my little trip to Trinidad and Tobago. The TOGS schedule was rather light while I was gone (just one run). For the November 8 and 16 runs, RX 3 is bad (warm). By the November 22 run, rx 3 is back to normal, and the qckfiles are looking quite good in general. The fall 06 semester is almost over for ALFALFA - it finishes on December 12, and the spring 07 semester actually begins on December 5. So as with last December, there will be two A2010 runs for a few days (Dec 6-12 to be specific). Some AGES runs are scheduled closer to Christmas, so I'll have to figure out how to check those data while I'm in Ireland!
The load-xing-point finished, but I found a bug that prevented the *auto_l.sav file from being fixed properly, so after doing that I'm running xfit again, and boy are the gains coming out much nicer. Look at this figure as an example. It shows day 60 crossed with day 122. The white points are with the old spectra, not corrected for bad receiver, and the red points are the new ones. This should significantly improve the look of our cubes. With the new gsr code the spectra are fixed with spfix or spfix1, not with fixrx like I said on Saturday.
We continue to collect very good data for fall06 TOGS. The RFI has been especially low as of late. Analysis of the spring 06 data is also going well. The first set of cubes I made had some problems because of a bad receiver in the A2011 data we merged with TOGS, so I'm rerunning the crossing-point codes now to get better corrections. It's taking really long to run the lxw wrapper, since it has to run fixrx on almost every A2011 scan. Still hope to get some nicer datacubes before my Caribbean vacation that starts in a week.
Observations have good very smoothly throughout this month. Unfortunately on the night of October 18 no data was taken because the encoders at the telescope were having problems due to a lightning strike. The quality of the data coming in has been very good, however. There were some RFI issues a few nights, including picking up the Puntas Salinas radar, but the strong RFI spikes were far enough away from our science band that our HI data are mostly unaffected. We're also making good progress on merging the spring 06 semester of TOGS data with another GALFA-HI project in order to try and get some crossing-point calibration information. TOGS is now off for 10 days, starting up again on October 30. From November 9-22 there's only one TOGS run so I'm going to the Caribbean around then (not Arecibo)!
TOGS observations continue to come in steadily, and the quality of the data is good. We've successfully "retrofitted" the spring 06 data into gsr2.2 format (able to handle more than 100 scans for a project), and I've rerun the spcor program on it too. The next thing to do, I guess, is try to combine it with some basketweave data to get some crossing-point calibrations done. Josh and I will try to get the spring 06 data completed by the time the fall 06 observations are all done. This should be around mid-December.
Things have continued to go well for new TOGS observations with ALFALFA. We have a week off while Arecibo does some World Day experiments. Josh has made some nice data cubes from fall 05 data, and after the break I'll try working with version 2.2 of our gsr software.
I am at Arecibo once again. TOGS is going as smoothly as ever, though I've had to delete a fair amount of extraneous FITS files from /share/galfa because the spectrometer has been left running long after TOGS finishes on several occasions. Not a big deal, and the telescope operators are such great people it's not really worth complaining about such a trivial thing. I'm here to observe another GALFA-HI project, but it is somewhat relevant to TOGS because we are observing with GALFACTS as a sort of test run before their big survey, to which TOGS2 will be commensally attached. We ran a lot of tests to enable this work to happen, so it will probably help a great deal when the big surveys begin.
Last week TOGS turned one year old. Quite a milestone, and lots and lots of data to show for it. I decided to start new files to keep track of things, starting when the new A2010 semester (fall '06) started on August 11. My record of the data from year one of TOGS in two files: The Year One Summary and the TOGS Qckfile Summary. Since the last blog entry we've made tremendous progress in many areas. Josh successfully merged the TOGS data with another dataset observed with the basketweave mode, so the fall '05 data have much less stripiness going on. The spring '06 semester of ALFALFA finished on June 22, and it was 102 scans in size. The stg0 processing for 100 of those scans is complete, and was done at Berkeley instead of Arecibo. I ran spcor successfully too, making a few minor code changes that anticipate the imminent release of GSR2.2. So I'm making some test datacubes right now. There doesn't seem to be much GALFA-HI basketweave data intersecting the spring '06 sky, so we'll have to try and rectify that.
A couple AGES runs got done in July/August too, and things have gone more or less smoothly in all respects.
I was in Calgary for a couple weeks, but managed to keep up with TOGS most of the time. We've lost a few days of data through unfortunate circumstances. Since we're doing TOGS with 2 separate EALFA projects, the wrong startup file has been run a couple times (ie. /var/togs run instead of /var/ages), so we lost AGES data on May 13 and 20. AGES finished its latest run on May 21, and it's been ALFALFA since then, and it's gone rather well. Short outage on May 25, and on June 10 we couldn't restart the WAPPs with start_gpib, so we found we had to update our recovery process since the path to the script had changed without us knowing about it.
Our disk of data is getting full, so we're hoping to do something about that very soon. Lots of data is now backed up at SSL as well as San Diego, but we don't want to delete anything from Arecibo if at all possible. Josh is making progress on combining basketweave scans with TOGS data, so hopefully soon we'll see some nice streak-free datacubes.
Here's the latest summary of qckfiles
KD
Lately we've been running TOGS commensally with AGES. They seem to be looking at a part of the sky with very low HI signal, around 2 K peak brightness. In looking at the qckfiles, it seems that one of the HI peaks has an odd repeating signature, where the features has a strong, narrow component that comes and goes. I'm not convinced it's real. We also had problems with receiver 5 (beam 2, pol B) for a few days, where it gave a really nasty narrowband baseline. Rebooting galfa1 made that problem go away. mh files are now made automatically and stored in /share/galfa/galfamh/, so I don't have to generate those anymore. I still make the lsfs files, which is good since I sometimes need to recover them using special procedures.
Here's the latest summary of qckfiles
KD
Here's another qckfile summary to show how things have been going, which is to say more or less smoothly. I was at Arecibo earlier this month and while there, there was one day where I got to start TOGS myself since the TO had not begun it. It looks like very soon some data will get shipped to Berkeley, so I can try to clean up some space on /share/galfa. I managed to pass the task of making the TOGS datacubes to someone more suitable - thanks Steve! I'll still do the stg0, spcor and xing (when ready) reduction stages. When does the spring semester end for ALFALFA anyway?!!? There are some more AGES runs coming up starting on Saturday.
KD
Lots of TOGS data has been coming down the pipeline. All of it has looked good too, though the LSFS runs for the March 12 scan did not produce good qckfiles. I was able to use the cal file from the previous day, fortunately. I've been running STG0 calibration on the old data too. I'm up to the first week of September 2005. This creates about 10 Gb of corrected spectrum files per day, so I'm starting to take up a fair amount of space on /share/galfa. I even made some efforts toward making a datacube of the August 2005 observations. Click to see a TOGS datacube slice.
Here's a more recent qckfile summary while we're at it.
KD
The TOGS runs have been going quite smoothly, and the amount of data coming in is quite large. Last night I think they did nine hours. I am slowly doing the stage-zero data calibration on the old data, which creates a few Gb of new files for each day of observations. I don't want to fill up /share/galfa too quickly, so I'm going to weed out any fits files that don't need to be there, ie. files I've labelled as USELESS in my observation summary because GALSPECT was left running.
KD
TOGS started up again Feb. 15, and everything was going well until last night, when a cable failure made it impossible for GALSPECT to write data to /share/galfa. So, we lost a night but it's been fixed already. Good thing, because A2010 is about to ramp up bigtime. I think there's only three nights in the next month where TOGS won't be running. I'm ready!
KD
The last few AGES runs went more or less OK - had to recover more lsfs files though. I don't see AGES on the schedule well into March, so I'll look forward to more ALFALFA commensal observations starting Feb. 15. Here is the rundown on all TOGS qckfiles to date. I have deleted a few lsfs files from the lsfs directory at Arecibo. The observation summary lists those (look for the word REMOVED in caps). This afternoon I should be visiting Josh to plan the next stage of reduction.
Kevin
We had a good telecon this afternoon to plan out the next stage of data reduction, among other things. Once that gets going the amount of disk space we occupy on /share/galfa is going to increase dramatically. Last night's AGES run was an odd one. For both their smartf runs, they had to abort and restart, but the aborted run actually contained a full smartf run, so an lsfs file got made. However, the qckfiles made from those lsfs files were horrible. So I managed to recover the two good lsfs files for the cal runs immediately following the bad ones. As a result there are 4 lsfs files for AGES, only 2 off which are any good. I'll probably delete the bad ones eventually, but the timestamps for the good/bad pairs are only about 180 seconds apart, which is pretty cool.
Kevin
Back in Berkeley. Things are running fairly smoothly, and TOGS observations will go on a hiatus from January 22 - February 15. I hope during this hiatus I'll get to bring all this data farther along the reduction pipeline.
Kevin
Tomorrow I leave back for the Bay Area. It was a good, productive 2 weeks in Alberta. The AGES runs are still going well, and ALFALFA starts up again in a couple days. Here is the latest TOGS qckfiles inspection summary. I've made two simple IDL programs, agplot and tgplot, that help me plot the qckfiles super-fast.
Kevin
Snowy Calgary. AGES TOGS is going well. For ALFALFA I have had to recover the starting lsfs files using the special lsfs_shell procedure, where I have to determine the starting indx number, for each of the last two days. But to my delight it's worked perfectly for both days. The qckfiles are looking great, so the extra effort I need to put in has been worth it.
Kevin
I am in Calgary, Alberta, until Jan. 15. I started reducing 2006 TOGS data yesterday, so I'm almost caught up. Just making the qckfiles for this morning's ALFALFA run, then in the afternoon I'll inspect all the qckfiles. I had to recover one lsfs file using lsfs_shell, for the Jan. 4 ALFALFA run, but other than that everything seems to be going OK from the TOGS angle.
Kevin
TOGS is done for the year, starting again on January 3, 2006. The EALFA groups are still getting some WAPPs confusion errors, despite restarting the WAPPs before each run. A few lsfs files have been giving bad qckfiles as a result. All the qckfile inspection reports are summarized in this file.
We are using GSR version 2.0 to process the data. I was previously using the same software but it was not officially 2.0 yet.
Kevin
Last night the AGES group started their run with TOGS commensally observing with them. So files with the ".togs." descriptor are ALFALFA-related, while AGES-TOGS files will use the descriptor ".ages." to identify them. AGES are using Doppler corrections, so the LO frequency will drift around a little bit, unlike ALFALFA observations. So there are now two TOGS projects running. The program onefile is still under development...
Here is the latest qckfile inspection summary. The AGES log page will help me follow the progress of that project.
Kevin
In the last week we've improved the way the TOGS calibrations are done at Arecibo. We restart the WAPPs before each run, just like we do for other GALFA runs. Also this morning we started using version 2.1 of CIMA (the evolved version). Josh has made a new program called onefile.pro which I will use to make the corrected and calibrated spectra from the raw fits files.
Kevin
I've added a new page for diagnostics of TOGS data. I won't be updating it too regularly, except to give my impressions of all the plots I've stuck there. This page does NOT include qckfiles - just the other diagnostics that Carl's programs plot. I'll screen-grab some qckfiles next time I look at some.
Kevin
Here's the current version of the log file laying out the qckfile inspections I've done for TOGS. Much to my delight, several qckfiles that I thought showed useless data are actually pointing to a slightly less bad problem, which is that an lsfs file might be bad rather than the whole day's data. I normally use the first lsfs file as the calibration file when running simpred_ch, the program that makes the qckfiles. But sometimes I guess the levels aren't right at the start or something, because the second lsfs file gives much better quality qckfiles. I'll put some figures up to show what I mean (so far I've overwritten the bad ones, but I can recover the ugly ones easily enough). Anyway, this shows why we really like to have two smart frequency switching runs per night!
this entry by Kevin Douglas
Almost done looking at all the old qckfiles. There's a log file describing these that I will post when it's all done. Since many of these were done before the new baseline-fitting procedure came about, some of the problems may not be problems anymore. TOGS started up again last night after a 6-day hiatus, so I'm working on those data right now too. Snezana reported from AO that there were lots of overflow messages once the run got going, so I'll look at the spectra to see if they tell us anything...
this entry by Kevin Douglas
Eventful Thanksgiving weekend. I came in on Friday and Sunday to check up on TOGS data. For Nov 23-25, things are more or less fine; a little problem on the 24th in vncviewer but data are fine. Might have been some dormant gdiag processes left running. November 26 the ALFALFA people had wapps trouble, so they consider their night a bust. Still, GALSPECT was left running so we might be able to salvage some data. I looked at about half the qckfiles yesterday, and about half of those look alright. I'll have to work on a webpage summarizing which qckfiles contain good and bad data for each night. Making qckfiles for Nov. 27 run right now. Should be no problems - 2 lsfs files were made. ALFALFA people did have to abort & restart partway through, so I'll see if that affects TOGS at all.
this entry by Kevin Douglas
Last night there was no smart-frequency switching done at all. There was a problem setting levels, so some time was lost. Spectra were recorded eventually, so fits files from 0001 to 0012 have good data. The LSFS file from the previous day's data was used to make quickfiles, and those look pretty good (much the same as others lately). Carl has figured out how to recover the smartf from the night of Nov. 20, so I'll try and make the LSFS file for that later today. Also experimenting with zapft, which can be used to take out a baseline ripple in a narrowband spectrum. Also Carl has written a new program to do the baseline fitting (a new version of polycorr), so I'm trying that out on TOGS data from 09/13, which seems to have worse baselines than other days for some reason. So far the results look promising. I swear I'll start putting some figures up to illustrate what I'm talking about.
this entry by Kevin Douglas
Quickfiles are now made for all data from the start of TOGS to the end of October. It is known that the levels for TOGS have not been properly set from Oct. 21 to November 16. Levels on November 20 were back to normal, but we're having trouble extracting the proper LSFS spectra at the start of the run, since the smart frequency procedure was aborted and then restarted. For last night's data the lsfs run went smoothly AND levels were good, so quickfiles were made for those data this morning. Here's an example of a quickplot for Nov. 21 TOGS data. We see that receiver 4 has some weird waveform in the narrowband. It's been there since Nov. 15. I am making quickfiles for Nov. 20 using the Nov. 21 lsfs file for now, until we manage to make a proper lsfs file for Nov. 20.
When done, all quickfiles will be in the directory /share/galfa/qck at Arecibo. Before the end of the month the new machine at SSL (ewen) should be up, at which point I'll start copying data from Arecibo.
this entry by Kevin Douglas
First TOGS-blog entry. TOGS has now been running at Arecibo for 3 months, and with the imminent release of the Galfa Software Reduction (or whatever GSR stands for) package version 1.2, now is a good time to start churning out reduced data for TOGS.
The first two steps in the data reduction are pretty simple: generate the mh files and the lsfs files. The mh files are all currently in /share/galfa/mhmx/ at Arecibo, and the lsfs files are in /share/galfa/lsfs_kevin/togs/lsfs/. They'll probably reside somewhere else eventually. I am now generating "quickfiles" for all the data taken from August 17 to the end of October. That should be done later today.
For now we have a text file giving a brief rundown of the data for each day. Much more will be added as things progress.
this entry by Kevin Douglas
this page maintained by Kevin Douglas
to Kevin's Research page
