Hi again everyone. Looks like I've broken my promise of weekly blog entries by 41 days or so. Oh well, nobody's perfect (except Angela, of course). Anyway, you know those times when you get so involved in your work that you do things like forget to eat or go home? It's been like that. (That and, unfortunately, I can't really bring myself to believe that posting to a web site is really part of what I get paid to do.)

This week I've been busy thinking about how to remove radio frequency interference (RFI) from the Astropulse database. As you may recall, Astropulse has a big problem with radar. There's a radar blanking system at the observatory, but that only tracks the FAA radar. There are about 6 other radar systems that we see in our data, and the radar blanking system does nothing about them. Matt's working on a way to remove the rest of them, but until then we need to figure out a way to work around them.

This is a waterfall plot of 618,813 pulses we've gotten back from Astropulse version 5.03 (as of yesterday). Rather than the usual spectrogram type waterfall which would be a plot of frequency versus time, this is a plot of dispersion measure (DM) versus time. DM in this case is how much sooner the high frequencies arrived than the low frequencies did. A dispersion measure of 15000 means that the 1.42125GHz radiation arrived 15000 samples (6 milliseconds) before the 1.41875GHz radiation did. I've compressed the time axis to get rid of the spaces between different observations.



The regions that are radar contaminated are pretty obvious. Anyplace that doesn't look smooth and random is radar. The easiest way is just to throw out any workunit where Astropulse says it detects multiple pulses that saturate the amplifiers. That's shown in the next plot.



That does a pretty good job of removing the worst times. But it leave behind some vertical RFI features, one at a DM of -14,200, another at a DM of 5,600, another at a DM of 11,000, etc. But we have other ways to remove RFI. RFI tends to show up at specific dispersion measures. One way is if the signal stays around when you are looking at another part of the sky. So I threw together a program (written in IDL) that grabs 10,000 pulses from around the same time, but from different parts of the sky from a signal I want to check, and then calculates the probability than that many signals would occur at that dispersion measure due to random noise. If the probability is less than 1 in a thousand, I consider it to be RFI. If there were no RFI, it would still remove one out of every thousand signals, but that's an acceptable loss. The next image is the signals that this algorithm determined to be RFI (25% of the total) the first time it was run.



And here's what it didn't think was RFI...



It's got some problems left in the radar region, but it got rid of a few of the leftovers that the previous method left. Best of all, I can run it multiple times on what didn't get flagged the first time. The second time I ran it, it removed 4% more of the signals. I'm running it a third time right now, but I haven't made images of those runs.

If we use both methods, we've got a pretty good system. Here's what's left over if I combine the first run of the RFI rejecter with rejecting the signals from workunits where there were multiple amplifier saturation events.



Now all that's left in the short term is to port the algorithm to C++ and have it write the results back into the database. I'd also like to know what that splat is at an index of 70 and a DM of 300 is. But that's a story for another time...

I'll try to make my next post in less than a week to make up for my missing time.

---

@SETIEric@qoto.org (Mastodon)

Thank you very much for so interesting and illustrative post.
If you could continue the story later it would be wonderful!

. . . Thanks for the Post Eric and, as well the Beautiful Artworks Sir!

> note: 'reminds' me of when another Scientist 'used-to' Post 'ere [reference to Kevin Douglas PhD w/ GALFA . . .]

---

BOINC Wiki . . .

Science Status Page . . .

Very interesting post, Eric.
Thanks you.

73 & clear skies from Bruno IK2WQA - brmoret_at_libero.it
Messier 13 Astronomical Observatory, Tradate (Italy)
Founder SETI@home Berkeley's Staff Friends Club
Founder SETI ITALIA Team G. Cocconi

Oh well, nobody's perfect (except Angela, of course).

"True-er" words were never said...

I'd say that kenzieB's comment of "bias it against signal strength. That is, any
signal above x dB has to be terrestrial and thus, can be ignored" was
a rather astute observation. Slightly modified version of "throw out any
workunit where Astropulse says it detects multiple pulses that saturate the amplifiers".

If only it was that simple......

From my radio transmitter experience and the small amount of knowledge I have of Aricebo's setup, I offer the following scenario:-

I woul guess that the interefering signal, although very strong, is probably ousite of the frequency band of interest and hence cannot be directly detected by Seti in the data. Instead what would be present in the data would be intermodulation products (signals) caused be the Radar (or other) signals saturating (overloading the inputs of) the amplifiers. These products could appear anywhere within the search frequency band and at any amplitude from below noise level to max recordable values, and would vary with any change in frequency or amplitude of the interfering signal.

I hope this gives an insight into the massive headache this interference causes for Seti.

Bruce
EDIT: Oh.. I forgot to mention there could be hundreds, or even thousands of intermod products generated simultaneously.

Chill out Boss!

But I am chilled out. :)

My post was only to help those who have little or no understanding of the problem gain a little insight into how complex it really is, and how difficult to eliminate.

virtual = not real

Ouch ... I've been exposed. :(

Oh well, nobody's perfect (except Angela, of course).

"True-er" words were never said...

. . . so, what you were both sayin' [changes name BAcK to nobody] ;)))

. . . watChes as Eric steps in and 'clarifies' and then 're-clarifies' everything



'. . . there's light @ the end of this tunnel' the Berkeley Feed exclaims


:>]

---

BOINC Wiki . . .

Science Status Page . . .

i have a question about seti mybee it is dumb but, one of the things you always need is hard drives. why does seti not use peoples spare hd space the same as they use spare prossecor time? or do they? there is alot of spare hd space. i have two 20 gig drives in my machine i do not even use. i know 20gig is a drop in the bucket but 20 gig times all your users?

---

i have a question about seti mybee it is dumb but, one of the things you always need is hard drives. why does seti not use peoples spare hd space the same as they use spare prossecor time? or do they? there is alot of spare hd space. i have two 20 gig drives in my machine i do not even use. i know 20gig is a drop in the bucket but 20 gig times all your users?

That is an application of Boinc that has been talked about a few times.

I guess it is rather low priority compared to other development focus.

Then again... If anyone can put some code together... With checks/security for data integrity and so on...

Welcome to the forums. Try a look in on "Number Crunching".

Keep searchin,
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

i have a question about seti mybee it is dumb but, one of the things you always need is hard drives. why does seti not use peoples spare hd space the same as they use spare prossecor time? or do they? there is alot of spare hd space. i have two 20 gig drives in my machine i do not even use. i know 20gig is a drop in the bucket but 20 gig times all your users?

I personally, would NEVER allow ANYONE access to my hard drives for ANY reason. You would be seriously asking for trouble. What a ridiculous idea! It's obvious why it would be a 'low priority'.

---

. . . Welcome to Posting Sorceress - How's Rico doin' btw

i have a question about seti mybee it is dumb but, one of the things you always need is hard drives. why does seti not use peoples spare hd space the same as they use spare prossecor time? or do they? there is alot of spare hd space. i have two 20 gig drives in my machine i do not even use. i know 20gig is a drop in the bucket but 20 gig times all your users?

I personally, would NEVER allow ANYONE access to my hard drives for ANY reason. You would be seriously asking for trouble. What a ridiculous idea! It's obvious why it would be a 'low priority'.

---

BOINC Wiki . . .

Science Status Page . . .

i have a question about seti mybee it is dumb but, one of the things you always need is hard drives. why does seti not use peoples spare hd space the same as they use spare prossecor time? or do they? there is alot of spare hd space. i have two 20 gig drives in my machine i do not even use. i know 20gig is a drop in the bucket but 20 gig times all your users?

Just to throw my 2 cents in with the other replies...it would take a significant amount of bandwidth to transfer the files to users' systems plus the possibility of data loss (user quits, system dies, etc.) makes it just not worth the attempt.

You already are alowing them acsess to your hard drive to run there software. what do you think you download and upload. do you think that just goes into thin air? think b4 you talk

---

you could use double or triple redundant saves. and require a connection of atleast 5mps conection speed

---

I personally, would NEVER allow ANYONE access to my hard drives for ANY reason. You would be seriously asking for trouble. What a ridiculous idea! ...

Do you run on a Windows OS & use proprietary software?

Ever read the "EULA"s?

(My understanding is that Microsoft users agree to Microsoft doing anything it wishes with your system and your files/data and all with utter impunity. It is well reported that Microsoft takes details of all the software on your system, and likely other details...)

Good first post!

Welcome to the forums,

Keep searchin',
Martin

---

See new freedom: Mageia Linux
Take a look for yourself: Linux Format
The Future is what We all make IT (GPLv3)

i have a question about seti mybee it is dumb but, one of the things you always need is hard drives. why does seti not use peoples spare hd space the same as they use spare prossecor time? or do they? there is alot of spare hd space. i have two 20 gig drives in my machine i do not even use. i know 20gig is a drop in the bucket but 20 gig times all your users?

User's hard drives wouldn't be suitable for most of the project's needs simply because there are delays involved with internet transfers. It would be impossible to have the active databases distributed to users, for instance. The difference between lab computers interconnected with gigabit ethernet and what can be done over the internet is huge.

As offsite storage of backup records, user's disk space could be a good idea. It would be another BOINC application, of the non compute intensive kind. Data could be delivered to hosts like a 10 MB or larger workunit, the application would simply report at intervals by trickle-up that it still had the data and be given a small amount of credit. From time to time the project might tell the host it no longer needed that data, and send some newer data to replace it. Then if the project ever needed that backup data it would be sent back and some more credit given. The same data would be given to several hosts, maybe with geographic diversity ensured by checking the host offset from GMT time. That kind of distributed offsite backup should be very robust. Still, it would be an additional load on the limited internet bandwidth the project has available, additional work for the project servers, and significant work for the staff to develop.

                                                                Joe

Ever read the "EULA"s?

(My understanding is that Microsoft users agree to Microsoft doing anything it wishes with your system and your files/data and all with utter impunity. It is well reported that Microsoft takes details of all the software on your system, and likely other details...)

Mostly as a pre-emptive strike against any claims that Microsoft evilly and maliciously changed something on a users' system or accessed something proprietary.

Of course, once they grant themselves that right, it certainly removes one barrier to evilly and maliciously changing something on a users' system.

---

i have a question about seti mybee it is dumb but, one of the things you always need is hard drives. why does seti not use peoples spare hd space the same as they use spare prossecor time? or do they? there is alot of spare hd space. i have two 20 gig drives in my machine i do not even use. i know 20gig is a drop in the bucket but 20 gig times all your users?

Just to throw my 2 cents in with the other replies...it would take a significant amount of bandwidth to transfer the files to users' systems plus the possibility of data loss (user quits, system dies, etc.) makes it just not worth the attempt.

I'm w/Randy on this one, could cause more trouble than it's worth. Not to mention people getting upset over rac scores if one of my gerbils died and did'nt validate someone else's work.

---

Nobody is nobody. Everyone has something to offer

I personally, would NEVER allow ANYONE access to my hard drives for ANY reason. You would be seriously asking for trouble. What a ridiculous idea! It's obvious why it would be a 'low priority'.]

You already gave "access" to your hard drive. Or where do you think SETI WUs and apps are stored while you process them? Where else other than the hard disk could that be?

---

Contribute to the Wiki!