The ISP shut it down.  All that is left is the relay that I’ve been running forever.  This ISP was on the “Tor Friendly” list (which I altered the wiki and noted it accordingly) and allowed IRC servers.  The reason for shutting it down? One php script probe (see previous post).  One.  The boilerplate emails stating what tor was, and how nothing is logged, blah blah blah, weren’t an acceptable reason for the probe (WTF?).  I guess I am responsible for /all/ traffic that goes in and out of my box per their AUP.. Which is IRC friendly….. Yeah…  So I cancelled my service, and are looking elsewhere.

Now riddle me this.  Why would an ISP enjoy common-carrier liability protection from say, usenet or IRC, but not a tor exit node?  It doesn’t make sense to me.  I set this exit node up 100% with the guidelines setup by the tor-project (short of forming an LLC for liability protection) with regard to dns entries with webpages, a dedicated email for abuse complaints, the whole 9 yards.  Still, it was shut down.

Yeah, you may say to just run another relay, but heres the problem with tor.  Unless you’re going to a hidden node within TOR, the entire project is only as fast as the combined speed as its exit nodes.  Sure, you can hop onto tor at 400MB/sec, but the net worth of the entire network is how fast you can get OUT.  I can bounce around servers until I’m blue in the face, does me no good if I cant get to where I need to go because the exit nodes are all swamped.

The whole situation is shitty.  Its been down for a week to let my self cool off rather than right an angry post naming the ISP in question.  Oh, the tor exit node’s DNS WAS tor-exit-readme.virtual-adept.net, just in case you were curious..

All this happening when the tor project could really use a 10 mb/sec exit node for Egypt.

Friend of mine and I were in starbucks enjoying overpriced coffee and using their way too slow free (unencrypted) wireless network.  I was checking my email (via SSL-IMAP) and surfing the internet through tor.  It was a little slow, but thats life.  My friend asked me why I was using tor since all I was doing was checking my email and surfing slashdot and other mindless webpages.

I put my laptop’s wifi into promisc mode and ran tcpdump.  Some idiot was using some filesharing program that I was too lazy to look up the port, another idiot was checking his gmail, another idiot was surfing youtube.  Few people were on facebook, etc.  I told him that I didn’t care to have people here know that I run a mail server, where my mail server is, or what I do on it.  I dont want people to know what im surfing.  Its none of their business.  ALL of my traffic does is hit a tor entrance and disappear (to them).  He understood, i wish more people understood.

So I decided to man-up and run a tor exit node.  Figure I use tor everywhere where there is a public wireless network, its the least I can do to give back.  I’ve ran a relay for years without any problems, so I figure that as long as I follow the documentation on properly running an exit node, use a stripped down ExitPolicy, and bookmark the boilerplate abuse templates I should be fine.  Day 2 of running an exit-node I get my first abuse complaint.

Massive hack?  Stolen credit card?  Letter from the FBI? Something tasty and awesome?

No, some “admin” saying his “IDS system caught” one request to /foo/fjl93rjs9fj/xploitable.php and decides to write this massive abuse email on how this server is trying to “hack” his website.  Really? Fucking really?  In this day of age of nothing but bots and zombies you are going to waste both my and my ISP’s time with something as stupid as this? My webserver gets hundreds of these a day, and I have NEVER EVER sent off an email to an ISP about it.  Its called running a public webserver on the internet.  Don’t like it? Unplug your server and save us all your leet admin IDS skillz.

What makes things even better, is that the IP address that he’s bitching about is mapped to tor-exit-readme.virtual-adept.net.  If you go to that URL, is a webpage saying that this is a tor exit node, what tor is, and how its just a traffic relay.

Now if you’ll excuse me, I’m going to write 1000 abuse emails for every invalid URL request that came into my server this week.

Random segfaults with vdelivermail that look like this:

vdelivermail[3675]: segfault at 0 ip 00007f4164e470b0 sp 00007fff1a370528 error 4 in libc-2.7.so[7f4164dcc000+14a000]

Solution:

In ~vdelivermail/etc:

vusaged.conf

Log:
Level = 1;
Socket:
Filename = /tmp/vusaged.sock;
UID = vpopmail;
GID = vchkpw;
Client timeout = 5;
Poll timeout = 1;
Detect client timeout = 4;
Queue:
Workers = 10;
Max queue size = 1000;
Polling:
Use Maildir++ format = True;
Directory minimum poll time = 120;
Count directory entry size = False;
Age Factor = 1;

vusagec.conf

Global:
Disable = 1;
Server:
Filename = /tmp/vusaged.sock;
Timeout = 1;

Restart vusaged and your problems should be fixed, at least it worked for me.  Why this bug managed to get into a stable version I have no idea, but hopefully this will save you some headaches should this be a production machine.

I got to walk around the vast empty InReach building.  I knew where all of my desks were throughout the almost 10 years I worked there.  Looked at the tiny pieces of CAT-5 jumpers that I wired by hand well over 10 years ago.  My eyes started to tear up to realize that I’ll probably never ever see the inside of that building again.  I spent most of my teenage years confined within that small building working on servers, putting out fires, dealing with shit that now-days I could of never deal with.  Remembering the laughs, the long nights, the sitting at my desk wondering how in the fuck I was going to fix this server (but yet always did), the walks to McDonalds.  A ton of people who read this (or my LJ) worked there and grew up there with me.  That faculity saw the birth and the move of DeadJournal and housed zillions of bytes of data that I could never replace.

I unracked my one server, and followed Mike over to Pac-West.  Considering that over 10 years ago VAdept (this server) started out in the PacWest colo was sorta ironic.  I could still hear the sounds of the Ascend modem boxes heating up the room like huge rackmount hair-dryers.  Visualizing the USR 5+U modem chassis with the HyperDSP cards (that could never work right).  The smell of a real colocation faculity, and how a few times I slept on that cold raised floor hearing the air woosh under it while waiting for an array to rebuild.  Remembering the time Andy formatted a web-server array by accident, then said “See ya!” and left me there to restore from tape (you dick, I still remember that.. ).  It was a very weird and awkward feeling to be there again.  Everything looked exactly how I remembered it to be.

So since I was there, I decided to help Mike rack the gear that he brought over from customers who didn’t care enough to drive down to move their own damn computers.  It was such a weird feeling, that its hard to describe what it was like to be sitting in that colocation again after so many years.  In a way its fitting that my server ended back up to where it was born, with real generator power, real Halon and temperature monitoring, real 24/7 NOC, a “real” facility.

But using the same rackmount screws, the same power cord, and the same piece of cat-5 I’ve used for years and years; I’ve managed to pocket a few handfuls of dirt from the grave of the InReach colo.  Really, my server is its legacy.

• No easy way to download all the entries into a format where the WordPress importer can understand
• No easy way to shove all of my LJ entries into one category.
• No security levels.  Hell, I’d be happy with setting everything to ‘private’ on wordpress but alas there is no way to do it.
• There isn’t a really clean way of doing this.  The wordpress LJ import plugin (so I’ve read) doesn’t do true XML parsing but is regex central, so like the jdump.pl xml output just makes the importer die.

Ideally there would be a plugin already written that would do all this for me, but searching gives me zip.

Anyhoo, about life:

So today I called the tile place.  B&W Tile (www.bwtile.com) out of Gardena.  Really nice people on the phone.  I just told them what my tile person told me and they faxed over a quote.  $1400 and that included tax and delivery.  Being shipped tomorrow, 14 boxes at 500 lbs.

I placed the order and we’ll see what comes in.

Hm, lets see how much FAIL AT&T can give me:

• http://att.net/blocks/ gives me a 404 error.  Way to go!
• My email server is on zero RBL’s and gets a clean bill of health from Spamhaus.
• Googling the error points me to: http://worldnet.att.net/general-info/521.html
• They have a link to a form there, its at http://www.att.net/general-info/bls_info/block_admin.html

Now here is where the snowball of stupdity just keeps on growing.  Now you have to paste in your error into the form box, but the form box is way too small to paste the entire message error in.  Furthermore it bitches if it sees any special characters (say, like < > which are PRESENT IN ITS FUCKING ERROR).  So in the “what changes have you made” section, I said that their error-message link was broken, the error pasting was too small, and i’m not on any RBL’s.

So if you’re an AT&T admin reading this, please fix your shit.  From one admin to another this looks really bad.  I know you’re a big badass telco/world-domination company, but not fixing this is just going to have my friend drop your service like a hooker with herpes and give their money to a company that has their shit together.

If you are interesting in helping, its 100% php/mysql backend and I’ll accept any help I can get. Its a bit rough to post publically right this very moment, but if you’re interested let me know and i’ll shove the SVN repo somewhere for everyone to nab once I get it somewhat stable.

Lets make a huge problem that in 5 years we will be able to make a “fix” and rake in the cash. Sounds like a Microsoft product.

Heres a temp solution before version 2.3 comes out that has a ton of anti-spam features built in. This solution assumes that you are using the short-url module which writes an .htaccess file with a bunch of mod_rewrite crap in it.
Put the following code at the very end of all the pregenerated stuff (but within the IfModule and /IfModule statements)

RewriteCond %{QUERY_STRING} .*AddComment.*$
RewriteRule ^.*$ - [F]

Basically all that does is returns a 403 Forbidden error with any URL that contains AddComment.  Yeah, its stupid.  At least this will tide you over since there really isn’t a good way to globally turn off all comments.  This isn’t tested very much so buyer-beware.

Since all the major vendors (and even the kernel developers) are blindsided by this (2/10/08 is a Sunday) someone puts out a hotfix.  So I apply said hotfix to my workstation, my internal file server, and a server in an undisclosed location that is NOT at my house.

Everything is fine, until the MOMENT I get into my car to go and do something that will take 4 hours.  Off goes the cellphone to let me know that all 3 machines have shit the bed.  Wonderful, just fucking wonderful.  Cue the terrible anxiety for the WHOLE damn 4 hour appointment.  I get home, just get to fixing the servers that have shit the bed when the pent up anxiety makes my dinner explode out of my ass in 10 min increments.

I read then to see that the hotfix is really hit and miss as to if it panics the kernel or not.  To make things even fucking better, I find a /proc option that will automatically reboot your server if it locks up due to kernel panic.  Go fucking me.