WebAdmin

I’m starting to believe that the above statement is an oxymoron. I’m probably not allowed to go into detail regarding my current security woes. However, I will generally ponder… For instance, why would a system only afew months out of date on OS patches be vulnerable to an attack? How can a vulnerability that has almost no resources online so thoroughly hose a machine? Why aren’t bells going off in some deep underground security bunker when a SINGLE binary file can gain ROOT access on any system of a particular OS release???

Moving on… why are security departments so reactive? When a server is attacked/hacked or cracked (shudup, rhyme ftw), they bust down doors and swoop in like an attentive FEMA. However, I’d much rather be hassled by them about my security holes before they are so thoroughly exploited. The word is proactive. Think, Secret Service, not FBI. Don’t break up the drug cartel operating down the street, let’s prevent the assassination from ever taking place, ya know? Dive into that bullet! Take one for the team!

Apologies to the four people that read this, I know it has devolved into some sort of pretentious foto-blog. I only post them because my pictures are awesome (oops, there’s that pretension)… that and work is boring and monotonous at this point. One has to have an outlet, no?

It’s nice to be home again! I just got back from a CMS conference in Atlanta, which I was expecting to boring as hell, but actually turned out very interesting. It’s not hard to figure out what CMS I’m speaking about, but I haven’t decided if I want to disclose the name of it because I’m not sure how it’s going to work out. Everyone knows I’m opinionated, however I don’t want this to turn into a rant blog and as it is, I will wait until I’m sure where this thing is going before I commentate.

While I was in Atlanta, I stayed at the Sheraton Buckhead. It was going well until Tuesday afternoon, when the power went out and stayed out until well past midnight. No air conditioning, no Internet…. well, that’s basically all I give a damn about! Luckily, the bar was open, so my good friend and co-worker Jonathan and I hung out playing some DS and drinking a very good micro-brew.

A good friend of mine told me that the sushi in Atlanta was really good. This didn’t seem logical to me, considering Georgia’s further from the coast than most cities you’d consider as good places to get sushi (San Francisco for instance). But what the heck… I went to a place called Hashiguchi Junior, and it was amazing. If you’re ever in the Atlanta area, you should definitely try it! My recommendation: try the White Tuna.

One Vote

So how many times do you get to vote in an election? Well duh, one person, one vote. This concept is lost on the Texas legislature, who ignore the principle and instead play a modified game of musical chairs, where the winner gets their law passed. Trust me, this video will majorly piss you off.

Peace out ya’ll.

Yeah, I know. Sorry. I’ve been busy load testing, as previously posted. I also got burgled, and have been dealing with insurance. It seems there’s a crime wave going on. I’m still sitting on glass in my car, as I can’t seem to find every piece.

We launched our beta website finally. I can’t link to it, because it’s a “private beta”. We don’t want too much traffic of course. But, considering I work for the University of Houston, I don’t think you’ll find it hard to locate the beta site if you’re persistent.

The MySQL Conference has made a “Call for Papers”. I’m considering sending something, but I haven’t decided what to speak about. When I figure it out, I’ll share it here for feedback.

Well, have a good week everyone. More on the beta website later.

We all hate web server load testing. You take everything you’ve worked hard on, and you abuse it until it crumbles. Having taken several cardiac stress tests, I have been the focus of real load testing. I have crumbled.

Done at the busiest time in a project, like when hardware has just been installed or right before the implementation of a new website, it’s failure will certainly guarantee missed deadlines and unscheduled rework.

Many people use apache’s ab (apache benchmark). It comes with apache web server, and it works well. What many people don’t think about is that it is NOT a browser. I know, shocking. What I mean is, if you point ab at index.html, set it to 100 concurrent sessions with a 5000 query assault, my old 486 web server will take that without blinking. The ab application isn’t a browser, and won’t request any of the other content associated with the page. No images, no css, nada. You end up load testing a static page, without loading what a real world user would. This is not reality!

The other popular test application is siege, which is similar to ab, but has more configuration options. My favorite is the URLs file that can be called via a config file (.siegerc), which allows the tester to give a list of all of the real content that is associated with a single page. Once this list is prepared, you can run the test, then divide the results by the total number of content in the URLs file.

This weekend I will (hopefully) also be giving the apache test application called flood a try as well. It is configured via an XML file, and seems very versatile. The apache foundation’s apps like to use XML anywhere they can. Although it looks promising, the last release is from 2002, and there is very little documentation since it is technically still in beta.

One last note on load testing. Be aware of where you’re performing your load testing from. If you’re testing, let’s say, a major university’s web system, doing the test from within the local network is a good idea. Saturating an inbound net connection is a good way to get on the bad side of network administrators. Leave network testing to them, and be sure to let them know when you plan to test before you start. You don’t want to surprise people with mass quantities of traffic. If you look like a Denial of Service attack, you’ll probably be treated like one.

Wow it’s been a long week. Thank goodness I have tomorrow off.

My crontab exploded. It really sucked, since many of our process rely on it -_- So I thought I’d document alittle about some cron subtleties, especially since they aren’t too well documented in other places.

First, new golden rule:
Don’t let regular users have a cron!
Because if they write bad scripts, they WILL cause the cron to break.

For those that don’t know, the cron in a UNIX machine is a service that runs applications in a users crontab (schedule of programs that need to be run) based on the time. Every minute it checks all of the crontabs on the system, and sees if anything needs to be run.

What I didn’t know, is that cron has a max queue. If that max is hit, it will first attempt to reschedule the jobs, but will continue to do them in order. Most cron setups don’t set a maximum, so the default is used, which is 100. That’s alot of processes, so it’s usually not a problem.

But when there is a problem, like if a program that is cron’d fails to close, and is set to run every minute, it will saturate that queue rather quickly. This is unfortunate, because it’s very hard to figure out that it’s the cron that is the problem. In my case, news of scripts not running slowly trickled in, but manually running the script cleared the problem, and is easily shrugged off. “Must have been a script error first time, oh well!”.

You can set the queue max in the queuedefs file (usually in the /etc/cron.d directory. This is a rather cryptic file that looks like this:

#
#
a.4j1n
b.2j2n90w

It’s pretty simple though. The “a” line stands for at. An at runs several jobs immediately, and with a normal priority. 4j means 4 jobs can run simultaniously, and 1n means they will have a “nice” of 1. A nice is a priority number, and 1 usually stands for “normal”.

The b stands for batch. It runs commands 2 at a time with a nice of 2. The 90w means that it waits 90 seconds in between every 2 jobs. It’s alot like an at job, but runs with less priority, and less process at a time.

These lines can usually be ignored, because the defaults work fine, and are seldom changed.

Back to the max queue. You can add a c line at the bottom, and it stands for “count” (or as the man says, the default queue). This is the maximum amount of commands that can be running by cron at any time.

I left my cron the way it is though. The default is more than enough, and while crontab dieing seems bad, a server that seizes because too many processes get started is MUCH worse IMHO. Monitoring is good, if you have important crons that you want to know are being run. Unfortunately, you usually use cron to run check scripts If you have a monitoring suite, you can use the log file to check for errors. The error that accompanies the queue maxing out is: “! c queue max run limit reached“. It’s owned by root, so keep that in mind. I wouldn’t recommend changing it’s permissions, it’s a highly sensitive file, and can give a hacker the info he needs to compromise a system VERY quickly.

Of course, you could always setup cron to send you and email every minute.

If they stop coming, well…. you know it’s not working!

Fun fact, there’s 525,600 minutes in a year. Cheers!

I’ll admit, I hate Java. It’s not that it’s bad in principle. Conceptually, it’s genius. Write one piece of code, run it on anything! Cellphone, Sparc, x86, RISC, Commodore. It just doesn’t work out that way in reality. First, even with the virtual machine, the compatibility doesn’t work. Actually, it’s more annoying because it’s suppose to be portable!

Also, as I believe John Carmack once said, Java takes a nice program and makes it run 100 times slower than it should. We just got a new CMS at work, which will remain unnamed for now. It does everything I could ask a CMS to do. Actually, it kinda resembles an uber CVS system. It manages versions of files, then pushes the files out to a location; whether that’s a webserver on another network via FTP, or the local file system.

However, it runs on JBoss in Tomcat which I have seen scary things from in the past. JBoss apps tend to need a metric crap load of RAM (that’s a new form of measure btw), and runs painfully slow on even our fastest systems. Only time will tell whether my fears are warranted. Luckily, it only handles the content, and none of the actually serving of said content (Apache FTW!).

I’m feeling very sickly at the moment, but I figure doing a post will help distract me.

Afew weeks back I began attempting to get Apache webserver to take a request, sans the extension, and give me back a php file with the same name (e.g. /admin displays /admin.php). I was puzzled to find that my rewrite wasn’t working as it should in some Apache configurations in my development environment.

Enter MultiViews!

“MultiViews” is an Apache option that was created mainly to deal with content negotiations where a server has several language specific versions of content available to web users (index.en.html, index.fr.html, etc). More info on MultiViews can be found at the Apache content negotiation pages. I use Apache 1.3, so please keep that in mind when reading this. I’ll explain why I use Apache 1.3 in a later post.  You must specify the MultiViews directive in the Options line of your httpd.conf or .htaccess file for it to function.

But what is it?

The above documentation sucks, so I will sum up what it says. MultiViews was created so when it is on, Apache will serve content that is derived from the request, based on the information the browser passes, and if the specified files are present. So, MultiViews takes a logical request and returns the contents of a physical file that is closest to what is requested.

What does it do?

If you request index.html, but specify EN as your language, Apache will return index.en.html to your browser. And if you request /admin (and assuming the directory doesn’t exist), Apache will return admin.html if it exists. Why? Because admin.html is of the html/text type (.html) and starts with admin. Your browser by default expects html/text. It basically guesses which file you are requesting based on the browser request. Keep in mind that this is based on the extension of the file and it’s associated type. Apache uses the left most extension first to make a match (index.en.html). For my purposes, this isn’t important, since I only care about one extension.

Yay! Wait, no yay. That’s not what I wanted. I wanted it to return admin.php!! MultiViews returns admin.html!

Luckily, if admin.html doesn’t exist, and admin.php does, it will return the latter. This is where I noticed something wrong while writing my rewrite. I won’t get specific, but I wrote my rewrite in such a way that it would deliver PHP before HTML (admin.php over admin.html, if both exist), but it was returning the HTML first, because MultiViews was governing what was returned, not my rewrite. It took me days to figure out MultiViews was the culprit… Ok, now we celebrate.

Known Issue!

Wait! A major issue has been documented around the net about MultiViews. Allegedly, if you use MultiViews to display PHP without an extension, the server will incorrectly report the mime type as application/x-httpd-php. Normally, the php parser returns text/html, and Apache usually reports it as such. However, the server will not return content to a browser that does not support that type; it just throws a server error, the dreaded error 406 - Not accepted. Well, most browsers will accept the application/x-httpd-php type. The Googlebot browser however, will -not- accept the application/x-httpd-php type. I would venture to say that not being indexed by Google is unacceptable to most people.

Is It Fixed Though?

After experimenting (by passing a spoofed agent string to my Apache webserver) I have discovered that this issue seems to have been resolved in my version of Apache 1.3 and in our testing version of Apache 2.0. So when I request a PHP file without an extension, Apache returns the correct html/text type. I can’t find any documentation that specifically mentions it being fixed, so I will move forward very cautiously. Luckily, we have a Google Appliance at work, so we can point it at a test system and see what happens, just to be sure. I would hate to redo the entire UH website with this technique, and then lose our ability to index the site… *shiver*

This gave me such a headache, I hope collecting this information in one place will help others. I’ll try to find out officially if this issue has been resolved in the latest versions of Apache.

Bad Documentation, Hard to Implement

While I’m on this subject, MultiViews’ documentation is painfully incomplete, and the option is also unfriendly and not configurable. I can’t tell it to prioritize PHP content over HTML, or atleast it’s not documented.

I shouldn’t be complaining, lest I forget that this webserver is -free- and open sourced, so if I want to correct MultiViews’ shortfalls bad enough, I should shutup and checkout some code from the Apache repository.</rant>