Question for Comair folks...

[General Lee]

What if the company HAD decided to just manually figure out who was where and then call you up and assign you a leg or two, or maybe a ferry flight, etc? That is what could have happened. Leaving on your own without talking to anyone is irresponsible.



Bye Bye--General Lee

 

[Ben Dover]

If Comair ran out of glycol, why didn't they just borrow some from Delta? If Delta was out, how did they get any flights in the air? Not trying to arm chair quarterback, just wondering why that was such a show stopper.

 

[surplus1]

You know, when I started this thread I really just wanted to know what happened. At least more specifically than the computers 'went down.' I was curious because it seemed unfathomable to me that all the flights would be cancelled. I certainly didn't mean to open the door for guys bickering over weather forecasts, regs, and professionalism vs being a sucker. I've managed to gleam some info, but I still don't understand why people didn't simply adhere to the schedules they had. Could the company not dispatch? Were they unable to adjust to any delays and it all snowballed?

ps- let's take it easy on each other!

Your question was a reasonable one and you didn't get an answer. I don't know all the answers but I'll attempt a more direct explanation.

With respect to the storm -

Management was very aware that the storm was approaching. There is a physical limit to the quantity of deicing fluid that can be stored. That limit is the capacity of the tanks, which are the property of the airport, not the airline. They were full to maximum capacity before the storm arrived.

A senior CMR manager attempted to hold some flights at out stations so that they would no come to CVG and be trapped by the storm. He was overruled by his bosses.

The storm was more intense than forecast; much more. They type of precip required much more extensive deicing than anyone could have anticipated. As the supply began to deplete, more was ordered and was enroute by truck, which is the only way to get it there. Due to the heavy ice, the entire supply was exhausted within 24 hours. The delivery truck convoy was stopped by the State police, which shut down the hiway system.

That is why the supply was exhausted. It was not due to poor planning.

The Computer System Failure -

The tracking and crew scheduling systems are two different software programs from the same vendor that interface with each other. They are both older technology that needs replacement. There is no "mirror" backup system for these. The back up system is a manual system that has not been used in years. Most schedulers are trained in it, but most have never had to use it except for a few "old timers".

Due to the very large number of flight changes, reroutes, aircraft groundings (unable to deice) etc., the systems were overloaded and both failed. That causes the loss of all data.

To operate a manual system, you must know where you are before you can begin. When flying an aircraft, you can't navigate to any point if you do not already know where you are. To start a manual system, you must also know where you are. When all data was lost, there was no starting point. Therefore, the only thing that could be done was to cancel all flights, manually locate all aircraft and crews, and start from scratch. That's what we're doing while the IT's are restoring the computer systems.

There should have been a "mirror" backup system which stores data on separate servers and updates itself every 5 minutes. However, that costs a fortune. It may surprise you (and others) but most airlines, including the very largest, do not have "mirror" backup systems.

In fact I believe that UAL is the only large carrier that does. American now has a partial mirror, why ... because they had a failure resulting from an improper entry that canceled many flights at two major airlines (AA and U). In August, NWA canceled 120 flights because of a comparatively minor failure in their computer system. In May, Delta canceled 40 flights because of a minor failure in their dispatch computer. To my knowledge, no one has had to deal with a total failure of their tracking and scheduling systems, in the middle of a major blizzard. That's what Comair is dealing with.

After the system failure, the phone calls from and to crews were so high in volume that they overloaded the telephone system and the schedulers, preventing calls from being made or received for long periods due to busy signals.

I'm not making any excuses, just trying to tell you what happened and why.

Those who complain about not being called at their hotels (in this thread) couldn't be called because no one knew who was in what hotel. That's what happens when you lose all data. You don't know where anything or anyone is. All of our scheduling has been exclusively computerized for years. There are no "paper" schedules other than a print out of your line at the beginning of the month. When trips are disrupted and rerouted due to massive weather systems, those original printouts become useless. The trip you were originally assigned doesn't even exist any more.

Crews are not the only thing that has to be tracked. We also have to track the airplanes themselves and there limitations, maint., etc., and the flights and their routings and re-routings. Both the tracking and the crew scheduling systems must operate together. The main failure here is not the crew scheduling system, its the tracking system.

The majority of those ranting in this thread about the justification of "just going home" without company contact, are not Comair pilots. I'm not saying that some CMR pilots didn't leave and go home. What I am saying is that most of the ravers in this thread about why that's OK are not Comair pilots. We do have some people with that type of attitude, but most of our people are not that way and will help the Company in time of crisis. Like all other airlines, we also have an a$$hole factor. Don't judge the book by its cover, especially in an anonymous forum where we don't know who is who.

On Sunday we were able to operate about 175 flights. Today the target was 60% (I don't know if we made it). It's a big problem and most of our people are doing everything humanly possible to recover as soon as possible. We are doing our best to help our customers and each other in the crisis.

The major flaw in all of this is the absence of a "mirrored" backup system. Why don't we have one? Because it is extraordinarily expensive and no one anticipated such a massive failure happening all at once. Perhaps we should have, but we didn't. The only consolation is that most other airlines, much larger than us, don't have such sophisticated backup systems either. After this debacle they will probably get them. I bet we will too.

Maybe George Bush should have ordered all airlines grounded before 4 of them were highjacked and flown into those buildings on 9/11. But, he didn't. Maybe the earthquake in Indonesia should have been forseen and Tsunami warning systems installed in the Indian Ocean to avoid the waves that haven't occurred in the last 100 years. They weren't. Maybe the US should have such a system in the Atlantic Ocean, just in case there's a big wave there. We don't.

Comair and its people regret this failure more than anyone. We also regret that we did not anticipate it and therefore avoid it. All we can do now is our best to fix it after the fact. We are doing that.

Meanwhile, we sincerely hope that all the other airlines with their perfect systems do not experience a similar failure, before they can install their own mirror backups and fail-safe programs. Who knows, maybe after Mineta conducts his "investigation" the government can create a new Agency that will prevent future computer failures. I'm sure it will be just as good as the TSA.

I hope that gives you a better idea of what happened. I hope it also gives the naysayers a chance to achieve their own perfect systems. We will learn from our mistake, and I hope others do as well.

One final word --- it's all the pilots fault. If we would just agree to fly for free, the Company would be able to afford a mirror backup system.

 

[General Lee]

Ben Dover,


The airport can only hold 7 days worth of glycol. Since the temps were very low and the snow enormous, it took more to de-ice each plane. The other glycol trucks (18 wheelers transporting it to the airport) were stuck on the highways, and the Highway Patrol would not allow anyone except emergency vehicles to travel on the slippery roads. That is the reason. Also, the airline (Delta or Comair) did not own the glycol, the airport did. The airport was responsible for getting the glycol.



Surplus1,

The pilots at Comair were not at fault, it was a system failure. But, I would like to know more about why some pilots went awol while at the out stations. That is an interesting story.


Bye Bye--General Lee

 

[typhoonpilot]

Excellent explanation Surplus.

What I would like to know is why the papers, at least the ones I'm reading over here, only say Comair and give no clue to the reader that Comair is affiliated with Delta ? Granted, the failure was at Comair, but the people affected were mostly Delta customers who bought Delta tickets to fly to a former Delta destination and had to endure one or more legs on a replacement jet.

Typhoonpilot

 

[michael707767]

The pilots at Comair were not at fault, it was a system failure. But, I would like to know more about why some pilots went awol while at the out stations. That is an interesting story.

For what it's worth, I don't disagree with the pilots who bailed. Over Christmas, no contact from the company, no way to get through to the company, I would not have waited forever. There comes a point when I would have bailed too.
Michael

 

[lowecur]

Retribution Conspirator???

Guess who's not getting any new a/c? RJET gets contract for 170's. DL putting pressure on Comair to reduce costs in line with other outsourced carriers. Is the word out that Comair will slowly shrink into oblivion over the next 5 years?

It wouldn't take much for some unhappy middle or upper mgt type that is computer savy to sabotage a system he or she knows can fail very easy. Bring in the FBI.

 

[Well-hung Start]

......but the people affected were mostly Delta customers who bought Delta tickets to fly to a former Delta destination......

Typhoonpilot

You're wrong here:

The people affected were ALL Delta customers. Not mostly......ALL.

Comair does not sell tickets.. It's ALL Delta. Try buying a ticket on Comair....cannot be done. Comair does not have the capability to sell even a single seat on a single flight. It's not even a Comair flight designator......it's Delta.

Comair is a lowest-bidder, bare-bones alter-ego of Delta Airlines and Delta is cleverly dissociating its name with that of Comair during this crisis. Clever.


Delta.

Big D.

 

[HarleyGuy]

Suprlus: thank you very much for the excellent answer. And the amusing ending. In the AF we called this sort of thing 'blood money.' (I assume that term is pretty common). Couldn't get something fixed until it killed someone. Maybe this is a big enough event that the airlines will get backups. We'll see....

 

[Ben Dover]

Ben Dover,


The airport can only hold 7 days worth of glycol. Since the temps were very low and the snow enormous, it took more to de-ice each plane. The other glycol trucks (18 wheelers transporting it to the airport) were stuck on the highways, and the Highway Patrol would not allow anyone except emergency vehicles to travel on the slippery roads. That is the reason. Also, the airline (Delta or Comair) did not own the glycol, the airport did. The airport was responsible for getting the glycol.
Bye Bye--General Lee

General,
So if the airport ran out of glycol how did Delta mainline or anyone else get off the ground? Or maybe they didn't, I don't know. Again, not trying to criticize, just trying to figure the system out.

 

[Slim]

Here are some facts which support the comments of Surplus1 and others...

1. I was working 22-24 December. I was one of the fortunate few to get home on the 24th. It was a result of staying put and being proactive rather than reactive (that probably comes from my military time.).

2. While in CVG on the 22nd, it took two hours to get from the gate to the deice pads. My flight to ORF blocked at 4:01. Scheduled block time was 1:25. Does this give anyone here an idea of the weather intensity when the storm hit? It's no wonder glycol supplies were depleted so rapidly.

3. In BOS, 24 December, we couldn't contact scheduling to confirm our schedule beyond our first flight. However, dispatch was sending releases and my crew was scheduled to go to RDU. So, we went. Upon arrival in RDU, the release showed us going back to BOS. So we went to BOS.

4. In BOS, rather than scatter to the winds as some did, we were persistently trying to contact scheduling. We finally got through and were scheduled to DH on DL back to CVG. That flight cancelled. Scheduling was planning to junior man us into Christmas in BOS. While discussing our options, a crew came into Ops looking for their airplane to reposition back to CVG. God works in mysterious ways! We called scheduling (three different people calling three different numbers!) and scheduled ourselves on the ferry flight. With scheduling's knowledge and concurrence, we returned to CVG.

One thing I noticed over the past few days is scheduling seems to rely a lot on EPIC, an internet information service for Comair employees, to "notify" crews of changes in schedules. Unfortunately, we crew members seem to perpetuate that practice. It's a convenience which scheduling seems to think every crew member has 24 hour access to. If our hotel didn't have Internet access for its guests, we would have been in limbo. I don't carry a laptop computer anymore, following my retirment from the National Guard.

My wife, a former computer programmer, noticed SBS has a limit of 32,000 monthly transactions. Since it's a five digit limitation, she questioned why the limit isn't 99,999!

The next few weeks and months should be very interesting!

Fly safe!

 

[chperplt]

Ben

Very few flights left CVG on the 22nd and not many more on the 23rd. This is Comair, Delta, US Express, CO Express and United Express included.

I sat in the back of a DL flight trying to DH for nearly 4 hours before returning to the gate and CX the flight. Airplanes couldn't be deiced. When they finished and did an inspection, they had to start over.. I haven't seen freezing rain that bad in a long time.

 

[Snoopy58]

My wife, a former computer programmer, noticed SBS has a limit of 32,000 monthly transactions. Since it's a five digit limitation, she questioned why the limit isn't 99,999!

Because computers don't work in decimal as their native language. Ask her how many digits 32k is in binary.

 

[surplus1]

Guess who's not getting any new a/c? RJET gets contract for 170's. DL putting pressure on Comair to reduce costs in line with other outsourced carriers. Is the word out that Comair will slowly shrink into oblivion over the next 5 years?

RJET's EMB 170 contract has nothing to do with Comair's operating costs, and everything to do with Delta's lack of capital. When you need a little and the wife you married is pregnant, you turn to the local hooker, especially when she's haveing a sale on spread legs and likes the color of your wheels. You might get an STD but you gamble anyway.

It wouldn't take much for some unhappy middle or upper mgt type that is computer savy to sabotage a system he or she knows can fail very easy. Bring in the FBI.

Source considered; reply unnecessary.

 

[SWA/FO]

yeah, ask her that!

 

[lowecur]

RJET's EMB 170 contract has nothing to do with Comair's operating costs, and everything to do with Delta's lack of capital. What a silly remark. GE is lending DL $500M, and you think they would have a problem leasing them 170's? Leasing can be arranged with no up-front costs. It's just DCI w/o is too expensive to operate. GE is supposed to lend UAIR $180M plus lease them 31 E-jets if certain caveats are met by 1/15/05. Plain and simple Surplus, Comair & ASA aren't getting anything until they can come close to matching RJET in costs. In fact it wouldn't surprise me in the least if they sold off all of their owned CRJ's to SKYW. I see SKYW has been picking some up from FLYI. I know SKYW wants that UAX deal. When you need a little and the wife you married is pregnant, you turn to the local hooker, especially when she's haveing a sale on spread legs and likes the color of your wheels. You might get an STD but you gamble anyway.

If your assumption is true, then once DL gets pension relief (one form or another in early 2005), then DL will have made most of the changes necessary to be viable for the future, and get complete access to the credit markets. When that happens, I still say DCI gets zero without concessions.

 

[LarryinTN]

The limit of an 8-bit integer is 32,767. Include zero and you can track 32,768 changes with an 8-bit integer index.

 

[DaveJ]

Uh, I think you mean 16 bit. The max unsigned number for an 8 bit integer is 256 (127 signed). Its 32768 (65736 unsigned) for 16 bit integers.

 

[Oakum_Boy]

ASA and Comair are running on fumes. It was only a matter of time before this happened. Mainline people must realize that no regional subsidiary receives as much support and attention as you do. There is little to zero support for us crews. WE ARE ON OUR OWN.

Whether it be dispatch, maintenance control, or whatever. These airlines are staffed by morons. The flight crew are the only way these disasters continue to operate. Pilots may not be as experienced as those at mainline, but we are certainly experienced with the daily cluster/goat rope that is reality for anyone working for a regional airline.The bottom line is that we are a low-cost mainline alternative, with crappy service, late departures and dirty cabins.

You can't blame pilots for heading home. There is no nobility in sticking it out when management continues to treat us like dirt. I would have been out of there for sure. I remember one Delta Captain who was convinced that we used their dispatch. Yeah, I wish...

 

[Slug]

OK, post 47 and 48 hurt my head.


Slug

 

[Slim]

Slug,

Those responses (47 and 48) had to be translated into English for me. According to my wife, if the language is FORTRAN, the 32,000 transaction limitation makes more sense.

Here's to no more headaches!

Fly safe!

 

[FlyComAirJets]

Freaking FORTRAN

FORTRAN, FORmula TRANslation. Learned it in college a gazillion years ago on a computer the size of twenty Buicks. Used a keypunch to hole IBM cards that fed into a hopper that was "read" into the machine and then compiled. Any fatal errors and it was back to the beginning.

Stone age technology, reminds me of one of those BC cartoon characters chiseling into a stone tablet.

 

[freightdogfred]

What reg is that? The 24 hours off evern seven days can be spent in a hotel, as far as the regs are concerned.

Mebbe in the freightdog world..... 1 in 7 for sked service means at base, free of duty..

 

[DaveJ]

OK, one more try. By the way, the computer language used makes no diference at all. Doesn't matter if it's FORTRAN, C, Basic, or assembly language. The issue is whether the programmer wrote the program to correctly handle arithmetic overflow errors.

Computers use binary arithmetic. All digits are 1s and 0s. An integer number can be signed or unsigned (unsigned means positive numbers only, signed numbers can be positive or negative).

Computers process data in chunks of bits.
A byte is 8 bits which can represent 0 to 256 (unsigned) or -127 to 127 (signed).
A word is 16 bits which can represent 0 to 65536 (unsigned) or -32767 to 32767 (signed)
A longword is 32 bits which can represent 0 to 4294967295 (unsigned) or -2147483647 to 2147483647 (signed).

So it looks like the program Comair was using was tracking change events with a 16 bit signed integer value. The number of changes hit the limit (32767) and then went 1 over. Whenever that happens, the positive number now overflows to a negative number and the computer throws a numeric overflow exception. If the programmer who wrote the program never anticipated that an overflow would happen, he probably did not code for the event and the program crashed (apparently without saving any data).

 

[surplus1]

For the computer experts:

The program(s) that failed is 1970's technology written mostly in 80 column FORTRAN. They run on an IBM server with AIX Unix.

I really can't say much more than that.

 

[chperplt]

USA today is running an article that says Comair will have the software replaces within a few months. The article seems to place blame on Delta because of the low IT budget it has allocated.

We're back to 100% today and all will be right with the world... unless GL or 737 have anything to say about it!

 

[surplus1]

Oakum Boy,

I'm curious. Where do you work?

 

[surplus1]

I still say DCI gets zero without concessions.

Apparently you don't even know what "DCI" consists of. So much for the credibility of your remarks.

 

[lowecur]

Apparently you don't even know what "DCI" consists of. So much for the credibility of your remarks.

Weak reply. Did you read my reply where I mentioned DCI w/o? Try sticking to the topic.

 

[737 Pylt]

Whether it be dispatch, maintenance control, or whatever. These airlines are staffed by morons.

To a certain extent you're right. You get what you pay for. If the airline is going to pay $5/hr schedulers, then these kinds of screw ups are going to happen, no matter what company you work for!
737