Josh Work Professional Organizations Trip Reports Conference Report: 2001 LISA

The following document is intended as the general trip report for Josh Simon at the 15th Systems Administration Conference (LISA 2001) in San Diego, CA from December 2-7, 2001.

Saturday, December 1

Woke up at 4am Chicago time to catch the 5am shuttle to O'Hare airport because United, in its infinite wisdom, decided to move my 9:30am flight to 8:10am. We get to the airport without incident, check in the baggage (though I used the self-service lines it still took 5 minutes to get an attendant to hand me the printed luggage routing tag. Cleared security fairly quickly; everything electronic had to be removed from bags so they could be scanned in the X-ray machine independently, and all coats had to go through the X-ray machine as well as hand luggage. Found the gate (with over an hour to go before boarding began), grabbed a quick breakfast from Cinnabon, and eventually got aboard.

Believe it or not, we departed early, landed early, and arrived early in San Diego. The flight had very little turbulence, no screaming babies, reasonable airline breakfast food, and a decent-enough movie (Pay It Forward). Unfortunately, the San Diego ground crew took 30 minutes to begin unloading the luggage. Even so, I made it to the hotel, checked in, and was more or less unpacked by 11:30am. Elapsed time, 9.5 hours.

Had lunch with Esther FIlderman, but since the SAGE Executive Committee was in session and not breaking for lunch, none of the other folks who'd arrived already, like Dave Parter and Trey Harris, could join us. But we had a decent enough lunch at the Sunshine Deli and Terrace restaurant by the hottub and heated pool.

After lunch, Esther and I went to start doing the setup for tutorial notes handouts. We opened boxes and arranged the piles of Sunday Monday and Tuesday notes and wound up with entertaining gaps where cancelled and never printed tutorials would have gone had they existed. After things were set up, a group of us gathered by the pool (since the hot tub was closed for cleaning), schmoozed until sunset, then went inside when it got too cold.

Registration opened early, around 4:30. I worked handing out tutorial notes for a while, but we had enough staff there so I went to help out at registration. Handed out registration packets and helped answer people's wireless networking questions until registration closed at 8pm, then went with AEleen Frisch and David Blank-Edelman to dinner at Kelly's Steakhouse on the convention center grounds. Good food, good conversation, though the fresh strawberry shortcake was neither fresh (the berries were obviously frozen and processed) nor shortcake (they used poundcake).

After dinner we adjourned to the Presidential Lanai Suite for some well-deserved drinking. This theme will probably repeat itself over the course of the week.

Sunday, December 2

Sunday I worked at the Registration desk for the most part. Nothing particularly earth-shattering happened otherwise. Except perhaps for an exchange with a couple of the student slaves:

Student 1: I didn't know San Diego was so close to another country.
Me: Well, you know what's said about Americans and geography.
Student 2: Yeah, we're right next to Canada!

Sunday night was the annual "formal attire requested" birthday party for Tom Limoncelli and Cat Okita. It was also Peg Schafer's birthday and another Lanai Suite drinking bash so, to save energy and avoid cliquishness, we held all three parties in the same place: The Presidential Lanai Suite. At least three gentlemen wore tuxedos and several others had a jacket or a tie or both, and several ladies were in evening dresses of various styles (including a Renaissence-style one with a matching feathered hat). We sang Happy Birthday, ate cake, schmoozed, and did some more well-deserved drinking.

Monday, December 3

Monday I attended the MetaLISA workshop, which is about managing systems administrators. After introductions (with the usual who you are, where you work, and what you wanted to get out of the workshop), we discussed the common topics.

We first discussed motivation and retention, trying to answer the question of how you provide motivation to help retain your quality personnel. We decided that providing a good work environment without major stressors would be better than just throwing money (salary) at the problem, that authority and responsibility should both be well defined, and that resources have to be made available to handle problems.

Next we discussed the different types of people. Some system administrators are of the "work 9 to 5, get a check, leave work at work" type, whereas others are of the "computers are my life so I play on them at home too" variety. One manager organized his group so the former group was given the trouble-ticket queue processing and the latter group was given more of the infrastructure and hard install problems.

We then discussed the career path issues. Several companies now have multiple paths and levels, such as Team Lead, Project Lead, and Assistant Manager, each with appropriate and well-defined levels of expectations, evaluation scores or results, experience, requirements, and so forth. Providing different levels of responsibilities, independence, authority, and even money (base pay increase) on a path for both technical and management types, junior to senior, seems to work well. Even better is when there are well-defined criteria for promotion and lateral transfers between tracks. Remember, however, to provide allowances for exceptions or case-by-case waivers in your written policies.

Next we covered professionalism. Some people lie on their resumes. Some people don't have the dress code covered and don't wear the right clothing (suit or t-shirt) to an interview or to the job itself, and don't do so even when informed to do so. Some people don't understand the concept of punctuality. Some people don't know how to be tactful, or to provide the right level of information, or even to say "I don't know" to the customer. One topic of discussion was how to educate these people on how to improve these skills. Information sharing, such as email lists, databases, and even IRC channels, helps teams share knowledge, cross-train people, and can provide a way to let everyone contribute. Admitting when youre wrong builds trust for when you're positive you're right. Encouraging people to ask for help can work, but so can offering help and asking people if they need help. However, the insecure may not respond or take you up on the offer. For these people, if you present a situation as a "show me what you did so I can learn," it may help. Don't use killing statements (such as "you're wrong") but ask leading questions ("what if").

Next we looked at determining what information is important (to share) and what is not (to keep political fallout from the team)? One technique is to have a staff meeting and say "here's the important stuff" and let folks leave if they don't care about the politics. Getting folks to realize "best" isn't always "right," or that politics can override the right technical basis, is necessary. The team needs to be aware that there are politics even if they don't know the details. However, email is often not the best medium for this; in-person and telephone contact may be a better (or at least good in addition to) way to contact and inform people. Also, some people may not have a framework to put the details in; giving them the framework and answering their questions is good. Some people, though, just don't care about the political issues. Sometimes, having face-time in meetings with your people and the lord high political muckety-muck may be useful.

Many people can follow a checklist and don't have problem-solving skills. How do you teach them to acquire the skills? Problem-solving is linked to curiosity and background and experience. Teaching people skills is important. Using child-raising techniques, such as brainstorming with a timer, may be helpful. Again, you have to be careful to ask leading questions and not use killing statements that make the other person defensive. People need to remember to look at the big picture so they would make the right big-picture decisions and so questions of direction get addred within the group. Also, the problem and scope need to be explicitly defined, because that sets some limits. Finally the instruction or detail level of the recipient may be relevant; instructions to senior people may be much less detailed than instructions to juniors.

The next major discussion topic was the balancing act between technical and management responsibilities and tasks. Some of the tricks include allowing the people who report to you assuming you're still technical, knowing that the theory can be as good as the technical details, keeping yourself informed about major issues, and using one day a week as a technical day for working on small projects. Also, if you only rise to the point of comfort, whether that's team lead, division lead, project manager, company head, or whatever, you may be better able to find your balancing point. One of the problems is trusting the people to whom you hand your pet projects off to do the "right thing" with them.

Finally we discussed moving from an ad-horoup to a more procedural-based group. Formalizing processes by documention. Document not only how things work, but also why the decisions were made. Pairing people, one to explain and one to write, can work well. Starting with something like script is a good start. Having a cheat-sheet or template can be very helpful. But you have to practice what you preach; document things yourself so your people will. Also, make documenting a requirement for the performance review.

After the workshop a group of us headed over to Old Town for a (touristy) Mexican dinner at the Old Town Café. This was followed by some hot tubbing (in the "maximum occupancy 10" tub where we once again managed to get 35 people in at once comfortably), and partying again in the Presidential Lanai Suite. There was some amusement when the wet horny broads showed up.

Tuesday, December 4

Tuesday was the Advanced Topics Workshop, ably hosted and moderated once again by Adam Moskowitz and co-piloted by Rob Kolstad. The 26 of us went around the room doing introductions (who we are, what we do, what our last project was, and what the most important lesson from 2001 was). The introductions generated interesting questions and topics for discussions: Random opinions, the "undo" command for sysadmins, hot tools, surprises from the past year, and our annual predictions.

Random opinions

We went through some of the topics of interest from last year and mentioned our opinions. People are indeed using SANs and NAS, since they're well suited to specific problems (such as archiving, Fortune 1000 companies, and so on). However they're not being used for general file services, mainly because the FibreChannel implementation is too expensive for general use.

We also discussed the centralization/decentralization pendulum which seems to be moving back towards centralization. Perhaps condensing is a better term, since places are condensing locations for their hardware and personnel but still keeping some geographic distance between them. Centralizing administrative functions is different than actual physical centralization, since (to use the SAN/NAS model), users don't care if the disk is local or across the continent as long as the performance is unaffected.

We're moving towards more of an ASP model within a given environment, be it company or infrastructure. The ASP model works well between divisions within an organization but not as well between different organizations, primarily due to trust issues.

The events of September 11th caused a shift in the thinking of some of the tight-fisted financial staff. They now realize how integral computing is to business, so colocation and backups are now more important.

The next major topic was mobility. Without mobility today's commonplace high-speed network infrastructures and reliable file servers make a lot of system administration fairly easy... workstations can be built from images or automated installation processes and all mutable data lives on centralized file servers where it's easy to back up and manage.

But mobility changes all that. Mutable data has to be local to the end-point (laptop, etc.), we can't expect network connectivity to be high-speed, and we have to be able to deal with connections over insecure networks. We have to deal with a host of security issues, find new ways of ensuring data availability, and be able to provide the needed services of various levels of network quality.

Mobility is becoming increasingly important... there are now many organizations where most end-points are mobile platforms. But IT infrastructures have not yet caught up to this changing reality. To deal with this we will have to abandon our traditional (and previously successful) modes of thinking and use technologies that involve disconnected operation, mobile IP, synchronization, transparent data encryption, and so on.

Wireless computing has changed our behavior; 70% of us in the ATW are on laptops. Our expectations seem to be that we're approaching ubiquitous computing; of those using laptops, about 2/3 use them to access remote services (mail, web, files) and 1/3 use them as the centralized storage point. This leads to the intrusion of mini-environments into your own macro-environment: Managing laptops, which can move from administrative domain to administrative domain (and pick up and distribute viruses and whatnot), and keeping them from screwing up your environment is a hard problem.

Recovery-Oriented Computing

Recovery-Oriented Computing (or ROC) is targeted to services. A PowerPoint presentation is available, along with information at http://www.cs.berkeley.edu/~pattrsn/.

The goals are ACME — Availabvility, Change, Maintainability, and Evoluntionary growth — instead of performance (which is what we've looked for in the last 15 years). And we're not doing that well.

One of the aspects is not just to get real data to improve reliability but to measure reliability and availability. Making the system administration tasks have an Undo function may be helpful. Think about the three Rs: Rewind (go back in time), Repair (fix error), and Redo (move forward again). We're looking to recover at the service level, not just at the server (hardware or component) level.

Right now the thought is to build a sample (prototype) email system as a starting point.

What about security breaches (intrusion detection)? Something similar can be done; this kind of technology would be good. You could roll back to before the intrusion, install the filter or preventative mechanism or whatever, then roll the good stuff back in again.

Simply changing (fixing, simplifying, etc.) the interface is insufficient. Work does need to be done on SA recovery interfaces but this is beyond the scope of the ROC project.

Hot Tools in Use Today or Coming Soon

Next we discussed the new tools, technologies, ideas, or paradigm we're investigating or using. The list included new IP telephony products; tricks for ssh and CVS; wireless networking; integration and aggregation of alarm, monitoring, and administrative functionality with automation; reducing information replication; load balancing; anomoly detection; miniaturization; mirroring network storage for high-speed failover; VMware; MacOS X; Java; and Perl 6. The list also included business problems as opposed to technology problems.

One side discussion was about programming languages. Some people like Java, others like C#. Java is the new COBol in that it's the new business language but not a system language. Some debate ensued, with no conclusion, about whether to teach C, C++, Java, or even Scheme first.

Surprises from the Past Year

Several people mentioned surprises they'd had in the past year. This list includes Cygwin, the PC Weasel, the dearth of middle-men in the DSL/POP/ISP markets, and the number of people running wireless networks without any security.

Predictions

We went through our list of predictions from last year's ATW and we were remarkably accurate for a change. Of our 20 predictions we were right on 13 of them (65%), up from 47% right in 1999.

Our predictions for 2002, with the number of people who agreed with each one in parentheses, are:

Evening activities

After the workshop I joined the deadbeef contingent, so called because the malloc() system call sets its value to the hex value 0xDEADBEEF when undefined. A bunch of us go out for a really good steak dinner once per conference; this year it was at Bully's, renowned as one of the best places in San Diego for their prime rib. We took over one of the smaller rooms and ate and drank and ate and ate some more. I was a bad boy and had some onion soup, the full (22 ounce) bone-in prime rib with baked potato, and mud pie for dessert. (Geoff Halprin also got the full-cut prime rib but couldn't finish it. Wimp.)

After dinner we rolled back to the hotel for yet another evening of drinking at the Presidential Lanai Suite. Luckily (from my point of view) they still had the cognac so I had a nightcap before heading off to bed.

Wednesday, December 5

Session 1: Announcements and Keynote Address

Announcements

The first session started with the traditional announcements from the Program Chair, Mark Burgess. Mark began with the following:

Mark then thanked everyone involved and announced the best paper awards. Since only one student paper was accepted the awards were:

David Parter, SAGE President, then spoke about recent events. He noted that the SAGE Secretary, Trey Harris, had recently posted emails to the sage-members list about filling the vacancies on the SAGE Executive and responding to the USENIX Board of Directors' actions of mid-November. SAGE-specific events at the conference include thommunity meeting (Wednesday 7-8pm), the Certification BOF (Wednesday 8-9pm), the USENIX Board of Directors meeting (Wednesday 9-10pm), the SAGE Locals BOF (Wednesday 9-10pm), and the restarting of chigrp (the Chicago SAGE local group, Thursday 9-10pm).

David then announced the 2001 SAGE Award for outstanding achievement: Hal Pomerancz, for his past efforts (including co-chairing LISA, serving on the USENIX Board, simultaneously serving on BayLISA (in San Francisco) and BackBayLISA (in Boston), his Perl Practicum series for ;login:, and his 15 years of leadership in systems, network, and security administration. Hal, accepting the award, thanked all his mentors and teachers and reminded everyone that "You never learn something so well as when you have to teach it."

 

Hello, Parter!

This past summer, Eric, a non-USENIX part of the Madison crowd, created the Hello Parter shirt. Some of us thought it'd be funny if we arranged, without his foreknowledge, to all wear them to the opening session on Wednesday during LISA. So, with some culling of email addresses from a mailing list, we arranged it. Unfortunately, as happens so often, real life got in the way and only about half a dozen of us managed to have the shirts at the conference. But it still surprised and amused him. (The "Hello, Parter!" shout from the audience when he started his talk didn't hurt, either.)

Mark Burgess returned to note the SAGE Certification web site (available at http://www.sagecert.org/) was up, the IETF was seeking feedback on SNMP MIBs in a BOF session, next year's LISA chair would be Alva Couch, and there was a program change:

Finally, Mark introduced our keynote speaker, science-fiction author Greg Bear.

Keynote

Greg Bear, author of over 30 books from Blood Music to his newest book, Vitals, spoke on the subject of "Slime versus Silicon." He applied a biological model of competition and cooperation to computing. Biological machines average being right half the time. "Randomness" is meaningless in biological systems.

Both science fiction fans and technologists are curious about things in a childlike way. They don't necessarily care about what's cool or fashion and fall below most of society's radar.

Bacteria can cooperate to form big neural networks. Lone bacteria have to cooperate to survive. An individual cell is as complex as an aircraft factory. So what administers the cell? Historically biologists thought deoxyribonucleic acid (DNA) was a top-down master, but they've since come to realize that DNA changes a lot.

A virus is a method of communicating signals between nodes in a network. We need these to function. Biology is the administration — both administering and being administered by — of cells. The complexity in biological systems is a necessary kluge. Even at a genetic level, there's some technology, or direction, or decision; the genome is like an ecosystem in that it's all social. A gene, in order to get something done, has to cooperate with hundreds or thousands of other genes. The genome is a jungle of cooperation and competition — somewhat like systems administration.

Systems administration experience, including engineering methodologies, can inform biology. We now have the language to explain complex biosystems. Mathematics is a language but it doesn't describe biology. Computers move numbers around. Thinkers move resources around. Both are used to solve problems. This informs both biology and sociology. So why can't systems administrators use our skills to help biologists solve these "weird" non-numeric problems?

Greg Bear's advice is to go forth and study biology to better learn how to administer systems, because slime and bacteria have been doing it for millions of years. Biosystems are networks of users with different priorities.

Morning Break

During the morning break I chatted with Ellie Young on a variety of topics, including the 2002 and 2003 LISA program committees, my recent layoff, what I want in a job, and how we can move the USENIX and SAGE organizations forward given the events of the preceding month or so.

Session 2: Quiz Show Preparation I:
Qualifying Examination

After providing moral support to Mark Roth at his talk about service management, I worked with Rob Kolstad on the LISA Quiz Show qualifying exam then had lunch with Rob at the all-you-can-eat buffet at Sunshine Deli. We agreed to go through the questions (and test the software, which was being rewritten from the ground up) at 4pm.

Session 3: Invited Talk:
2001: A Communications Anniversary

I attended Peter Salus' talk about various milestones in communications. However, since someone else was assigned to do the writeup I didn't take notes. Aside from his voice problems (he was recovering from a cold), Peter was his usual entertaining speaker and the talk was quite enjoyable.

Session 4: Quiz Show Preparation II:
Question Vetting

Rob Kolstad and I went through and rewrote some of the questions that had given us trouble at USENIX. We removed duplicate categories (such as multiple "Monuments") and cleaned up some of the ambiguities. We left the rest for the quality assurance run on Thursday.

Evening activities

This evening a group of a dozen or so of us went to the on-campus steakhouse for dinner. At my end of the table, including Peter Salus and David Grieg, we talked about all sorts of things, including politics, humor, the state of the industry, the economy, and other topics now lost in the mists of memory.

After dinner, since the SAGE Community Meeting was standing room only and seemed just a wee bit hostile, I went to the convention center lobby and kibbitzed on a game of Settlers of Catan. After the game, it was off to the hottub for some soaking, off to the Presidential Lanai Suite for some drinking, and off to bed for some serious sleeping.

Thursday, December 6

Session 1: Invited Talk:
150/5,000 Years of (E-)Commerce: History Repeats Itself Again

I attended Dan Klein's talk about how history is again repeating itself in terms of commerce applications — what didn't work and was forced out of the market 5,000 years ago still doesn't work in an e-commerce web-based model. However, since someone else was assigned to do the writeup I didn't take notes.

Session 2: Guru-Is-In Session:
Writing Papers for USENIX Refereed Tracks

This morning, in part to give Lee Damon (coordinator) a break and in part because I wanted to attend the talk, I volunteered to introduce Tom Limoncelli at his guru session on writing papers for LISA. Lee must have been desperate for gurus for this slot since Tom's never had a single paper accepted to a USENIX-sponsored conference: he's always been accepted in doubles. Either two papers, or a paper and a talk, at LISA (twice) and NETA (the once).

Tom started why noting why publishing a paper was good. It helps the community and it starts to change your career (allowing for both peer and management recognition, and providing ammunition when your boss needs to justify your next raise).

How does one start writing a paper? The advice here is to write what you know. Are you doing anything to make your life easier? Automating a task? Writing a cool tool? Working on a neat project? Providing a case study, whether positive ("Here's what we did and it worked") or negative ("Here's what we did, how it broke, what we did to fix it, and what we should've done to begin with")? Asking yourself "What have I done that nobody else has" is an excellent way to start. Then follow that up with the terms and concepts, stating the problem, its scope, and how you solved it provides a good basis for your paper.

Don't forget to survey the literature. Now that the new book, Selected Papers in Network and System Administration, or "The Best of LISA" as it's been called, is published, there's a single place to start for finding references. Add to that the resources available to all USENIX Association members on the http://www.usenix.org/ web site and you're definitely off to a good start.

Tom also discussed the evaluation process, based on his experience serving on or alongside several program committees. The readers consider whether the paper is enduring and whether it can result in a good presentation. Papers are evaluated on several criteria, including the technical quality of the work, the presentation of the paper, whether it advances the state of the art in systems administration, and whether it's relevant for LISA or somewhere else.

If your paper is not accepted, don't consider that anything more than a setback. Papers are usually returned with commentary that explains why it was not accepted and suggestions on where to submit it (if not LISA next year), along with commentary on the paper and its quality and presentation and so on.

If your paper is accepted, meet your deadlines. Work with your shepherd, whose job it is not only to nag you to meet them but also to help you by providing constructive feedback on what is good and what isn't. The shepherd is a resource to help make your paper the best it can be. Remember that they have their own lives to live but they are willing to help you out — just don't deliver a draft and expect same-day turnaround.

Some other additional commentary included:

Finally, we discussed some paper ideas and how best to present them for future conference paper tracks. I won't go into detail here because I hope they all become good papers, accepted to future LISA conferences, where they can be summarized in ;login:.

Session 3: Invited Talk:
What Sysadmins Need to Know About the New Intellectual Property Laws

Lee Tien, a lawyer from the EFF, spoke about what we as systems administrators have to know about intellectual property and copyright legislation, which has changed recently here in the United States. (I can hear the cries of "US cultural imperialism strikes again.") The short answer, according to the speaker, is "a lot." He provided a general overview of the issues, but when in doubt always contact your own lawyer, attorney, or counsel.

The theme of the legislation of late has been to figure out who controls the technology. Copyright law provides the creator or author or publisher of a work or expression fixed in some tangible medium, including electronic media such as RAM and disk storage, the right to exclusively copy, sell, distribute their work or expression, and the right to authorize others to do so. Copyright infringement is when someone does this — copies, sells, distributes, or whatever — without authorization. There are two kinds of infringement, direct and indirect. Direct infringements are those where you yourself are the violator. For example, if I were to give away copies of a tape or CD of music by Metallica, that's direct infringement. Indirect infrigements are when there is a direct infringement and you're involved intermediately. There are two types of indirect copyright infringements. The first is contributory, where you condone or help the direct violator, have knowledge (which has been extended to mean both "you know" and "you have reason to know"), and materially contribute to the violation, which includes the control of the facilities or the systems. The second type is vicarious, where you have direct infringement, the right to control, and a direct financial benefit. The example is of a tenant/landlord relationship. Since finiancial benefits are typically not present for systems administrators this probably doesn't appply to us. However, knowledge or reason-to-know do not apply to vicarios infringements.

So what can we as systems administrators do? Well, in smaller environments, we can avoid infringements. Unfortunately this doesn't scale well. There's the so-called betamax defense, which says if something can be used for substantial noncontributory use it's okay — but the courts aren't buying this argument yet, because it's only been applied successfully thus far to contributory, not vicarious, infringement.

What about Napster? They should have known there was infringement going on, and they provided the software and hardware (servers), so they've got contributory infringement. They also performed direct violations, and affected the right to control (vicarious) and cost the copyright owners revenue (vicarious). And even if only contributory infringement is involved, you can't foist it off and say it's someone else's problem once you have knowledge of it. So the advice here is to take cease-and-desist letters very seriously.

What about new legislation? Some case law shows that some knowledge is essential. The Digital Millenium Copyright Act (DMCA) Title II provides safe harbors for ISPs and other providers, though the safe harbors are very complicated. A safe harbor provides immunity for monetary damages only and is intended to limit the legal exposure of the provider. There are four of them defined: Transitory network passage, where all you do is deliver bits from one place to another, as in the Usenet model; system and caching, where you provide the hardware and OS but no monitoring; user stored files, where you provide the disk space; and search-and-retrieval tools, such as Yahoo! The definitions and requirements and exceptions are all very complex, written in legalese, and there's very little case law behind them. In general, though, you have to meet the specific criteria for a safe harbor, you must have an anti-infringement policy, you must accomodate and not interfere with standard technical measures to protect copyrighted works, and you must comply with notice and takedown requests. Unfortunately some of these terms are legally ambiguous, such as "standard technical measures" and "anti-infringement policies."

The big question becomes who controls the technology of the Internet? The RIAA and others want to control it because it can be used to copy and distribute works they own the copyrights to. The DMCA, in the opinion of the speaker, is qa strategy to control devices and it doesn't provide exceptions like the betamax rule, so it requires the right to control access and to make devices to circumvent access controls.

I have to admit that I wasn't thrilled with this talk. The speaker was fairly monotonic (not quite droning but close), had no slides (and therefore no visual interest to the talk), and no handouts or other material, making the dry legal details thoroughly uninteresting, even to a layperson interested in the law.

Session 4: Quiz Show Preparation III:
Quality-Assurance Run

This afternoon we performed a full quality assurance run on the code and vetted the questions for the game. We — Rob Kolstad, Dan Klein, Tom Limoncelli, Adam Moskowitz, Hal Pomerancz, John Sellens, and I — went through all 6 categories and all 5 questions for the first three games and the final round, as well as the entire tiebreaker game. (That's 150 questions.) We also detailed all the software issues, both bugs and enhancement requests, since (as previously noted) Rob was rewriting the code from the ground up (adding some 600+ lines between Wednesday and the show on Friday).

Session 5: Invited Talk:
CNN.com: Facing a World Crisis

I attended most of Bill LaFebvre's talk about the CNN.com website issues on the morning of September 11th. CNN doesn't want the slides made available, mainly because the PR staff doesn't know what happened, but their site served a record number of page-views and hits by an order of magnitude and the request rate doubled every 7 minutes. An approximate timeline has been recorded, however:

Time Action or event
8:45 AAL 11 crashes into Tower 1
8:45 Load: 84,719 hits/minute; 10 servers
8:50 Load: 87,610 hits/minute
8:55 Load: 129,086 hits/minute
8:56 Story published on web site
9:00 Load: 229,006, more than doubled in 15 minutes; at this point monitoring was disabled, so specifics aren't definite from here
9:01 First 911 page received by tech. staff
9:03 UAL 175 crashes into Tower 2
9:03 Server swings begin: Tech staff has TVs — see second crash on TVs, know they will need more machines. This will be far bigger than anything they had imagined.
9:05 Congestion collapse begins
9:11 Second 911 page — call-in number
9:15 Switch to split page
9:17 NYC airports shut down
9:27 Third 911 page — only 8 ports available for inhouse bridge
9:40 US airspace closes
9:42 18 servers (probably before 9:42)
9:43 AAL 77 crashes into Pentagon
9:47 Page stripped to minimum
10:05 Tower 2 collapses
10:10 UAL 93 crashes in PA
10:30 24 servers
11:00 44 servers: 20 boxes in 1/2 hour
11:15 Momentarily shuts down port to recover and reconfigure servers
11:30 html services restored; out for 2.5 hours and couldn't serve images
12:25 Monitoring partially restored
13:00 8 more servers restored; Bill has no idea where they found those boxes
13:30 Image service restored
14:21 Moved (2?) servers back to Cartoon network; experiencing very high loads, kids home from school
14:55 Published light page
16:15 Peak load: 1,110,284 h/m

And the peak hits per minute (H/M) and total page-views (PV) were:

Date Peak H/M Total PV
9/10/01 156,191 14.4M
9/11/01 1,110,284 132.4M
9/12/01 948,244 304.8M

And they were down for the worst of it. Some notes:

Evening activities

After Bill LeFebvre's CNN talk was the conference reception. The Town & Country had converted their parking garage underneath the convention center into a bigtop, laying down carpet, putting up neon lights, and a variety of carnival balloons and other festive decorations. The food was burgers and hot dogs and miscellaneous non-lettuce salads, with root beer floats for dessert.

During the reception I met with a couple of kind gentlemen who might actually want to interview me more formally for a job with their company. (Ah, the joys of unemployment immediately before a conference.)

After the reception was the regular scotch BOF. Unfortunately (from my point of view, anyhow) these have become de facto raucous parties and have moved away from the original goal: Quiet conversation, contemplation, tasting and evaluation and commentary on fine or unique scotch whiskeys. Security apparently had to break up the party, which reformed in another location without its original host.

Another friend was hosting a small poker party for fun; I played a few hands, munched some cheetos (or "cheezy poofs") and chocolate chip cookies, and headed out in time to catch the tail end of the scotch BOF disaster. I moved food upstairs to the new location and went from there to the hottub for a soak and then to the Presidential Lanai Suite for some quiet party.

Friday, December 7

Session 1: Hallway Track

I spent this morning doing the Hallway Track, chatting with folks, getting set up for the Quiz Show, and checking my email for the first time in a couple of days.

Session 2: Quiz Show Preparation IV:
Final Run-Through

We went through the questions one final time with a nontechnical member who's helped us before (Ronise Zenon). Ronise was originally skeptical of being able to answer any questions — but when we conned her into helping last year she wiped the floor with the other testers so we brought her back this year to help out. We replaced an ambiguous question with another one so it's time well spent.

Session 3: Quiz Show Preparation V:
Coding, Prizes, and Production

In the last preparation before the Quiz Show, Pat Wilson and I went through the prizes and allocated who would get which. Some were obvious (such as the cash awards and a brass-cast daemon), but most of the rest needed thought and negotiation so we could assign them fairly to all participants. Once that was done, and Rob had finished the final bits of coding, he and Dan worked out the handoffs and transitions for the show and we went to set it up once the session ended.

Session 4: LISA Quiz Show

This year's Quiz Show began with closing remarks from Mark Burgess, thanking us for attending. Dan Klein then proceeded to introduce our host, Rob Kolstad, and run the game. We've got all new software, mostly new buzzers, a new logo, and new theme music (licensed no less); and the LISA Quiz Show XP (for "extra fun," according to Rob, though based on the number of vendors who contributed items to give away, perhaps "extra prizes" would be more appropriate) was off and running.

We had the usual three rounds with three contestants each:

Game 1
Categories City Monuments, Acronyms, D Animals, Home Appliances, Exotic Dishes, and TV Shows X
Players Brandon Allbery (2800 after adjustments), Darrell Fuhriman (3300 after adjustments), Andrew MacPherson (3400 after adjustments)
 
Game 2
Categories Cartoon Characters, Sports, Raw Foods, E Animals, Perl, TV Shows XI
Players Tim (800), Ken Herron (1100), Doug Hughes (1600)
 
Game 3
Categories Autos, html, Airports, Ine Words, Animal Children, TV Shows XII
Players Rich Schultz (300), Mark Logan (1200), David Grieg (2200)
 
Finals
Categories Rivers, Novelty Songs, Weather, Ine Animals, Perversions, and RHPS
Players Andrew (2000), Doug Hughes (1300), David Grieg (2800)

Another new feature of the Quiz Show is automated closing credits, detailing the producers (Dan Klein, Rob Kolstad, and Josh Simon), the testers (Tom Limoncelli, Adam Moskowitz, Hal Pomerancz, John Sellens, and Ronise Zenon), the vendors who supplied prizes (Addison Wesley/Pearson, Computone, Eagle Software, Lumeta, O'Reilly & Associates, OpenBSD, Red Hat, Tadpole, TeamQuest, Tricord, USENIX Association, and VMware), the licensing information ("Music licensed through ASCAP and BMI"), special thanks to the MSI audio/visual team, and that next year's LISA is in Philadelphia in November 2002.

Prizes for the contestants included the following:

Approved SAGE Sysadmin

Barb Dijker, on behalf of the Certification committee, presented a brief overview of the various SAGE certifications to be made available.

The core certification will be called "Approved SAGE Sysadmin" in the United States and "Approved Regional Sysadmin Expert" elsewhere. The various levels, each prepended to the core certification, include:

  • SAGE Level 1, Does Ultimately Mostly Backups (DUMB)
  • SAGE Level 2, Sets-up Mail and Routes Too (SMART)
  • SAGE Level 3, Just About Completely Kewl (JACK)

Niche certifications are also available, including:

  • Security, the Tough Internet Gateway Handling Troll (TIGHT) certification
  • Management, the Kills Impending Simple Solutions Monthly and Yearly (KISSMY) certification
  • When Ignorance Perturbs Everything (WIPE)
  • Brandish Any Reasonable Evidence (BARE)
  • Making Obvious Outsiders Nauseous (MOON)

Evening activities

I went to dinner with a handful of folks at Forever Fondue and pigged out on, surprise surprise, fondue. Afterwards, too tired to try to figure out transportation to Pat's party at her house, I visited (and tended bar at) the Dead Dog party for a while before heading back to the room to go to bed.

Saturday, December 8

At last, a vacation day. I managed to sleep in until well after 10am, started working on the trip report, and wound up having lunch with Esther and Bob at Mimi's (I think). Delicious food and not hotel or Fascist Valley Mall prices, either. We'll have to remember this when we're back at the Town & Country in another couple of years.

After lunch, I hung out by the pool until Esther and Bob came back from chores (Esther fighting with the hotel and Bob packing), where we adjourned with Frank to the hot tub. Aaron Mandel later joined us there, though he didn't come in. Apparently, however, I managed to completely miss the earthquake: Between 3:30 and 4pm, a 5.2-magnitude quake hit in Mexico, about 52 miles west-southwest of Yuma Arizona. Bob felt and heard it; Esther heard it, and I missed it completely (sitting right next to Bob). We knew it was an earthquake when Dan Klein asked out his 4th- or 5th-floor room if the pool had just moved because of the quake.

For dinner, nine of us — Jef Beentjes, Esther Filderman, Bob Gill, Trey Harris, Tom Limoncelli, Randy Paterno, Joel Sadler, JD Welch, and I — went to dinner at Lips, a drag restaurant. The food was quite good and the acts wonderful (if too short). Unlike the place we went to in New Orleans (Chinese food and loud enough to make your ears bleed), Lips was not too noisy and the food Americana-based. We made sure Joel got to sit by the stage so the performers could pick on him — and they did, several rubbing his head, one smooched him (there'll be a photo online soon), and another checked her hair and makeup on his bald shiny skull. There were also about 6 birthday participants that were embarassed by being brought up on stage and crowned and photographed with the stars of the show, but only one of the audience members was male — and stripped of his shirt, handcuffed to the chair, and whipped cream applied to him by one of the stars while his girlfriend took pictures. (Joel has some pictures of this on his digital camera, assuming they came out.)

After dinner, we adjourned to the hotel where most of us collapsed in the hot tub until closing and then went to bed, since President Parter had checked out of his Lanai Suite.

Sunday, December 9

Woke up, finished packing, and checked out, splitting the hotel bill painlessly enough with my roommate (Joel). Marriott could learn something from the Town & Country; I was expecting to confuse the person behind the desk but she managed the task without any trouble whatsoever. Whatever complaints you might have about the Town & Country — and the only real two I have are the expense of phone calls ($0.95 per 30 minutes regardless of destination) and the lack of high-speed in-room Internet service — the desk staff certainly doesn't get any from me.

I shared a taxi ride to the airport with Joel and Frank. Joel and I were at adjoining gates so we hung out together and played a game of cards. (Contract gin. We each won 6 of the 12 hands, but I managed to hang on to win the game on points.)

The flight itself, while fully booked and fully filled, was otherwise uneventful. Mild turbulence on climbing and descending, but the baby didn't start screaming 'til the end. Being in the back row meant I was among the last off the plane, and it still took half an hour from the time I got off the plane until the luggage arrived on the baggage carousel. Got home to find the condo still standing and the smoke detector complaining of a dead or dying battery — which is just fine, so eventually it shut up.



Back to my conference reports page
Back to my professional organizations page
Back to my work page
Back to my home page
Last update Feb01/20 by Josh Simon (<jss@clock.org>).