Conference Report:1999 USENIX General Technical Conference
Conference Report:Monterey CA, June 6-11, 1999
Today was a travel day. Left the house around 10am (Central), flew from O'Hare to Monterey via Los Angeles. Arriving in Monterey, there were no cabs or shuttles at the airport, so I called the cab company on their free phone (which was hiding behind their desk) to let them know there were no cabs there. They sent one cab, so 3 other folks and I split the cab ride to the hotel. We got to the hotel around 5 in the evening (Pacific). Checked in (and ran into a few friends of mine from past conferences there), went to the conference registration desk (and met still others), helped with registration for a while, went to the conference Welcome Reception, and then went to dinner (Delminico's on Fisherman's Wharf) and then the hot tub with the conference groupies.
Worked at conference registration, helping with set-up and through the morning rush. Since I wasn't taking a tutorial today I did some hallway track — conversations with colleagues and USENIX staff about work and upcoming conferences. I also helped with the Terminal Room setup and worked on reviewing more of the extended abstracts for LISA '99 (November in Seattle).
Sunday evening we went to a mediocre restaurant, Rosine's, which served American-ish food for too much money and too slow service. (We got there before the dinner rush and left after the rush. And there were only 9 of us there.) In their favor, though, our expectations had been set fairly high and someone told us they did better at breakfasts.
Went to the hot tub again and continued the tradition of staying there until we got kicked out at closing (10pm). Resolved to extend the hours.
Today was the day of my tutorial, Administering Windows NT: A Course for Unix Administrators. It was basically an overview of Windows NT tailored for the system administrator, and drawing comparisons between it and other operating systems (such as Unix and VMS).
After the tutorial, went to a group dinner I regularly organize at both the USENIX and LISA conferences. 19 of us wound up having dinner at Rappa's at the end of Fisherman's Wharf. While their beer selection was pitiful ("We have Bud, Bud Light, Coors, Coors Light, Miller, Miller Light, Sam Adams..." — well, at least one was decent), the wine list and menu was very nice. I — and two others at my end of the table — had a wonderfully thick cioppino, a seafood stew with lots of fresh clams, mussels, shrimp, scallops, salmon, and halibut, in a tomato-based herbed broth.... And it was quite filling; no room was left for dessert.
After dinner, and some time in the hot tub (abbreviated because the security guard — excuse me, I mean "loss prevention officer" — still kicked us out at 10), a few of us went over to the brew pub behind the Doubletree hotel for drinks and munchies. I stuck with iced tea (and went through about 6 glasses in under an hour, mostly because I was dehydrated).
Tuesday was a free day for me; I had nothing planned. I worked a while at Registration (getting the early-arrivals for the conference itself and the Tuesday-only Tutorial arrivals checked in), read a bunch more abstracts for LISA'99, did some pre-planning for later in the conference (making sure that Jamy Libsack, the only other CT person there, and I didn't attend the same sessions during the conference, for example), read my email and processed the finished abstracts' scores (thank goodness for the terminal room), and wound up going wine tasting in the afternoon with a few friends (including 2 members of the LISA'99 Program Committee) down in Cannery Row. (Ordered some nice Johannesberg Reisling — Jekel is the winery — and some other miscellaneous stuff.)
That evening I hosted the (regular) BOF for Gay, Lesbian, Bisexual, and Transgender Workplace Issues. We had a decent attendance, and the trend of companies that provide decent benefits to nonheterosexuals seems to continue to increase. (Domestic partnership coverage on health plans and inclusion of domestic partners and their families in the leave policies (sick, family emergency, funeral, etc.) were the two major areas of benefits involved.)
After the BOF, a bunch of us met up with some other folks and wound up having 20 for dinner at Montrio, an American bistro about a block from the hotel in Old Monterey. For the food buffs out there, if you're ever in Monterey, check these folks out. We split some appetizers at our table; I think the best was the risotto with artichokes, baby spinach, bacon, and pine nuts. One of the special entrees was a 7-ounce filet mignon in a burgundy reduction that looked very good, but I wound up getting the roasted duck with pearl barley and wild rice in a gin and plum reduction. Fantastic. And even when you add in wine and dessert (most people didn't eat dessert; we were too stuffed), the costs were still reasonable. (Not quite in the per diem, but oh well.) We returned to the hotel and went to the BayLISA sospitality suite. BayLISA is the SAGE local group for the San Francisco bay area, and I spent some time chatting with a few of the members about how they run their local group, to see if the Chicago group could steal any ideas of theirs.
After the hospitality suite, the traditional hottubbing resumed.
The conference itself started this morning. The session opened with the traditional announcements as well as a few not-so-traditional goings-on. First, Andrew Hume (President, USENIX Board of Directors) presented Judy DeHarnais (USENIX Conference Coordinator — for all the USENIX Association conferences and workshops) with a lovely bouquet of flowers for her 20 years of service. As anyone who's ever gone to a USENIX conference or workshop will readily attest — especially if they ever peeked behind the scenes or volunteered to help out in some way (either officially or un-) — the conferences would not run anywhere near as smoothly as they do if not for Judy. (And she hates being the center of attention, so she was truly embarassed by the standing ovation.)
Avi Rubin, the conference program chair, continued with announcements. Some of the more interesting were:
He then thanked the program committee, the USENIX staff, the FREENIX program committee, the Invited Talks chair, the USENIX Liason, the speakers and presenters, Peter Beckman (who ran the Extreme Linux Workshop), all authors and reviewers, the volunteers for the terminal room, AT&T for letting him use business time to run the conference, and all of us for coming. Presenters and speakers were asked to meet with their session chairs during the post-keynote morning break, and to facilitate this he had pictures on the screens of the session chairs.
Clem Cole, the IT chair, thanked USENIX staff, the program committee, and his co-chair in absentia, and announced two schedule changes (which, as it happens, were both relevant to what I'd signed up for):
Peter Honeyman (a member of the USENIX Board of Directors, the USENIX Board liason to this conference, and the chairman of the USENIX Scholastic Committee) spoke next, and explained that:
Avi Rubin returned to present the awards for outstanding papers. Since so many papers had student co-authors this time through, instead of the two "best paper" and "best student paper" awards, they announced three "outstanding paper" awards by category (each of which got about a third of the combined $2000 in prize money):
Andrew Hume returned to present the other two annual awards:
Finally Avi Rubin introduced the keynote speaker, John Osterhout. John spoke for about an hour on integration applications. In summary, he compared system programming languages (such as Algol, Pascal, and C) with scripting languages (such as sh, perl, and tcl (which he invented)), and gave some reasons why scripting might be better than system programming when it comes to writing code to integrate a user interface (such as a GUI) with an existing (i.e., legacy or newly-developed) program.
After the morning break, I went to Melinda Shore's invited talk about the world of IP Telephony. Melinda wanted to dispel hype about the topic, give us a view of some changes that are in the works, and noted that accurate information is hard to come by.
The real reason to do telephony or voice over IP is cost. It's a lot cheaper to route IP datagrams (packets) than have a dedicated point-to-point connection for voice. IP telephony is really end-to-end aplications (such as CUSeeMe and NetMeeting), as well as a gateway to "real" telephony, provided by Internet Telephony Service Providers (ITSPs).
IP Telephony is service driven (with traditional telephony (plain ol' telephone service (POTS)), video conferencing, voicemail and email integration, kiosks at airports, www browsers on your cell phone, home phone, or even pager), as well as new stuff we haven't thought up yet. Melinda went through terminology and acronyms and provided several scenarios for how IP telephony is most likely going to be used, then she went into a discussion of standards.
The key is interoperability. Voice people see the terminal (phone) as dumb and the network as smart (the common channel signalling model), whereas IP/networking people see the terminal (computer) as smart and the network as dumb (the SIP model). More information is available at http://www.openh323.org/ including links to the standards, the bodies and organizations behind them, and so on.
The rest of the talk involved individual standards and how IP telephony works from a conceptual basis. Additional details can be found at ftp://ftp.lightlink.com/pub/shore/usenix.ppt, and either PostScript or PDF will be made available some time after the conference.
After a nice relaxing lunch with the Works In Progress (WIPs) coordinator, Peg Schafer, I went to the vendor show. Nothing jumped out at me this time around, though I did have an interesting discussion with the Midwest Marketing Manager for VA Linux Systems, Rich Furst, about who Collective Technologies is and what we could help them with in terms of joint sales. Rich was bemoaning that while they could provide the product, providing support and operations assistance was beyond his company. I said "We can do that," and gave him my card. (I'll copy this information to the leads-mw mailing list.)
The cutest give-away (in my opinion, anyhow) was from RedHat Linux. They were providing "Linux Kernels" microwave popcorn.
Jim Duncan graciously stepped in at the last minute to give a talk about security and intrusion detection. By and large security is misunderstood. There are three major components: confidentiality (access to information), integrity (guarantees as to the reliability of information), and availability (performance). Jim went into some detail about the real cost of hiding events (as opposed to reporting them to some agency, such as CERT, or to the vendor of your networking equipment), including morale, disruptions to the production environment, poor return on investment, and reduced confidence of the company. For example, if the public is misled, the impact of the crime cannot be known and justice cannot be served. The real debate is between convenience versus preservation.
In Jim's opinion, the current major issues are legislation (especially knee-jerk legislation), such as that involving Hollywood, guns, encryption, schools, and health care; intellectual property, including bad patents that should never have been granted; poor corporate decision making; and critical infrastructure assurance.
The question was asked, "What can we do to help?" His top three answers were (1) increase your involvement, especially at the local level, which tends to be more effective than at the national or federal level; (2) take a teacher to lunch; and (3) generally be a good example.
First I went to the SGI hospitality suite on the tenth and top floor of the hotel. There was very little food and next to no SGI presence (only one current employee of SGI, and he kept a low profile; there were more ex-SGI folks in my group than current SGI folks at the hospitality suite), so we went out to dinner. We walked over to cannery row and went to Bubba Gump's, which (as you might imagine) is a restaurant with a theme: Forrest Gump. (Oi.) Dinner was mostly uneventful and reasonably priced (for touristy food). After dinner we walked the remaining two blocks to the aquarium, where the evening's reception was being held (desserts and open bar). We'd had the reception there 4 years ago (when LISA was in Monterey in '95), and the new section with the jellyfish exhibits had opened up. If you're ever in Monterey, consider a visit to the Aquarium; it's great.
Since a group of us are Babylon 5 fans, we were going to go back to the hotel for the *Crusade* premiere. Unfortunately, though we made it back on time, the hotel's TNT feed was from the east coast, so we were 3 hours late and missed it. Drat. So we adjourned to the hot tub. (Is anyone sensing a theme here? :-) )
Margo Selzer talked about gizmos, or those nifty cool thingies you find you can't live without, including cell phones, PDAs, and so on. She went on to talk about embedded databases, where the gizmo is an application of some kind instead of a general-purpose computer.
Embedded databases are those which are hidden from the end user. For example, Netscape Communicator and Sendmail Pro both use embedded databases (specifically, the Berkeley DB from Sleepycat) in the product, but the general end user is unaware that the database even exists.
Challenges with embedded databases are those topics which are either hard to do or interesting. These include hands-off administration (as there is no DBA), so backups and restores and log archival and reclamation must all be transparent and automatic; simple and robust, since data loss is not allowed; low latency, as the application must be fast; and a small footprint.
I attended 2 of the 3 paper sessions in this block.
Rob Miller (the student) and Brad Myers (his advisor) worked on solving the problem of processing structured text, such as html, C source code, tabular and prose ASCII text, and so on. Historically, programs like awk, perl, lex, and yacc were used with editors to do custom programming to parse output. However, the problems with this are that custom programming is revision-intensive, especially when the formatted text changes.
The solution that they came up with is a lightweight method that separates the structure description from the text manipulation functions. There are Unix-style command-line tools (such as grep, sort, and diff) for text, as well as a graphical user interface. The tools are written in Java so they are highly portable, reusable, and provide high functionality to the user.
The rest of the talk was a demonstration of the GUI, Lapis, which is a web browser. The user can define structures automatically or manually, can load and save labels and searches, can filter out (and omit from the display) unwanted items, can sort on any field, and can even perform some mathematical operations (sum and average were demonstrated), on Java, html (both formatted and raw), C, and even plain text.
Source is available at http://www.cs.cmu.edu/~rcm/lapis/
The general concensus of the group of us was that this talk could be summed up by the chroot(2) system call. In effect, SBOX takes a virtual web server's cgi-bin directory and does a chroot() on it to prevent it from being abused by other sites' cgi-bin scripts on the same physical but different logical web server. The speaker was entertaining, but the talk itself didn't offer anything novel.
I skipped the third paper in this session so I could get some lunch and do data entry of the remaining reviewed abstracts before the LISA'99 Program Committee meeting. Lunch was at a taqueria a few blocks away in Old Monterey and was inexpensive and very filling. (The mango salsa was truly yummy, too.)
After lunch and data entry, I went to the Referreed papers session for the Web Servers papers.
Because the web is slow (hundreds to thousands of milliseconds) and unreliable (a 15-day MTTF), fixes are expensive (T1 lines cost on the order of $1,900/month), and latency can range from the good (30ms) to the bad (400ms) within the US, not to mention the 500-3000ms latency outside the US, the authors were motivated to seek a solution.
One solution is to replicate the web site (i.e., proxy caching). However, these are ineffective as servers don't know about the proxy cache so the data in it gets stale, and they typically have less than a 50% hit rate. A second solution is server clustering, routing requests to different web servers (such as via round-robin DNS), which works but does not scale well. A third solution is to distribute copies of the data onto multiple servers (mirroring). This works but it is possible for the mirrors to fall out of synchronization with each other, which means different users see different data. This is generally considered less than ideal.
What the authors hit on was Web++, which uses a client-side applet and server-side extensions, called a "servlet." The client implements replica selection and fail-over, and the server maintains the replica directory and transfers data via HTTP and can balance the load across servers. This gives the benefits of all three solutions — replication, proxy caching, and mirroring.
The paper has the detailed statistics, but in general 90% of the samples were within 30% of the value for 483 minutes, which is a 60-70% improvement over random selection of the replica (mirror). The servlet preprocesses the html (with a 14% increase in time). Evaluated performance on static-sized files between the United States and Europe showed that the Web++ replica system had a 48% improvement in response time (latency between time of submit and time of last response received), and Web++ was 2.2% better than the optimal ("send many simultaneous") connection.
Because of the trend towards server clustering for scalable HTTP performance and the need for better performance with HTTP 1.1 (Persistent HTTP, or P-HTTP), the authors modeled a single front-end web server with multiple back end servers clustered together for load averaging. Previous work led to LARD, which distributes the request to the back end usingcaching, which was experiencially better than weighted round-robin selection.
P-HTTP is a proposed enhancement to HTTP 1.1 that allows one TCP connection to carry multiple HTTP requests, instead of a one-to-one mapping. This avoids the extra open and close of the socket connection (so overhead decreases) and avoids the problems with TCP slow-starts or latency (latency decreases). The mechanism is that the client talks TCP to the front end node, which hands the TCP connection off to the back end node it selected. The client and back end nodes are effectively talking to each other directly and the front end node is out of the loop.
So what if you use LARD with P-HTTP? The first request uses LARD and hands off the client's TCP connection to the back-end node, X. On subsequent requests, if the data is cached at node X, use X immediately. Otherwise, if the disk is not active on X, use X; otherwise use LARD to another back-end node, Y. The goal was to get as close to the ideal node as possible.
A prototype was implemented that transparently runs on off-the-shelf web servers (such as Apache). The front-end and back-end nodes have loadable kernel modules, and a dispatcher (LARD) runs on the front-end node. In practice using LARD and P-HTTP was 26% better than LARD alone; together, this is up[ to a 300% improvement over weighted round-robin.
The goals for the Flash project were a high throughput, very portable web server which could handle a wide range of workloads. During the process, the authors examined optimization, concurrency, and whether architecture affected the implementation. They treated the server process as a black box and ignored (made no changes to) the kernel.
In general, the HTTP connection process between the client and the server can block on I/O in two places: the network or the disk. For concurrency the network, disk, and application should all be able to run concurrently. The possible architectures for this include the following:
The ideal situation is a single (shared) address space, good disk I/O, no synchronization required, and low per-client resource use. The authors came up with the Asymmetric Multi-Process Event Driven (AMPED) model by adding a helper thread to help the event drivers. Flash is an AMPED architecture that uses writev() and application-level caching; flash sits between the helper thread (process) and the main web server. While detailed benchmarks are in the paper, AMPED could handle more requests with less latency per unit time better than any of the other architectures. However, further testing showed that the architecture — MP, MT, SPED, or AMPED — is not a first-order effect, and that optimizations are. As the number of client connections increases, MP performance drops but other architectures are still okay (though MT eventually saturates as well), mainly due to overhead.
In the real world, the authors found that flash is up to 30% faster than Zeus (the SPED architecture) and up to 50% faster than Apache.
In lieu of the fourth session today, the LISA'99 Program Committee members present at USENIX, along with the relevant USENIX staff and board members, met and discussed the status of the LISA'99 conference. (In a way, this was a shame, since I wanted to go see Tim Bass' Invited Talk about the Langley cyber attack, but the committee meeting had to take precedence.)
As of the meeting, there were 67 submitted abstracts for the referreed papers track, 20 Invited Talk submissions, and about 20 possible Practicum track ideas. Furthermore, three or four possible workshop ideas were floated to run parallel to the Tutorials track for senior-most administrators.
We also discussed fun things to do during the week, including some that could be prepared in advance (for example, commercials like we did at the San Diego LISA, or USENIX' Funniest Home Videos). We also discussed what to do for an endnote in lieu of the LISA Quiz Show which, while fun, is getting a bit long in the tooth.
After dinner (again at Delminico's on Fisherman's Wharf; I had a stuffed salmon that was very good (although the waiter neglected to mention that part of the shrimp, scallop, crab, and bread-crumb stuffing included fungus (which I don't much care for))) I briefly sat in the SAGE BOF. It was basically a member of the SAGE Executive Committee saying who SAGE is, what we do, and what future plans were, and was targeted mostly to non-SAGE members. We also asked what we could do to improve the LISA and LISA-NT conferences.
I wandered out of the SAGE BOF and into the History of UNIX at the University of California at Berkeley talk that Kirk McKusick was giving (interestingly enough, opposite Linus Torvalds' BOF next door). It was an interesting overview of what the UCB team did in creating and maintaining first ancillary tools, then major utilities, and finally a complete distribution of UNIX. (At the intermission or break, Kirk also sold the "Berkeley Daemon" shirts — polo or tee — with the daemon running after the glowing orb.)
The authors wanted to solve two major issues. First, server applications have many open network connections; second, the select() system call does not scale well, even when it's optimized. This led them to look at an event-driven system, where the threads do I/O, to see if it alleviates the problems.
The problem with select() is that the entire register of bits is created by the application, copied into and out of the kernel, on every select() call. And if select() is poorly implemented, the bit representation is copied several times in the kernel. By default, select() uses a state-based view, where the state of each file descriptor is given by the corresponding bit. An event-based view is better than the state-based view, hypothesize the authors, because it leads to less work in the application and less translation (between state and event) work for the kernel (which uses events internally).
The problems with event-based code are three-fold. First, frequent arrivals of events can lead to excessive overhead. Second, frequent arrivals of events can penalize an application uninterested in specific events. Third, storage requirements for events (to ensure not losing any) may be excessive.
The authors' approach was to create an event-based view with low overhead. Other design aspects included:
The new API, detailed in the paper, boils down to the following code snippet:
while (1) {
n=get_next_event(max,event,timeout);
if (n<0) { error_handle(timeout) }
else {
for (i=1,i++,i<=n) {
handle_event(i)
}
}
}
The authors implemented this functionality on Digital Unix 4.0 and found, in terms of throughput versus the number of connections, that it was 26% better than select() and 11.6% better than an "improved select()" they'd written up previously. Even varying the request rate the new event-driven API provided lower response time.
I was paged out of the latter two paper sessions by the client, and went on to prepare for the USENIX Quiz Show.
I and three others went to Rob Kolstad's room to go through the questions and answers for the USENIX Quiz Show (conference endnote). We threw out some already-used categories, rewrote some poorly or ambiguously worded questions, and wrote the Geek Chic categories for all four rounds. After this, we went down to the Member Services desk and grabbed the questionnaires so we could score them and choose the six highest-scoring and three randomly-chosen entries as the nine contestants (along with a list of alternates).
Peter Salus gave a talk about the history of Unix and how the rise of Linux paralleled it in many ways. I'd heard this talk before, so I absented myself from the room and went to work setup for the Quiz Show.
The USENIX Quiz Show (UQS) returns to the conference in its traditional endnote position. With Rob Kolstad hosting and Dan Klein running the computerized game board, the nine contestants got to vie for the championship of the conference.
Since we can't provide all 120 questions and answers, we can at least provide the categories by game and the contenstants:
Asterisks indicate the winners in each game. All contestants received prizes, including O'Reilly and Associates works and a 100base-T Ethernet card; game winners each received a OpenBSD glass and at least $50 in cash; the grand prize winner received a copy of Sendmail Pro and $100 in cash, as well as the drivers for the 100base-T Ethernet cards.
(It's just as well we took good notes this time. After the LISA Quiz Show in Boston, nobody had brought the names of the contestants — or even the winner! — home from the conference so we had some difficulty in identifying the winner for ;login:.But don't tell anyone, since we're all kind of embarassed about it.)
While the conference is technically over, a bunch of us went out for a nice Italian dinner (Cibo, across the street from the conference center) then ran up to the traditional end-of-conference party.We asked the folks tending bar what they needed then ran out to the store to get it (using the overage from dinner's bill-paying to cover the cost).Hung out at the party until I realized I was falling asleep listening to a (rather enjoyable, really) conversation about O''Reilly and Associates and royalty checks.
Before bailing for the airport, a few of us went out for brunch. The hotel staff had suggested a "really good place" for brunch — and it happened to be Rosine's, where we had a less than stellar experience for dinner Sunday night. However, our waitress was great (and provided assistance for one of our party who couldn't decide what to get by telling him that the "chili egg puff" was "disgusting," "pre-packaged," and "gross." The food arrived quickly and was just great, so I'd say that Rosine's is good for breakfast foods and not necessarily where you want to go for dinner. (Your mileage may vary.)
The flight(s) home were... entertaining. The flight out of Monterey was delayed due to the late arrival of the aircraft. On our first landing attempt into LAX, we apparently were coming in too steep and too fast, so we wound up aborting the attempt and trying again. This (of course) caused me to miss my connecting flight back to Chicago. The kind folks at American Eagle said "Well, our next flight out's not until 11:30pm, so let's see if we can book you on another airline so you don't have an 8-hour layover here." Yowch.
They bumped me onto a United flight that was leaving at 4:30 (so I'd be an hour behind schedule). Unfortunately, it didn't depart until 4:40, and got caught by a Chicago-based ground hold due to bad weather. We finally left around 5:30 and landed in time for me to get my checked baggage after midnight. Bleah. (It's worse than you think: Getting home at 1:30am was bad enough, but I had to leave at 5:30am Sunday morning to catch a flight to Houston for a friend's wedding.)
Back to my conference reports page
Back to my professional organizations page
Back to my work page
Back to my home page