Conference Report: 1998 LISA
Conference Report: 1998 LISAThe following document is intended as the general trip report for Josh Simon at the USENIX/SAGE LISA Systems Administration Conference in Boston MA, from December 6-11, 1998.
Travelled to Boston. Believe it or not, the flight was very nearly perfect — no screaming babies, almost no turbulence, mostly empty of passengers, my luggage was the second one on the carousel.... American Airlines gets a Thank You letter from me on this one. (Hey, I complain enough when they screw up, may as well tell 'em when they do right too.)
Registration opened at 5, but around 4:30 I was tapped by the Tutorial Coordinator to help hand out tutorial notes. I registered myself (there's occasionally an advantage of being considered as a staff person) and went to the handout room for tutorial notes. When the doors opened at 5pm, I worked for a couple of hours handing out people's tutorial notes and chatting (briefly — I was working after all) with friends and answering folks' questions and so on. Was fun. Bailed at 7 for dinner with friends from past conferences, so I missed the CT get-together at Dick's Last Resort.
Met up with friends in the lobby for dinner at the nearest Legal Seafood. Of course, it's 7:30pm on a Saturday night, so we're expecting a long wait. Turns out that a party of 30 had done a no-show, so they quickly reset the tables for us and we were seated within 5 minutes of arriving. Both the food and the service was excellent. (For example, I don't like calamari and yet enjoyed the calamari in a kung pao sauce.)
In this day-long class, Ed DeHart (formerly of the CERT Coordination Center) went over Internet-related security topics of interest to the Unix systems administrator. He went over why we should care about security and suggested policies in the areas of authentication, access, privacy, acceptable usage, handling incidents, integration with purchasing, and setting user expectations. He then went over some common security areas to look at, such as /etc/rc scripts and directories, network services such as finger (in.fingerd), telnet (in.telnetd), and rsh (in.rshd), and ways to ensure their security. Ed also provided various references for system security utilities, including Tripwire (which checks a file's integrity and changes in content, links, dates, and permissions), MD5 (for checksumming files), crack (for breaking easily-guessable passwords), COPS (for checking system configuration), PGP (to sign or encrypt data transferred between hosts), and ssh (to encrypt the equivalent of an rsh session).
We then went over how to secure a web server, including techniques like mounting the document root directory read-only (for infrequently-changing sites), securing the CGI directories, and only enabling server-parsed html in specific trusted directories. We also discussed how to secure an FTP server, including ownership and permission issues. We went on to discuss network-level security, such as philosophy (do nothing, deny everything and allow specific services, or allow everything but deny known security problems), IP services and addresses to filter at a router level, smurf attacks, packet filtering, and how to restrict services. We also went over some useful network administration tools, including monitoring software (like tcpdump, ARGUS, Netlog from TAMU, and vendor programs like HP OpenView), wire cutters, and firewalls.
We wrapped up with a brief discussion on intrusions. In general, the procedure (which should be written before the first incident) should first determine the magnitude, then protect the evidence, report the problem appropriately, deal with any press fallout, and find the intruder. Of course, site-specific requirements can change the order; for example, if your site is a bank and the cracker is sending dollars to their account, you want to use the wire cutters first and notify management later.
Finally, Ed provided information resources, including organizations (www.cs.purdue.edu/coast, www.first.org, www.cert.org), mailing lists, USENET news groups, books, BOFs at the current LISA conference, and a cracker's web site.
Went to dinner at what the concierge said was a nice restaurant for Cat Okita's 30th birthday (as a surprise for her) called Stephanie's. The food and service were... less than stellar. We were not impressed. (When I mentioned this to friends, some were mildly surprised but most thought that the restaurant had seen better days.)
Attended two half-day tutorial sessions on the Domain Name Service (DNS). In the morning session, we went over the very basics of DNS administration. DNS, for those who may not have dealt with it, is a hierarchical distributed database that uses a hierarchical naming scheme. It allows the delegation of parts of the hierarchy, much like a Unix file system allows different users to own different parts of the file system tree. We went over how the name space is partitioned (with a digression on the changes in the top-level domains, from the original 8 — com, edu, gov, int, mil, net, and org — to the ISO 2-letter codes — au, ca, fr, and us — to the "new" top level domains like web and bus), and how authority is delegated.
We went over the actual protocol format (not quite at the bit level, but close) and how the resolver (client) communicates to the server. We went over the differences between recursive resolution and iterative resolution and how caching improves performance. We then went through an example of configuring DNS for a make-believe company, Foo Enterprises, building named.boot and db.foo (in the BIND 4 syntax). This introduced the SOA, NS, A, and PTR record types. We continued working on the example to include the CNAME record.
We wrapped up the morning session by going over the differences and communications between primary and secondary DNS servers, and how to use abbreviations instead of fully-qualified domain names in the configuration files.
The afternoon continued building on the concepts from the morning session. We introduced the concepts of subdomains and delegation and introduced the concept of zones. We also were introduced to the glue record and those remaining resource record types we'd not gone over in the morning: HINFO, MX, RP, TXT, and WKS. We went on to cover the new syntax of BIND 8 and the relevant nomenclature changes and configuration language changes. Next, we learned about tools that assist in administering a DNS zone, including dig, doc, dnswalk, h2n, and nslookup. We finished up with an overview of how to get DNS to work in a firewalled environment for both outbound and inbound requests.
In the evening we went to the South End Grill, a fantastic restaurant for the LISA motss.dinner. This tradition, started more or less at the 1995 LISA in Monterey CA, lets a bunch of us go out to dinner and have a really good meal and just a great time. (A few of us also plan out the next evening's Birds of a Feather (BOF) session, more on which later.)
We effectively took up the back half of the restaurant at 3 large tables. Our table (6 people but we grow to 7 as a late-arriving local gets there) had three wonderful appetizers: a hot and spicy chicken tenders and wings, a shrimp and calamari with tangerine fennel garnish, and a grilled duck sausage in a wild berries reduction with fingerling potato salad. Absolutely fantastic. And while the server may have been a wee bit too perky, she definitely knew how to do her job well. (Hint: If I don't notice when my glass gets refilled, the waitroid is doing great. She managed it at least once and possibly twice.)
Workshop day: ATW, the Advanced Technical Workshop. While we were supposed to let the coordinator know in advance, somehow my email didn't get to him or bounce back to me, so I wasn't on the list. Fortunately, before the session started a few people cancelled and I could get in. We talked about the issues we saw currently and coming up soon for the day, and I think most if not all of the attendees got a lot out of it.
The evening included co-hosting the GLBT* Workplace Issues BOF. We went around the room and said whether our company was or wasn't GLB-friendly, whether or not we had domestic partner or spousal-equivalent benefits, whether sexual orientation was in the Equal Opportunity statement, and so on. USENIX President Andrew Hume made his usual appearance to ask us to think of what the Association could do to make things better or easier, and if we were seeing any problems, and so on. Nobody had any new issues to raise (we will probably work on a booklet on how to help your company establish these policies, including arguments to cover the issues of cost and most of the other objections typically raised in implementing policies of this nature, but that's on hold while the publication schedule is revised).
We got up in time for the ungodly early (7:30am) breakfast meeting, and wound up being among the first to arrive. Once everyone had arrived and gotten some food, we went over what we'd be doing at the booth. I then went up to help out with Registration (and registered the CT folks from breakfast who hadn't picked up their packets yet). Adjourned from there to the keynote of the conference.
The keynote session began with the program chairs, Xev Gittler and Rob Kolstad, thanking the coordinators of the various tracks — Dan Klein (tutorial), Pat Wilson and Phil Scarr (invited talks), Lee Damon (guru), Joel Avery (global LISA workshop) and Adam Moskowitz (advanced technical workshop) — as well as the folks in the USENIX Conference office, Lynda McGinley for the terminal room, Verio for Interenet access, Evi Nemeth et al. for the MBONE broadcasts, everyone who scheduled and hosted BOFs, and the vendors for the exhibition. They went on to announce a few last- minute schedule changes (a few people didn't show up so their papers were not presented, for example) and to present awards.
The award for Best Student Paper went to C. S. Yang and M. Y. Luo for "Design and Implementation of an Administration System for Distributed Web Server" and the award for Best Paper went to Mark Burgess for "Computer Immunology." The 1998 SAGE Outstanding Achievement Award went to Tina Darmorhay for her work in the founding of SAGE, her efforts in editing ;login:, and her efforts to give back to the systems administration community. (This came as no surprise to me, as I was the chairman of the committee that selected her name and recommended her to the SAGE Executive Board. But it sure seemed to be a surprise to her!)
The keynote address itself was Eric Allman (the author of sendmail and now the President of Sendmail, Inc.) who spoke about Open Source Software (OSS). The thrust of the keynote was that OSS was good for all, as it can provide more dedicated resources than a single vendor can, and it leads to a more stable code base and more tools, with shorter turnaround time for bug fixes and functionality enhancements. His predictions, which seem to make sense and weren't particularly earth-shattering, were:
The next session I attended was the panel discussion on teaching system administration. The panelists — David K, who teaches system administration at Florida State University; Matt S, who works at the Cisco Networking Academy at the Maryland Virtual High School; Dan Klein, who (among other things) is the USENIX Tutorial Coordinator; and David K, who works for Digital Island — each gave a 10-minute position statement then discussed various issues and took questions from the audience. In general, the panelists all agreed that:
After the panel discussion I went to lunch, then quickly changed into the booth drag — blue denim shirt and khakis — and went to work the booth for an hour. Generated a few leads, passed out a few shirts, and had fun. Didn't realize my shift was over until 15 minutes after it ended.
After changing back into conference drag, went to the panel discussion on University issues. Basically, it was representatives from various universities — Texas A&M, the University of Minnesota, the University of North Carolina, and the University of Wisconsin — introducing themselves and giving a brief overview of their environment. They provided some detail on both the good and the bad of their environment, and discussed some aspects such as security, disciplinary actions, tools, and some issues such as scaling services in large environments.
After the sessions I went out with a smallish group to Legal Seafood for dinner and then to the hot tub to wind up the day. When the tub closed down, we went to the bar for dessert and I wound up in a long discussion with the program chair for LISA '99 about things to consider for putting a committee together.
Started the day blurrily (as I was coming down with what seems to be an upper respiratory ailment; there was a cold going around the conference). My first session was Dan Klein's "Succumbing to the Dark Side of the Force: The Internet As Seen from an Adult Web Site." As I was his transparency turner up front, I didn't get to take notes, but it was effectively the same talk he'd given at USENIX in New Orleans, without any defensiveness on why he is the technical person for a dozen Internet pornography web sites. He went over some of the technical issues for maintaining such a site, and noted that porn sites tend to have better security and adult- verification than some banks. In fact, due to the nature of the adult entertainment industry, technological advances (such as better or faster compression algorithms for data) and similar innovations may come more quickly from pornographic sites than from other industries. The talk was (as at USENIX) very well attended. (And no, he didn't show pictures; the talk was rated PG-13.)
On the technical side, Dan talked about various techniques to reduce the load on a web server. They include: load sharing, load shedding and load boosting. Load sharing is basically using DNS entries in a round-robin fashion to distribute the load. The main issue with something like this is making sure that all of the servers have the same data.
Load shedding is where there is a front end server. This server hands off initial requests to back-end servers that have the real content. The problem here, just in the above, it keeping everything in sync.
Load boosting is done on the client side. He had discussed earlier how a lot of sites make their money based on the number of hits a given URL receives. Thus sites will have banners and javascript programs that pop up other windows that access the same URL. Load boosting is where you turn off javascript on the client to prevent these other windows from appearing, thus reducing the amount of time it takes to load in a page.
One item that is good practice that he mentioned is keeping logs. Logs help in a number of different places. They help plan for the future. They help determine possible security breaches. And, in the case of legal action, they can help cover you if someone falsely accuses you of something.
I wasn't in an 11:00 session as the Practicum track which I was going to attend had been cancelled. At 12:30 I attended a brief memorial service for John Lions, author of the Lions commentary on Unix V6, which was perhaps the widest illegal distribution of copyrighted material in the technical arena at the time. (Some people still have fifth-generation photocopies.) Following that, I quick-changed into CT Booth drag and worked the booth for an hour. (Luckily, a friend brought me back a sandwich from Subway, so I could eat something.)
Following the booth stint, I went to John Okamoto's invited talk on Repetitive Strain Injuries (RSI). He went over some brief anatomical terms, discussed ergonomics, and provided information on symptoms and diagnoses and available treatments. He also provided information about legalities (including workers' compensation for on the job injuries), which while focussed on California law since that's his state of residence were nonetheless useful in the general sense.
I felt less than well (in all honesty, I felt downright lousy), so I skipped out on the 4pm Certification debate and went to take a nap. I'd planned on getting up in time for the 5:30pm SAGE Candidates Forum; I didn't. Nor did I wake up in time for the conference reception or most of the hospitality suites this evening. I did get out to a brief Italian dinner (very good, lots of food), but went back to bed immediately thereafter. All in all, I got about 10 hours of much-needed sleep.
Woke up a little less blurrily than Thursday, but realized I was indeed ill. Went to the Network Administration Panel, which unfortunately had to be redone at the last minute as none of the original panelists could make it. This was unfortunate in the specific case because the presenters were all senior executives from companies who were more or less selling their product. Most of the presentations were on the concepts behind their product, none of which were remarkably new or interesting. (The moderator later told me he wished he had a gun; I didn't ask if he meant to use it on them or on himself.)
Following the panel discussion, I adjourned with a few others to Rob Kolstad's room, where we set up the LISA Quiz Show equipment and ran through the questions. We managed to correct some typos (such as the browser company "Nescape") and removed a category we'd had last year, and rewrite questions that we thought inappropriate for the audience, and packed up in time for me to get to the LISA 99 planning meeting. We had a brief discussion about what did and didn't work with the new Practicum track and the workshops and how to improve on them. We also decided on a theme for the conference (which I can't announce yet as it's being rewritten into Marketing-speak first), suggested some keynote speakers and topics, and had lunch.
After lunch, rushed (a bit late; the meeting ran long) down to the mailing lists referreed paper presentation. Missed the first speaker on mailman, but enjoyed finding out about how the firewalls list was moved to a new system and the administrators had to change the queueing model so the 4500-recipient messages could get processed without huge backlogs. They basically split the recipient list into smaller chunks and use hard (not symbolic) links to the message file, then run one sendmail process for each message queue to get messages out faster. The third of the three papers was on request, a task tracking tool, and its upcoming beta release of version 3. In short, it's become very modular, is very extensible and configurable, and looks to be useful for small to medium environments.
After the session I helped set up for the LISA Quiz Show, hooking up and testing the equipment and acting as one of the judges for the contest. Once we got rid of a bad extension cord in the buzzer system, it went fairly well. We had 3 rounds, then the 3 winners competed to determine the 1998 champion, then we had a tournament round where he competed with Hal Pomeranz and Snoopy to establish once and for all the champion and thus the recipient of the 1998 LISA Quiz Show cap. (Amusingly, Daniel — this year's winner — beat Snoopy to the punch on virtually all of the European History questions.) Daniel won.
Went out to a Chinese dinner (Mary Chung in Cambridge) with a small group, and returned in time to work the bar for a couple of hours at the regular close-of-conference party. After that (which was lots of fun, as I got to chat with pretty much everyone there), went to bed around 11 or 12 ish.
Slept in; had lunch with a friend from the conference who stayed over as well. Went to Peg Schafer's birthday barbecue bash in the evening.
Travel day. Got to the airport with enough time to switch to an earlier flight so managed to get home by 2:30pm Central and to finish up writing the trip report and session summaries.
Revised trip report and break-out writeups, filed expense report and time sheet, and prepared for the per diem report.
Back to my conference reports page
Back to my professional organizations page
Back to my work page
Back to my home page