Conference Report: 2011 LISA Advanced Topics Workshop

Tuesday's sessions began with the Advanced Topics Workshop; once again, Adam Moskowitz was our host, moderator, and referee. We started with our usual administrative announcements and the overview of the moderation software for the new folks (more than in any past year). Then, we went around the room and did introductions. In representation, businesses (including consultants) only outnumbered universities by about 9 to 2 (up from 4 to 1); over the course of the day, the room included 6 LISA program chairs (past, present, and future, the same as last year).

For the third year in a row, our first topic was cloud computing. We still don't have a common definition, though the room seemed to agree that w're moving towards the "Whatever as a Service (WaaS)" model with Software, Platform, and Infrastructure as the most common. One problem is the relatively low amount of data on the scalability of the services; when the cloud is abstracted away from our control, there can be problems when production has capacity or bandwidth or speed requirements. When your growth in 18 months is to the same size as the current cloud, that won't work. Anything with growth may not be appropriate for the cloud, though that's not necessary true for well-understood and well-behaved web applications. For some, the consumer view of "A place somewhere out there to keep my data" is a good definition. This isn't new to most of us. Businesses with certain regulatory requirements (FERPA, HIPAA, and SOX) may have requirements preventing them from moving certain data (and thus the processing thereof) to the cloud. The ability to spin up a machine and provision it via some configuration management system without having to do actual work (racking, connecting cables, and so on) is a good thing. We'll probably come up with different terminology in the industry.

Next up we discussed increasing regulation of Internet activity. Governments don't seem to have a clue about the Internet; each country doesn't seem to understand that they don't control the whole Internet. There's a lot more censorship on national boundaries (Australia, China, and Egypt were mentioned, and before we went to press the USA had legislation pending as well), and we're concerned where this might be leading. SAGE-AU managed to get the Australian legislation put on hold. The room seemed to be split on whether lobbying would really have any effect, though stewardship (such as ARIN for IP addressing) might be a good thing. This was a fairly gloomy discussion. One person noted we have to give politicians an alternative: They'll take the most expedient thing. We need more companies to help enable the environment we want.

After our morning break we had our annual lightning rough of new-to-you tools this past year. The answers this year seemed to fall into two categories, software (Augeas, cloud-based VM systems, Dropbox, Evernote, f.lux, git, Google+ Hangouts, Internet in the pocket (any smartphone), OneNote, OpenStack, Puppet, Trac, vimdiff, vpnc, WordPress) and nontechnological (changing jobs, getting engaged, new mattress, paying others to do work for you, taking vacations, and teambuilding exercises at shooting ranges).

That segued into career paths. The general question was how to move out of a too-stable, unchanging environment where there's no opportunities for growth without going into management; several believe they're in that kind of workplace. Companies more and more seem to claim to say "We're interested in people who've been around for a while" but the reality is that they're hiring younger, inexperienced people because they're willing to work ridiculous hours for less money. Becoming senior often leads to becoming siloed. We took a straw poll: One person has been at his job for 17 years; about half a dozen were at 7 years or more. Having a technical growth path is important. The problem with even tall technical tracks is they get narrow pretty quickly. Having other senior people around (even in different silos) to learn from can be helpful.

On the subject of interviewing, understanding the deeper concepts is much better than trivia; make the interview questions open-ended so you can see how the candidate thinks. When you interview for a new job, always target the job after that. Remember that you're interviewing them as much as they're interviewing you. It's also probable that you know more than you realize. Practice interviews are good. Honing your higher-level thinking and problem solving skills is also useful. We all have contacts; use your networks and possibly bypass the formal recruiting process.

On the subject of hiring, several have had problems finding enough high-quality people in the pipeline. Finding those who're willing to and interested in looking at the big picture is problematic and frustrating. One person believes that intelligent companies don't care so much about you knowing everything already but just being "clue-ready" to pick up their oddities (though hr has been filtering a lot on the specifics). Hiring managers working with hr to build the filter may be helpful. However another is seeing the opposite: word on the street is that managers want very specific things. This may be region-specific. Any company needs to understand there's a learning curve, and hire people with clue so they can learn the specific technology. It's not a bad thing to come in understanding the space even if you don't have specific experties. Demonstrate proficiency on the stuff you're doing now and how you've been able to pivot in the past. However, it may depend on where you're going: Big outsourcing organizations nowadays seem to look for the specifics so they can hit the ground running at the client, whereas research organizations or universities may be more willing to hire clue-ready people without specific skills or experience in a specific technology. One person had a senior position open for 6 months; they'd find a candidate who they liked, but would take too long to get back to them and the candidate would slip away. They wanted to hire a candidate to revitalize and reinvigorate the team. They got a new recruiter on the hr team and within a month they filled all 3 open positions with amazing people. Sometimes you really do need a good recruiter.

The next major discussion was about what the DevOps and sysadmin community needs but doesn't have. We already have contacts, national and regional conferences, some sorta-magazines, a mentoring program (LOPSA), and mailing lists. One immediate response was lobbyists, linking back to the previous discussion on regulation. Some disagreed, believing that improving public perception of what we do would be helpful even without poltiical lobbying. One believes that "sysadmin" is too narrow a term; many of us do more than just systems. We'd be better served as a community if we had better labels (for example, service administration). It's systems, databases, services, networking and connectivity, and so on. DevOps is another facet of the whole, and it's being integrated, but names are important and the "sysadmin" name may be too restrictive. One possible problem is a universal professional identity is missing from the field. The medical profession (doctor/nurse) was brought up as an example. However, humans for example only work in specific known ways; IT can work in many different ways so it's more complicated.

One tangential discussion was on the term DevOps. Some see it becoming as much a buzzword as cloud. We're not integrating the big DevOps communities into the USENIX/SAGE/LOPSA community. Is it "deploy multiple times a day to Production"? "Continuous integration via Hudson or Jenkins"? It should also be remembered that what works (or not) for websites definitely won't for larger enterprises. Even Configuration Management hasn't penetrated as much as people seem to think it has. We don't have best practices for CM yet. We don't have best practices for code review yet. There are no white papers on this.

After our lunch break we took a quick poll: Only 11 of the 26 present at the time still run their own email service (either at home or offsite), and 9 more have stopped doing so in the past year. One hasn't outsourced because it keeps his skills sharp. Another has his public blog administered by someone in Romania for $50 every time the blogging software needs to be updated.

Our next discussion was on large scalable clustered storage. One company represented generated a lot of data (1TB/day) that they nedd to keep forever, and they see that growing to 10TB/day. The question was asked what people are looking at for data, if they're staying with spinning media or moving towards flash or other solid-state drives. Much depends on your use profile; one site uses EMC Celera for non-parallelized storage, but their profile is user home directories and scientific data in an NFS model. Most people with large storage needs seem to be using GPFS. Other mentioned products include Fusion I/O cards, Infiniband, NetApps, and Violin. The network wonks present wondered about the network behind this large storage; consensus seems to be to use dedicated networks, though some have upgraded their network switches to terabit backplanes. On the subject of fail-over most seem to be failing over the servers but not necessarily the storage independently from the servers.

Next we took a quick lightning round asking what the next useful fad will be in the next two years. Answers included configuration management, death of tape, decline of social networking, increasing use of App Store-like software distribution within companies, infrastructure as a service (IaaS) increasing, IPv6 deployment, JSON APIs, mobile security, more private cloud products, moving away from big iron databases towards NoSQL/MongoDB, moving away from running machines towards providing APIs, moving away from the cloud back to local, SSD not spindles as primary storage, statistical analysis about systems, UI improvements (facial recognition, motion detection, and Siri- or Watson-like interfaces), and virtualization.

We next discussed workstation replacement. Only 4 people said they use virtualized desktops. Some environments reimage the workstation on logout (mainly in public labs), and most seemed to prefer physical workstations due to performance issues. Environments that use spare CPU cycles for processing (such as Condor) prefer physical to virtual workstations for performance reasons. Virtual desktops assume high-bandwidth and low-latency networking between the user and the physical hardware, which is not universally ture. Furthermore most seem to think virtualized desktops don't save money; hardware costs are falling and local processors and capabilities are getting cheaper, so centralizing the services for anything other than administrative overhead may not have a benefit except in areas where power and cooling are your expensive limiting factors.

Next we discussed life balance and stress management. IT culture seems to still be 60-80 hour work-weeks, which leads to a lot of burnout. Some places bought toys like ping-ping tables ("startup mentality"), but we should change the culture more towards mentoring the younger or juniors, learning how to say No despite pressure, and educating management to have them cause less stress. There's a difference between good stress ("I bet you can't do this over the weekend...") and bad stress ("... or you'll be fired on Monday"). At one represented employer it's good to hit or be within some percentage of the service level agreement, but bad to be outside that percentage, even if it's responding too fast. In other words, meet but don't exceed your SLAs.

IT often manages to pull a magic solution when backed into a corner, so expectations are set (perhaps unreasonably) high. One method of push-back is to say "Here's what I'm working on, what do you want me to drop to work on this new thing" and let management make the call. If your work runs into your personal time, you can use some of the work day to recover (such as running errands, making doctor's appointments, and so on). One noted that adding a fitness regime can help with stress as well, though not even half of those present have a regular fitness routine. Another pointed out that there are strict rules for what overtime is allowed in Europe, and there was a brief tangent on cultural differences between US and European time expectations.

One person's employer allows everyone to take one day per month to not come to work (managing their time); the requirement is to remain in town and available if you're needed. They tend to use it for kids' functions or doctor's visits. The president of that company's general attitude is that if there's a crisis of technology, he asks if anyone's going to die if it isn't fixed immediately. The trick is convincing your management of that. However, if management doesn't support having a work-life balance you're working in the wrong place. The final comment was that you get more respect by respecting yourself and enforcing your own work-life balance.

We had a very brief discussion about patents. There have been a lot of lawsuits about technology. One person was subpoenaed in a patent suit between two companies he'd never heard of; having written a PAM module a decade ago was apparently evidence of prior art. Do software patents help or hinder innovation? The way the (US) law is written and the decision is done, except for clear prior art the Patent Office has to grant the patent because something isn't prohibited, which can hinder innovation. How sysadmins look at things is different than how the law is written. LISA is important because papers help show prior art. A couple of years back a commercial company tried to patent configuration management, but Anderson's 1994 paper was prior art such that the patent was denied. The best way to fight this is publish your work: Once it's published it's prior art. However, many are held back by fear of being sued for violating someone else's patent.

Next someone asked if there was management-level publicity about sysadmins going away? Some are seeing a lot of this, in part because developers see that cloud services let them jump right to release. Others noted that the thought of some new technology making sysadmins obsolete has been said for the past decade, such as with autonomic computing. One suggested that we could change the sysadmin role away from "operations drone" towards "architect." With the automation and configuration management tools we have today much of the "mindless" tasks can be automated away, and the sysadmin can take on more of a higher-level architect, designer, or decider role and improve the service and infrastructure. Another idea was to have a gatekeeper between Development and Production, selling that their knowledge of security, process, scalability, and so on is important and relevant. One environment bills every product and service back to the requesting department. It was noted that the real answer depends on the actual cause. What's tickling management's nerves? If it's cost, argue about the cost to the business in the event of outages, in terms of financial impact, publicity, and goodwill.

After the afternoon break, we discussed women in technology. One of our participants is involved in a number of research areas and focus groups. They asked if we're seeing women in technology, if they are showing up in applicant pools (under or overqualified), if we have any outreach ideas, and so on. One environment has a lot of women in both tech and leadership roles and are seeing qualified candidates of both genders, though women tended to be more on the development than the sysadmin side, and almost no women DBA candidates. Another environment has a lot of female developers and project and program management, but practically none in service engineering/SA.

Some say that IT in general has a lot of unfriendliness towards women. One person observed that when we say "We're not creating a hostile work environment" may be untrue. We need to treat candidates or colleagues on solely their technical merits not on their gender. In the past, women needed to be aggressive enough to get past the Old Boys' Network to get in. Also there's the culture of "She's not that good" from guys, which may be subconscious from many men. One person noted that the unconscious gendered-biased behavior is learned. One job he was at had 40% women in technology. They felt little to no bias against them because so many were there it was considered "normal."

One person has been interviewing students for a decade and in that time he's had all of 3 female applicants. He was able to hire 1; the other 2 weren't the best for the job at the time. That one has since left, in part because she didn't have the skillset yet. It's too late if we wait until they're in industry; we need to get them involved earlier. We need to instill the interest in technology at a younger age (college is too late). He sees no non-US females and only a small number of women overall. Most in the room seem to agree that we need more women in the field: We need to get more women (girls) involved in Science, Technology, Engineering, and Math (STEM), especially where there's no tech classes. However we've observed that SEM is easier than T.

Tangentially, someone was triggered to think about statistics by a previous discussion. We're likely to look at more complex metrics over time that are statistically defined (95% of requests under n milliseconds, 99% under m milliseconds, and so on). Running larger scale services you can't use "10% above peak" as a metric. It's also an educational problem. We need to think statistically about latency and capacity and what's "good enough." Similarly we need to move from "I ran this test 10 times and got results of x" towards "I have a 95% confidence that..." is a better metric and one we should move towards. We're also seeing a drive for comparisons (such as this week versus last week) and trending analysis. Several people think this could make a good Short Topics book. The final comment was that you have to know what's normal before you can define abnormal.

We ended the workshop up with our final lightning round, asking what is going to be new-for-you in the next year. Answers included architecture design and development, career planning (finding better jobs, increasing team visibility in a good way, managing the existing job, moving between generalized and specialized), data mining and statistical research, decommissioning old hardware delegating everyday tasks, doing more DevOps type work, hiring more people, managing more data, publishing new books, recreating his environment from the ground up, and training interns. However for some, not much is changing that quickly.

Back to my conference reports page

Back to my professional organizations page

Back to my work page

Back to my home page