Bryan Cantrill (VP of Engineering at Joyent) gave an excellent and entertaining presentation about Debugging Production Systems. It is well worth an hour of your time if you work on large distributed cloud stacks in production. Some of the key things I took from the video:
Every failure is sacred. Seek to build a system that properly stores and formats stack traces for future analysis.
Cloud architectures are full of abstraction layers where “ungentlemanly” failures occur. Instead of crashing, things stay up, but performance is degraded.
It is these non-fatal pathologies that are particularly difficult to debug. They often lead to cascading levels of system instability and throw the system into untested states. It is this spiral that is often behind most airline disasters – a recoverable, non critical failure occurs then various failover/automatic systems either over-react or under-react then the airplane becomes difficult to control and systems no longer make sense so serious human error occurs.
When looking at a core dump, you are acting as a scientist and you are testing hypotheses and proving theories with data. When looking at non-fatal pathologies, you are acting as a physician and treating symptoms. Debugging this way makes it very difficult to determine root cause.
The talk ends with a small demo into the capabilities of DTrace. Amazing core level magic is going on there that can get static snapshots, enable transient failures to manifest as hard failures and generally reduce the effort required to gather failure data in dynamic systems.
Adrian Holovaty of Django project fame has launched an HTML5 interactive music notation app that works really well. Back when I was learning how to play, everyone owned a Marantz tape deck that you could slow down to “half-speed” (optimally an octave lower) in order to transcribe solos.
These were durable machines but they were expensive and they tended to make great music sound like it has been run through a Freddy Krueger varispeed filter. This sonic alteration made the transcribing experience somewhat of an uninspiring slog that only the most tenacious could endure on a regular basis. You also still had to fumble around with the physical sheet music and try to keep up with the recording. Soundslice actually ‘plays’ the sheet music in front of you where you can watch the printed notes scroll along and become sound. When you slow it down, you do not lower the pitch.
Music notation at its best should offer a concise map that describes what you should be playing or hearing. More often than not, especially with jazz solos, notation can become its own cryptic puzzle that seems to lose critical meaning disconnected from the nuances of the original recording. Soundslice seems to solve many parts of this issue. One feature I would like to see is the ability to loop over a selected subset of bars as this is something I constantly need when figuring out a tricky part but this is a very strong start and an impressive web UI.
It is often said that great start-ups are born when the founders are the users and the problem to be solved is going to make that founder/user person happy. You get some important related things for free with that setup:
A very tight and inexpensive user-product feedback loop
Version dogfooding by the same highly invested people during critical early iterations.
The state of mind of an ‘imagined unicorn user’ does not have to be imagined at all but is instead the direct feedback of a real person in the room building the product.
How do large companies create this loop? Is it even possible? It seems to require support for internal skunksworks projects (as opposed to hiring consultants and outside agencies) that is tasked to solve real problems that the company is experiencing. One shining example of such a project becoming a product is the internal Amazon project that is now AWS.
I may love geeking out and working on big complex systems but I also appreciate business simplicity and good content. I have found Scott Allen’s AngularJS courses to be very useful and it turns so do a lot of other people. Scott has found success paring his considerable teaching talent with simple onscreen videos and distributing them on platforms that enable wide on-demand distribution. This is a profitable solo business that does not require running complex propriety stacks – just a laptop, a good microphone and some talent.
I was interested to see that “Senior Web Developer” is the forth unhappiest job in the world according to Forbes. I have had this job several times in several industries. Indeed, these were often dirty, frustrating jobs – especially back when web standards were non existent and IE ruled supreme. But fourth unhappiest of all – really? Have you ever cleaned a deep frier at a chain restaurant or worked nights at a call center? Surely those must be unhappier jobs. Web development is pretty fun these days unless you work at a place that has no soul, no budget or no sense of what the internet is for. ..or is that just me?
My friend Daragh from Cork recently spun a tasty set that I have listened to 3 times now. Daragh has a visual artist’s sense of sonic texture and is fearless about allowing a song out of its genre box to make new trouble. Check it out here.
Jazz Wax just posted a timely piece on Sonny Stitt on his interesting use of the experimental Varitone or Electric Sax. The goal of this 1965 instrument was to integrate the organic reed vibration sound generation process and electric manipulation and amplification of the sound. The instrument would provide a suite of on-board tools and would “get out of its own way” during performance – no tripping over microphone stands, fiddling with outboard effect patching or trying to double lines with a synthesizer to make a custom “modern” sound. Instead, the musician would have these options built into the instrument itself and these processes would be integrated into act of playing. Sonny Stitt was a daring experimenter and a believer in this idea. He was able to make some interesting music with this instrument despite its many technical limitations.
The overarching goals of the WordPress admin has seemed to me to be similar – integrate the content creation process with the web publishing and layout process such that the tools “get out of the way” and creative workflow is not only not compromised but at times enhanced. These are lofty goals and they are never achieved in one iteration. Many of the problems that were so difficult to solve for the Varitone would be trivial to solve now but others are still persist. I own a digital piano that is in many ways a superior instrument to most analog pianos, yet at times it still is not a real piano and then it fails badly. Technology is not cool in and of itself but only if it improves something. Sometimes complex technology goes largely unnoticed because it is so well integrated into something that someone already does. WordPress 3.3 is an important release because it smooths out a lot of rough admin usability edges, gets out of the publisher’s way more than ever, and makes it more fun to publish stuff. Sonny would enjoy playing with his release namesake I am sure.
An old essay called The Tyranny of the Extroverts resurfaced in my feed reader the other day. It got me thinking about some of the talented and shy developers and musicians I have had the privilege to work with over the years. These are the kind of people who trouble getting past an HR interview because of shyness or awkwardness. It scares me to think that a general gatekeeper might throw a potentially valuable candidate out the door before anyone else on the tech team would have a chance to talk to them because s/he seemed “weird” or “too quiet” or was unable to engage in tedious small talk involving American team sports. I consider myself to be a core introvert who can pass as a quasi-extrovert. It is impossible to be a manager without conjuring some style of extraversion from somewhere. My soul though remains allied to the introvert. The most difficult skill I have ever developed- the piano, was something I worked on and still continue to work on mostly alone. My great uncle was a renowned writer. He wrote alone for hours every day. When a talented software developer tells me s/he wants some time alone in a secret wi-fi treehouse bunker to think something through and bang out an idea or two I am inclined to make that happen.
The “sideman” is not a member of the band. He is the faceless hired gun who willingly accepts the lesser piece of the action despite often making a vital contribution the sound. Some great musicians ultimately emerged from these behind-the-scene roles to become stars in their own right but many more did not. This is a labor day salute to some of my favorite obscure records that were fronted by great sidemen who did not quite make stardom.
James Burton: The Guitar Sounds of James Burton 
Burton is the ultimate session guitar player. He is probably best known for his excellent guitar work with Elvis Presley in the 70s Vegas era but he also worked with Elvis Costello, Buffalo Springfield, Gram Parsons, Ricky Nelson and Joni Mitchell among others. His signature axe is a blonde Telecaster. This record was not a big commercial success has got some really nice playing on it. There is a great Burton video where he talks about his various picking styles and influences:
Melvin “Wah Wah Watson”: Elementary 
I first became aware of Wah Wah Watson from his ultra funky guitar work on the 70s Herbie Hancock records but then his name started to pop up everywhere. The funky bluesy stuff on Marvin Gaye’s Let’s Get it On? Wah Wah. The groovy licks on Michael Jackson’s Off the Wall? Wah Wah. He has also done work for Blondie, Stevie Wonder, you name it. This record is really fun to listen to when you are looking to hear something in that late 70s funky fusion pocket without heavy foreground.
Dennis Coffey : Hair and Thangs 
Coffey is a true Detroit Soul session guitar superstar. He laid down countless gritty Motown tracks including Diana Ross, The Temptations and Edwin Starr. His agility in many styles from Rock to Jazz to Funk and his innovative use of the wah-wah pedal enabled him to introduce many subtle genre cross pollinations in his session work. Coffey was the first white artist to be invited to perform on Soul Train. His first solo record is a classic.
Nicky Hopkins : The Tin Man Was a Dreamer 
Nicky Hopkins performed session work for the Beatles, Kinks, Who, Jeff Beck Group, Steve Miller Band, Jefferson Airplane and all of the Rolling Stones records from 1967-76. It is almost impossible to listen to a “Classic Rock” playlist for even a few minutes without encountering his nimble playing. His first solo effort of three is an underrated gem.