Working with Time Series Plots in R

john — Tue, 09 Apr 2013 02:28:15 -0700

Life in Gilt Tech involves much lively discussion on KPIs, consequently we spend a lot of time looking at time series data.

We use a variety of tools to manipulate and present this type of data, including R, a language and environment for statistical computing and graphics.

To help us through the process of standardizing the way we represent our KPIs we have developed a simple, R based open-source utility for easily representing time series data originating from a variety of sources.

If you think you might find this useful, or you’d like to contribute, check out the time series plots in R on GitHub or read the full blog here

Scaling Agile at Gilt

john — Thu, 07 Mar 2013 09:29:44 -0800

Spotify recently documented their progressive approach to scaling Agile development with a fairly large team. Gilt’s approach has many similarities, but since some Spotify best practices are Gilt anti-patterns, it’s worth a closer look.

The Gilt tech team is about 100 strong. Most of our development is done in Manhattan. We’ve also got a small team in Portland, Oregon and a larger one in Dublin, Ireland.

Initiatives

The cornerstone of the Gilt process is the initiative. We define an initiative as “a project that we expect to work on for the foreseeable future.” Our foreseeable future is typically 3-12 months, depending on the area. An initiative might be to increase organic search traffic to the site.

We work on the order of 10 initiatives in parallel. The decision to green light an initiative depends on a number of factors. The most significant factor is a mathematical model of how we expect the initiative to perform, but we also consider softer factors like direct customer happiness and innovation.

Prioritizing a set of initiatives focuses the technology group on an initiative portfolio. This portfolio makes a clear statement on what is important, and indirectly, what is not. This has had a profound effect, halting our previously unending feature level prioritization discussions. This change has made not only our team happier and more productive, but also our stakeholders.

Unlike Spotify, we don’t maintain a roadmap document. Our roadmap is simply the sum of active initiatives. We revisit the set of prioritised initiatives every few months.

Read the full post on The Gilt Tech Blog.

Employee of the Month Award - Good or Bad?

john — Mon, 08 Oct 2012 11:58:32 -0700

The employee of the month award is pervasive in our corporate culture. Websites offer Step-by-step guides and will sell you blank merit certificates by the hundred. But is it a good idea? The system typically works as follows: Once a month a manager, or committee of managers, picks a single outstanding individual as employee of the month. An announcement is made and the employee enjoys a set of benefits typically including an item of memorabilia. Read the full blog on Employee of the Month Awards.

A look at Google’s Peer-to-Peer Bonus System

john — Sat, 06 Oct 2012 12:48:46 -0700

Google are associated with innovation in every aspect their business. Their management techniques are no exception. Google employ multiple award systems to motivate employees and perhaps the most notable is their peer-to-peer recognition program. This program allows Google employees to proactively recognize their peers for doing something big or small that goes above and beyond the call of duty. Peers often reward an activity that would have gone completely unnoticed by managers.

Read the full blog on Peer to peer recognition at Google.

Leadership

john — Sun, 26 Feb 2012 06:19:49 -0800

Is great leadership the key to an organization's long term success?

Check out Jeffrey Cohn's great piece on the subject, where he discusses the need for integrity, passion, courage, vision, judgement, empathy and emotional intelligence from our leaders.

Managers! Become the Flywheel.

john — Mon, 13 Feb 2012 08:43:49 -0800

Arguably, the job of an engineering manager is to hire and sculpt a development team that is not only highly productive, but also precisely resourced for immediate business priorities.

Easier said than done. Businesses are highly erratic organisms. Especially start-ups. From week to week our business environment changes and company priorities evolve accordingly. In response to this, so does the position of senior management on the best way to adapt.

The Hungry Academy - An Innovative Way to Hire Developers

john — Mon, 09 Jan 2012 06:39:54 -0800

It’s always hard to hire great developers. If you want to hire them in volume you need to think outside the box. It looks like the team at LivingSocial and JumpStartLab have done just that with their new Hungry Academy program. They have setup an engineering training academy with some of the “industry’s best programmers” and are offering market compensation to anyone that gets accepted.

Hiring Great Drupal Developers - Buyer Beware!

john — Fri, 18 Nov 2011 08:44:32 -0800

Hiring Drupal developers is difficult. Hiring great Drupal developers in the current market often feels close to impossible. They are highly sought after and most of the people on the market, in all honesty, aren’t very good.

I’ve put together a list of the best Drupal interview questions that I’ve used over the years to screen Drupal candidates. Hopefully you’ll find them useful.

Building Great Tech Teams: 12 Critical Questions

john — Mon, 17 Oct 2011 09:39:06 -0700

I recently had the pleasure of talking at First Capital's CTO summit. I hosted an interactive discussion on team building for around 100 CTOs and VPEs of small to medium sized, venture backed technology companies.

Here's how the audience voted on 12 critical questions:

#1: Should you hire specialists or generalists? (52 Votes)

Technical Phone Interviews

john — Wed, 05 Oct 2011 03:20:29 -0700

I spend a lot of my time hiring. Recently I’ve streamlined my technical phone screening. I’ve defined a “Technical Bar” that focuses on seven skill areas.

Read more on my technical phone interviews blog

REWORK your Hiring Practices

john — Tue, 02 Aug 2011 18:31:04 -0700

REWORK, the latest book to emerge from the 37Signals.com, Jason Fried and David Heinemeier Hansson, delivers some strong opinions on hiring.

I loved the book (and the attack ad below promoting it). I spend a good deal of my time striving to hire world-class web developers, and much of what Jason and David had to say resonated with me.

I cracked a smile at the book’s opening assertion; resumes are ridiculous documents, beloved by the mediocre, filled with half-truths and exaggerations and perfect for spamming hundreds of potential employers. And my heart warmed at the suggestion that cover letters, on the other hand, are potential gems, written in the voice of the candidate and often specifically directed at the position in question. I always read the cover letter first.

Read the full blog

Managers, stay out of the way!

john — Mon, 25 Apr 2011 17:44:10 -0700

Successful managers learn how to stay out of the way and let their teams work effectively and independently. Staying out the way doesn’t mean putting your feet up on your desk and playing Angry Birds on your iPad while your team does all the hard work. It means creating an environment where smart people feel empowered to recognize, own and solve problems.

Here are some ideas to help you as a manger. I’ve included a few quotes from General George Patton, since apparently even our great military leaders advocate staying out of the way.

Gilt's 5@5

john — Mon, 14 Feb 2011 05:59:37 -0800

Few people would argue the importance of great communication across a development organization. It’s vital that developers are aware of the work of their peers, not only the “what”, but also the “how” and “why”. Achieving this is easier said than done. Traditional approaches such as wiki documentation, meeting minutes, email lists and chat channels have value, but also have significant drawbacks.

Optimizing your iPad and iPhone applications

john — Wed, 09 Feb 2011 05:25:06 -0800

The Gilt mobile team have been making a series of optimizations to the iPad and iPhone application over the last few months. Check out the video of the improvements and find out the details behind them on the Gilt Tech Blog.

Purge Your Bug Database

john — Sat, 18 Dec 2010 06:31:46 -0800

Most companies with a shared refrigerator have a strict policy designed by a cunning admin to keep the fridge useful. A common approach is as follows:

Each week:

Ask employees to claim their items in the fridge by writing their name on it.
Wait 1 day.
Throw away any unclaimed item.

More Digg Technical Talks

john — Mon, 16 Aug 2010 15:46:24 -0700

We had another couple of great external speakers talking at Digg HQ. The open-source master Doug Cutting stopped by the Digg office to talk about the growth of the Hadoop platform. He talks in detail about Avro. Avro is an RPC and serialization framework that has some interesting differences compared to the popular Thrift framework.

Continuous Deployment at Digg.com

john — Thu, 22 Jul 2010 16:45:03 -0700

Digg's Andrew Bayer has just written a blog describing how we use Git, Hudson, Selenium, Puppet and Gerrit to manage continuous deployment at Digg.

Andrew describes how we get developer commits to production quickly and safely using a combination of automated packaging and staging, web based code review and automated testing (unit and selenium)

Read the full blog here.

Digg Technical Talks

john — Mon, 05 Jul 2010 08:41:44 -0700

We recently started inviting external speakers to talk at Digg HQ on a variety of Technical subjects. Jack Dorsey the founder of Twitter and Square stopped to talk about the growth of Twitter and the resulting engineering and cultural challenges. He talks about applying the lessons learned there to his latest venture, Square. Fascinating stuff.

Digg and Drupal

john — Mon, 19 Apr 2010 17:52:35 -0700

We've recently started using Drupal 6 at Digg.com for all our content needs. So far so good. Everything from our jobs page, to our site tour, to the Open Source site we launched three days ago, is managed via Drupal. Read more about our use of Drupal on the Digg Blog.

Dealing with the Data Deluge

john — Sat, 17 Apr 2010 13:59:31 -0700

I gave the keynote presentation at Under the Radar's "Commercializing the Cloud" conference, It's about NoSQL and picking distributed database technologies that might help you with the "deluge of data" that the world is experiencing. I've included the presentation below.

A Geometric Progression of Effectiveness - The Agility of Interruptions

john — Tue, 16 Mar 2010 13:14:11 -0700

André Maurois (1885-1967) wrote that "The effectiveness of work increases according to geometric progression if there are no interruptions." At Digg we struggle between the clear benefits of uninterrupted work and the need to be agile in our communication.

Saying Yes to NoSQL; Going Steady with Cassandra

john — Wed, 10 Mar 2010 12:12:30 -0800

The last six months have been exciting for Digg's engineering team. We're working on a soup-to-nuts rewrite. Not only are we rewriting all our application code, but we're also rolling out a new client and server architecture. And if that doesn't sound like a big enough challenge, we're replacing most of our infrastructure components and moving away from LAMP.

san francisco nosql meetup

john — Mon, 15 Jun 2009 09:16:52 -0700

last week, i had the good fortune to attend the san francisco nosql meetup organized by johan oskarsson.

the meetup discussed the limitations of traditional relational database technology at scale and the open-source alternatives currently available with similar functionality to amazon's dynamo google's bigtable.

smartly purge your old backup files on linux

john — Mon, 23 Feb 2009 21:08:13 -0800

if you backup your *nix box, eventually you'll get into the business of purging your old backup files to preserve disk space. a reasonable way to do this is to use the find command to identify old backups and delete them. you should, however, consider doing something a little smarter than this.

creating an agile engineering work space at digg

john — Tue, 06 Jan 2009 21:22:11 -0800

digg has been growing like crazy, creating space problems for our development department, resulting not only in excessive density, but in our team being spread across several floors. these space issues have caused significant challenges for our emerging agile development environment, stifling informal communication and making it difficult to organize teams into cohesive groups.

amazon release their elastic block store, ebs

john — Tue, 26 Aug 2008 06:42:33 -0700

a while ago i posted some performance benchmarks for drupal running on a variety of servers in amazon's elastic compute cloud.

amazon have just released ebs, the final piece of technology that makes their ec2 platform really viable for running lamp stacks stuck as drupal.

ebs, the "elastic block store", provides sophisticated storage for your database instance, with features including:

high io throughput
data replication
large storage capacity
hot backups using snapshots
instance type portability e.g. quickly swapping your database hardware for a bigger machine.

laying the agile groundwork at digg

john — Tue, 22 Jul 2008 20:51:03 -0700

i'm currently working on re-engineering many of the development processes at digg. we're adopting a number of practices from the agile world that complement the type of development that we do. these practices include: build automation, automated deployment, daily scrums, short releasable time-boxed iterations, simple design, refactoring, and just-in-time specification.

we've decided to be agile about our adoption of our new agile processes, introducing them incrementally, measuring the results along the way and iterating as necessary.

as we worked through the cascading dependencies of our adoption path, it quickly became clear that automated testing had to be a cornerstone practice if we were going to make a success of the others.

a new jmeter book from packt

john — Wed, 09 Jul 2008 14:26:40 -0700

recently i posted a couple of introductory articles on jmeter, a great apache open-source tool that allows you to measure the performance and scalability of a wide variety of services, especially web-applications.

i wrote these articles because although the online documentation provides reasonable reference material, it doesn't serve well as a jmeter introduction or tutorial.

things have changed a bit since then. the uk-based publishing house packt publishing were kind enough to send me a copy of emily halili's newly published book on jmeter, which is as far as i can tell, is the first book dedicated to the subject.

digg made me an iPerson

john — Thu, 12 Jun 2008 19:49:17 -0700

i started working at digg.com a few months ago. on my first day i was handed a brand new macbook pro. a reasonable person would have been delighted, but i was filled with dread. i had started a new high-pressure job, and the only tool at my disposal was a mac. not only that, i shuddered at the very thought of becoming an iPerson.

my most recent experience with apple had been extended subjection to a dog-slow powerpc running mac os 7. after which, i have always assumed macs to be fisher-price type devices, designed for those too misguided or incompetent to operate even a second-rate device like a wintel box.

lamp on amazon ec2 shaping up nicely

john — Mon, 14 Apr 2008 21:02:46 -0700

recently i posted some encouraging performance benchmarks for drupal running on a variety of servers in amazon's elastic compute cloud. while the performance was encouraging, the suitability of this environment for running lamp stacks was not. ec2 had some fundamental issues including a lack of static ip addresses and no viable persistent storage mechanism.

amazon are quickly rectifying these problems, and recently announced elasic ip addresses; a "static" ip address that you own and can dynamically point at any of your instances.

today amazon indicated that persistent storage will soon be available.