Spotify recently documented their progressive approach to scaling Agile development with a fairly large team. Gilt’s approach has many similarities, but since some Spotify best practices are Gilt anti-patterns, it’s worth a closer look.

The Gilt tech team is about 100 strong. Most of our development is done in Manhattan. We’ve also got a small team in Portland, Oregon and a larger one in Dublin, Ireland.

Initiatives

The cornerstone of the Gilt process is the initiative. We define an initiative as “a project that we expect to work on for the foreseeable future.” Our foreseeable future is typically 3-12 months, depending on the area. An initiative might be to increase organic search traffic to the site.

We work on the order of 10 initiatives in parallel. The decision to green light an initiative depends on a number of factors. The most significant factor is a mathematical model of how we expect the initiative to perform, but we also consider softer factors like direct customer happiness and innovation.

Prioritizing a set of initiatives focuses the technology group on an initiative portfolio. This portfolio makes a clear statement on what is important, and indirectly, what is not. This has had a profound effect, halting our previously unending feature level prioritization discussions. This change has made not only our team happier and more productive, but also our stakeholders.

Unlike Spotify, we don’t maintain a roadmap document. Our roadmap is simply the sum of active initiatives. We revisit the set of prioritised initiatives every few months.

Read the full post on The Gilt Tech Blog.

The employee of the month award is pervasive in our corporate culture. Websites offer Step-by-step guides and will sell you blank merit certificates by the hundred. But is it a good idea? The system typically works as follows: Once a month a manager, or committee of managers, picks a single outstanding individual as employee of the month. An announcement is made and the employee enjoys a set of benefits typically including an item of memorabilia. Read the full blog on Employee of the Month Awards.

Google are associated with innovation in every aspect their business. Their management techniques are no exception. Google employ multiple award systems to motivate employees and perhaps the most notable is their peer-to-peer recognition program. This program allows Google employees to proactively recognize their peers for doing something big or small that goes above and beyond the call of duty. Peers often reward an activity that would have gone completely unnoticed by managers.

Read the full blog on Peer to peer recognition at Google.

Is great leadership the key to an organization's long term success?

Check out Jeffrey Cohn's great piece on the subject, where he discusses the need for integrity, passion, courage, vision, judgement, empathy and emotional intelligence from our leaders.

Arguably, the job of an engineering manager is to hire and sculpt a development team that is not only highly productive, but also precisely resourced for immediate business priorities.

Easier said than done. Businesses are highly erratic organisms. Especially start-ups. From week to week our business environment changes and company priorities evolve accordingly. In response to this, so does the position of senior management on the best way to adapt.

It’s always hard to hire great developers. If you want to hire them in volume you need to think outside the box. It looks like the team at LivingSocial and JumpStartLab have done just that with their new Hungry Academy program. They have setup an engineering training academy with some of the “industry’s best programmers” and are offering market compensation to anyone that gets accepted.

Hiring Drupal developers is difficult. Hiring great Drupal developers in the current market often feels close to impossible. They are highly sought after and most of the people on the market, in all honesty, aren’t very good.

I’ve put together a list of the best Drupal interview questions that I’ve used over the years to screen Drupal candidates. Hopefully you’ll find them useful.

I recently had the pleasure of talking at First Capital's CTO summit. I hosted an interactive discussion on team building for around 100 CTOs and VPEs of small to medium sized, venture backed technology companies.

Here's how the audience voted on 12 critical questions:

#1: Should you hire specialists or generalists? (52 Votes)

I spend a lot of my time hiring. Recently I’ve streamlined my technical phone screening. I’ve defined a “Technical Bar” that focuses on seven skill areas.

Read more on my technical phone interviews blog

REWORK, the latest book to emerge from the 37Signals.com, Jason Fried and David Heinemeier Hansson, delivers some strong opinions on hiring.

I loved the book (and the attack ad below promoting it). I spend a good deal of my time striving to hire world-class web developers, and much of what Jason and David had to say resonated with me.

I cracked a smile at the book’s opening assertion; resumes are ridiculous documents, beloved by the mediocre, filled with half-truths and exaggerations and perfect for spamming hundreds of potential employers. And my heart warmed at the suggestion that cover letters, on the other hand, are potential gems, written in the voice of the candidate and often specifically directed at the position in question. I always read the cover letter first.

Read the full blog

As a software system grows, it's important to start evaluating new features not only on how hard they are to build but how hard they are to maintain and most importantly, test. If you want a reasonable assurance that a feature is going to work correctly after each new release, then somebody (or something) needs to test that new feature before each release.

More features means more testing for each and every release, and therefore a system gradually gets more and more expensive.

Kiva Engineering is an Agile shop. We're better at some parts of Agile then others, but we hold true and fast to the principle of "Release Fast, Release Often". We aim to release every two weeks, and for the past 9 months our largest deviation from that has been a delay of one day.

this entry was originally posted on build.kiva

At Kiva, our development process is based on Scrum – with work broken into two week Sprints (we call them “Iterations”). We’re good students of Scrum in some regards (we do release every two weeks, without fail), and bad students of Scrum in others (it seems like we still bite off more than we can chew in each iteration). Like most Scrum teams, the center of our universe is our ticket tracking system. We use Redmine – albeit a rather customized version. We’ve put quite a bit of time and effort into our ticket tracking system – so we thought we’d give you a tour.

Few people would argue the importance of great communication across a development organization. It’s vital that developers are aware of the work of their peers, not only the “what”, but also the “how” and “why”. Achieving this is easier said than done. Traditional approaches such as wiki documentation, meeting minutes, email lists and chat channels have value, but also have significant drawbacks.

this entry was originally posted on build.kiva

At Kiva, we tend to attract and hire engineers with a strong creative, entrepreneurial streak. This aligns well with Kiva’s office culture – and with our overall mission – an innovative, technology-driven approach to alleviating poverty. However, as the team grows and we become more specialized in an effort to become more efficient, sometimes our day to day work doesn’t always give us an outlet for our creative sides. After a long project involving concepts like currency exchange loss, it’s easy to feel like you spend your entire working life chasing a few misbehaving pennies around the globe, and that your particular slice of the Kiva pie has become a little divorced from the glorious big picture.

In June 2010, many of us were finishing up some pretty tough projects, and as a team, we were in a bit of a funk. We talked as a team about what we could do to restore our energy and excitement. Our solution was an idea called “Innovation Iteration”.

this entry was originally posted on build.kiva

2010 was a year of growth for Kiva Engineering. We started the year with 8 full time engineers (and 2 managers) and ended the year with 14 engineers – a growth rate of 75%. We also kicked off 2010 with a switch to a new ticket tracking system (Redmine), that allowed us to more accurately track exactly what we’ve been up to. Let’s take a look shall we….

Most companies with a shared refrigerator have a strict policy designed by a cunning admin to keep the fridge useful. A common approach is as follows:

Each week:

1. Ask employees to claim their items in the fridge by writing their name on it.
2. Wait 1 day.
3. Throw away any unclaimed item.

We had another couple of great external speakers talking at Digg HQ. The open-source master Doug Cutting stopped by the Digg office to talk about the growth of the Hadoop platform. He talks in detail about Avro. Avro is an RPC and serialization framework that has some interesting differences compared to the popular Thrift framework.

solr lucene is good technology to use for searching over a corpus of tweets. if you take the content of a tweet and dump that into the default solr lucene "text" field, you'll do pretty well. however, if you look at your results closely, you'll find one subtle, but very annoying problem: searches on a hashtag term will match the non-hashtag term.

Digg's Andrew Bayer has just written a blog describing how we use Git, Hudson, Selenium, Puppet and Gerrit to manage continuous deployment at Digg.

Andrew describes how we get developer commits to production quickly and safely using a combination of automated packaging and staging, web based code review and automated testing (unit and selenium)

Read the full blog here.

We recently started inviting external speakers to talk at Digg HQ on a variety of Technical subjects. Jack Dorsey the founder of Twitter and Square stopped to talk about the growth of Twitter and the resulting engineering and cultural challenges. He talks about applying the lessons learned there to his latest venture, Square. Fascinating stuff.

We've recently started using Drupal 6 at Digg.com for all our content needs. So far so good. Everything from our jobs page, to our site tour, to the Open Source site we launched three days ago, is managed via Drupal. Read more about our use of Drupal on the Digg Blog.

I gave the keynote presentation at Under the Radar's "Commercializing the Cloud" conference, It's about NoSQL and picking distributed database technologies that might help you with the "deluge of data" that the world is experiencing. I've included the presentation below.

André Maurois (1885-1967) wrote that "The effectiveness of work increases according to geometric progression if there are no interruptions." At Digg we struggle between the clear benefits of uninterrupted work and the need to be agile in our communication.

The last six months have been exciting for Digg's engineering team. We're working on a soup-to-nuts rewrite. Not only are we rewriting all our application code, but we're also rolling out a new client and server architecture. And if that doesn't sound like a big enough challenge, we're replacing most of our infrastructure components and moving away from LAMP.

both the 5.x and 6.x versions are now available for download on github. sorry, i just can't do CVS anymore. to download:

1. start by going here: http://github.com/cailinanne/log4drupal
2. then click the all tags drop-down and choose the appropriate version
3. then click the download button

a full description of the module is available here

i've noticed two trending topics in the tech world today: social graph manipulation and map-reduce algorithms. in the last blog, i gave a quickie guide to setting up hadoop, an open-source map-reduce implementation and an example of how to use hive - a sql-like database layer on top of that. while this is one reasonable use of map-reduce, this time we'll explore it's more algorithmic uses, while taking a glimpse at both of these trendy topics!

last week, i had the good fortune to attend the san francisco nosql meetup organized by johan oskarsson.

the meetup discussed the limitations of traditional relational database technology at scale and the open-source alternatives currently available with similar functionality to amazon's dynamo google's bigtable.

if you're exploring hive as at technology, and are looking to move beyond "hello, world", here's a little recipe for a simple but satisfying first task using hive and hadoop. we'll work through setting up a clustered installation of hive and hadoop, and then import an apache log file and query it using hive's SQL-like language.

unless you happen to have three physical linux servers at your disposal, you may want to create your base debian linux servers using a virtualization technology such as xen. for a good guide on setting up xen, go here. for the remainder of this tutorial, i'll assume that you have three debian (lenny) servers at your disposal.

let's get started