This post is a guest post from Breanden Beneschott, Co-Founder at toptal.com. Toptal is an exclusive, global marketplace for top software developers and top companies. The author’s views are entirely his own and may not reflect the views of Higherclick KFT.

The software development industry is both enormous and rapidly growing. DataMonitor predicts that the global software market will reach  $299.1 billion in 2014—an estimated increase of 32.6% from its 2009 statistic. The growth in this industry is fueled by innovation and the creation of new companies as well as from the successes of older and more established companies.

Along with the rapid growth of the software industry is the ever-increasing demand for software engineers. However, this intricate industry consists of more than high tech start-up companies in need of able developers. Today, almost every company needs IT or software engineering talent in some capacity—from a Fortune 500 company requiring IT infrastructure to support its millions of daily transactions to a small business attempting to exploit the convenience of ecommerce and a website. With the interweaving of technology and business has come an industry demand for software engineers that far exceeds the readily accessible supply. Further complicating the search for developers are the facts that the United States itself churns out very few new engineers annually, and the good engineers are very expensive and hard to find. Google, for example, knows just how valuable talented developers are and consequently hoards the best developers through the employment of large salaries, big benefits, and lots of freedom.

The lack of adequate engineers has caused many companies to outsource their work overseas in order to meet the rising demands and cut costs. Outsourcing firms such as Infosys and websites like Elance, oDesk, and Guru have contributed largely to the oversea outsourcing fervor as they allow companies to easily locate, hire, and manage talent remotely.

The problem is that the outsourcing model is broken, which is evident by the fact that over 58% of outsourced IT functions end in failure that costs companies billions of dollars each year.

When a company decides it needs to hire an onsite PHP developer to build a new feature for a social gaming website, for example, the traditional route is as follows:

1. Start by asking their personal network for recommendations or if they know any “ninjas or rockstars” who are available (though, it must be noted that this is akin to asking if someone has spare money to bestow upon you—good luck).
2. Post on CraigsList where they will receive hundreds of applications from unqualified people in Pakistan, India, etc., and even if they do get some good applications, chances are they wont notice them through all of the noise and/or because the best engineers are often the least aggressive self-marketers.
3. Pay to post on LinkedIn, Monster, or dozens of other websites (where they will likely receive the same results as they did to their CraigsList post).
4. Pay a king’s ransom to a headhunting firm to find someone too expensive and who may or may not be an effective worker.

The end result of this process will be months wasted, risky hires made, and tens of thousands of dollars lost directly (with potentially millions of dollars lost indirectly).

But suppose that the company decided to try outsourcing. After all, it is cheaper and the talent pool is larger. They would, for a single hire, perhaps go to a site such as Elance or oDesk. Elance, for example, has 1,365,264 freelancers who have done $495,990,005 in business. All a company has to do is find someone with a good rating, who has done several jobs, and who has passed the Elance PHP tests with flying colors, right?

Not exactly.

Every metric and filter on sites like Elance and oDesk have been bested by bad developers. Bad developers can simply pay to have poor ratings removed (if their previous clients even bother leaving them at all) or create a new profile. They can also Google the answers to the online skills tests while they take them (which translates into the ironic statistic that the worst developers often score >90% on these tests while the best developers often score between 60-80% because they performed their evaluations honestly). Furthermore, poor developers can post jobs earned by nepotism rather than by actual merit or simple projects in order to increase their statistics.

Therefore, while brilliant software engineers are in high demand, they are nearly impossible to find—especially when non-technical people are doing the searching.

In the following quote, Paul Graham, esteemed essayist and founder of the famously successful YCombinator incubator in Silicon Valley, points out just how detrimental the lack of supply of good developers can be to the industry and why the current system is broken:

When I think about what killed most of the startups in the e-commerce business back in the 90s, it was bad programmers. A lot of those companies were started by business guys who thought the way startups worked was that you had some clever idea and then hired programmers to implement it. That’s actually much harder than it sounds—almost impossibly hard in fact—because business guys can’t tell which are the good programmers. They don’t even get a shot at the best ones, because no one really good wants a job implementing the vision of a business guy.

So how does a person go about picking quality programmers when they themselves are not a programmer? As far as I know, there isn’t really a good answer to this question. They would probably have to hire a good programmer who could then evaluate and hire other good programmers. But if they can’t recognize good programmers in the first place, how would they even perform this preliminary task?

The biggest industry obstacle would seem to be finding and hiring good engineers, as the current methods are broken, the supply is diminishing, and the demand will only rise further as the industry continues to expand.

These are the problems we are working to solve at toptal. We believe part of the solution relies on making sure that you always have great developers doing the searching and screening for other great developers. Within our company, teams of elite software engineers work in countries such as Columbia, Brazil, Argentina, Hungary, and Russia as toptal recruiters (often while completing advanced degrees or post doctoral studies). Their commission-based job is to find people like themselves—whether that be at international mathematics or TopCoder competitions, through GitHub profiles or obscure blog posts, or from mythical rumors of a genius hacker camped out in a basement somewhere. The origin, education, or age of these developers is not important as the toptal assessment process is designed to find the best and quickly weed out everyone else. After having worked with hundreds of developers across the world, we’ve developed the following screening process:

1. A rigorous, timed algorithm test
2. Multiple technical interviews with other established developers
3. A verbal interview to assess English proficiency and personality traits
4. A paired programming project

Of course screening software developers is still an art and a science — it’s also a lot of work. If you have questions about our processes or our company, feel free to shoot me an email: breanden@toptal.com.

If you enjoyed this article, please share it around .