Why should you NOT choose a software vendor based on their estimates?

Most people, when looking for a software vendor to build their custom products, automatically ask potential suppliers for workload estimates. How long will it take, how much will it cost, to create the product they want. Next, they compare vendors based on the numbers they provide.
Fortunately, most of them (more often those who are not doing it for the first time) are not choosing the cheapest suppliers all the time. Otherwise, the percentage of IT projects failures would be much higher than nowadays.
Remember, there will always be someone who says that they can do it cheaper.

On the other hand, the highest price does not mean that the vendor delivers the best quality services.

How are the projects estimated by software vendors?

Usually, To provide a reasonable estimate, a software vendor will ask you to specify your requirements. They may even offer you a business analyst service to help with that. Some of them will also help for free, hoping that you will stay with them thanks to that favour.

“The better the specification, the higher the probability that the estimates will be more accurate”. Does it sound reasonable? Of course, it does! Unfortunately, it does not make sense at all, and most probably, you have just wasted days if not weeks on preparing useless specifications. But let’s leave it for now, I will explain later on why it does not make sense… Keep on reading…

So, given that you provide quite nice, extensive project scope specification to the vendor. How are they going to turn this specification into a number that represents the cost and/or time that is needed to deliver the product?

In most cases, the process looks similar. Some project managers sit down together with some (lead) developers or with the entire dedicated software development team. They discuss the scope, collect some questions, do some research on possible solutions, split the scope into estimable requirements and guess how long would it take to deliver each requirement. For each requirement, they make some mistakes, for some of them their estimate way too high for others is way too low but according to the Law of Large Numbers most of the mistakes are cancelling each other, and the total sum of estimates is more or less accurate. After that, the project manager multiplies the results by the “empirical experience factor”, and voila, your estimates are ready.

The “empirical experience factor” is the number usually from 0.8 – 3.5 that the project manager knows from his empirical experience of working with the team that is going to deliver the project. In other words, this is an accuracy of the estimates done by the estimators in the past. The number could be lower than one if the manager together with a salesperson found out that the client might be willing to increase the initial budget after the part of the work will be delivered (vendor lock).

There are plenty of techniques to go through the estimation process. Starting from using Story Points and playing Planning Poker (Story Points are then transposed into time estimates [SIC!]), through affinity mapping and t-shirt sizes till empirical data of average cycle time in Kanban method multiplied by the number of decomposed requirements. But that is the topic for separate articles…

What is the definition of a successful project from a software vendor perspective?

What defines the right software vendor? If your vendor can deliver the agreed scope on time in a given budget and with decent quality (assuming that as a client you can verify all the quality aspects), does it mean that they are automatically a right vendor? According to most clients’ definitions, they are perfect vendors.

From the software vendor perspective delivering agreed scope on time, below the budget, in a way that the client will be satisfied with the quality of the final solution is for sure the success definition. Of course, the software vendor has to aim to keep their costs below the total client’s budget because they need to have some margin on the top of the costs to make some profit.

The client gets what they ordered on time. They pay no more than they expected so they should be happy with that. And usually they… well… often, just after the project is finished, they are quite satisfied, since they have heard about so many project failures in the past. So many projects that were over time, over budget and delivered with low quality and here they get what they ordered at a price that was known from the very beginning – that sounds great.

But after time, their happiness often turns into misery…

What is a successful project from the buyer perspective?

If you are paying for building your software product, your success definition, even if unspoken aloud, is usually quite different from “it was delivered on time, within the budget and with a low number of bugs”. What most probably you care the most about is how much money you will make/save thanks to that product? Regardless if you are a startup developing a new solution or a corporation performing digital transformation at the end of the day, the only thing that matters is how much money your product will make/save for you one way or another. The money you saved by choosing “the right vendor” that was able to deliver the product on time, within the budget was a small success (but still an achievement of some kind) compared to the money you are aiming for with your product in the future.

What usually happens a couple of weeks or months after the “successful” project delivery? Often, after the product release, when users start using it, some part (most?) of the scope, so carefully defined and well estimated, then developed with high-quality standards, now is not surviving the clash with the market and real users’ needs and expectations.

Usually, it occurs that the budget you have spent on the product was just a fraction of what you need to satisfy a significant number of customers using your product and you need way more than that now to adjust and extend your product to meet their needs. It is worth mentioning that, meanwhile, your competitors also moved forward and you have to make moves with your product to simply catch up after the market.

Hopefully, your product already gets some traction and starts making money that you can spend on additional development. Probably, the quality of the product you paid for was really good, and it is not a problem (called Technical Debt) to now redesign, and/or extend it without additional “refactoring costs”.

Otherwise… well… not every product succeeds according to the data… Now you know why at least some of them have not…

Why estimating the project scope does not make sense?

Software estimates fail.

Even if your assumptions about users’ needs are more or less correct, the chance that it will not change with time is still low. Maybe you already have done extensive user experience research and tested the market well. Perhaps you surveyed a lot of users. Perhaps you even already have some contracts signed with some clients before the product launch (as some of our clients did in the past). Even then, most probably estimating this scope is useless… I will tell you why, based on our experience.

Usually, when working on new products with our clients, we are aiming for fast product hypothesis validation. Our promise of “going live in less than three months” is based on our experience, not theories. To validate the hypothesis and meet the market needs, we deliver the product in an incremental way, where we have some new potentially shippable (or at least presentable) product increments even 2-4 times per week. We show that to our clients, and we encourage them to show that to their clients/users and stakeholders. This is an excellent source of feedback.

What usually happens in that cycle, is that regardless of how proper the initial scope documentation was, after a couple of weeks from starting the development, our teams are working on something different than what was agreed on the product roadmap at the project start. More than that, often at that time, we are working on requirements that have never been on the roadmap and in the specification before that project started.

I think I do not need to tell you what happens after we go live with the product. The scope usually goes crazy when users start using the product (even the early stage, only an MVP version). When we are beginning collecting real data, not just users’ opinions and feedback, we can make much better product decisions – there is no guessing anymore. Also, A/B testing became helpful after the user’s volume grows. At that point, many brilliant new ideas of the product’s functionalities appear.

What should I do, to build my dream product?

Do not understand me wrong. I am not saying that not knowing how much money you need to build your product is a great approach and you always should jump into the product development regardless of your wallet size, armed only in your optimism and opportunism.

On the other hand, if you’re sure custom software is the right choice for you and are serious about building a product you already know how much you can spend on building it, and you do not need to waste time preparing detailed documentation to learn if it is enough.

Most of the software vendors will be able to answer the question formulated in the way: “Is XYZ enough to build a product/platform/app that is doing ABC and is similar to QRS?” or even a much better question formulated like that: “I want to build a product that is similar to QRS and is doing ABC, but at the start, I want to spend XYZ so what could we do for no more than XYZ together?”

Most people working for software vendors are kind and always try to be helpful. With open questions formulated like that, you are increasing the chances they will use their creativity and tremendous experience to help you. Even if your XYZ budget will be way below what is needed to deliver the actual product they will for sure advise you on what you could do now. In some cases, it might be for example a piece of advice to use some part of your budget to aim for additional investment, or validate your business hypothesis using some marketing agencies and search for investment later on with the validated hypothesis. Some of them might even connect you with their battle-tested marketers, startup accelerators, freelancers, growth hackers, even some business angels or serious investors (if your product idea sounds good for them).

Well, at least this is the way we are doing it, but I am pretty sure that we are not the only ones out there.

When just asking for estimates, which is a closed question where an answer is simply a number – the price, you limit the possible response to only the number nothing more than that. You limit your chances to get real help and find real partners, not just service providers in product development.