Skip navigation
All People > Florian_Cheval > Florian on software

Last time, we looked at a formula to evaluate the effectiveness of a piece of software to complete a particular task/solve a problem.


It was basically saying "amongst 2 pieces of software, the best one is the one that will get the more users to complete the task. In case of a tie, the fastest alternative wins."


So now you're going to come and tell me that you've developed this piece of software, that does exactly the same thing as the leading product in this problem space, except that you can do it x% faster.


According to the Good Software Evaluation, you have a superior Product and you can rightly expect to become the new norm and get ridiculously rich.


But you don't. You've shown your thing to many a customer of Leader's Product, they all tell you nice things and that it would certainly be appreciable if they could solve this faster than they currently do.


Apparently, If you're working for an Israelian or Japanese company, you're waiting by the Fax machine for the order forms to come by, but it remains awfully silent. Don't panic just yet. Tis not the fax machine that's broken: you're just not revolutionary enough.



Some would say you're not "innovating" or "disruptive" enough, but these are just buzzwords and I hate buzzwords, so let's try and explain what's happening here just using normal people everyday words.


Think of a product you have to use every day of your life. Your toothbrush. Your soap. Your boiler. Your micro-wave. Your shaver. A pen. Your car. Whatever.


You've got one firmly pictured in your head? You can just see yourself interacting with this product?


Good, now I'm going to come up with a "superior" product to the one you currently use.


I'm going to create for you a faster toothbrush. A safer micro-wave. A closer shave experience. A cheaper pen. A more economic car.


Will you buy it?


- It depends.


How much faster? What do you mean safer? I don't care about being shaved closer, my current shaver is perfectly fine. How cheaper can it get? How much will I save?


Let's take some examples.


Your dentist tells you that you have to brush for 3 minutes. If you're anything like me, after 1 minute you really have no clue where to go next, even your gums and possibly your nose feel really clean and your arm is sore.


My new toothbrush will get your teethes spotless 2% faster!


You're probably not going to run to the store to get a new one for such a negligible increment, right?


Let me come back with version 2.0, it's now 10% faster. Still not rushing to the store now, are you? This starts to seem like a modest differentiator, like all these toothpastes who claim to make your teethes whiter/shinier.


My Engineers have discovered this amazing way to intertwine the molecules of the brush, and it's now 50% faster! I sense to perceive a genuine interest? You may even be considering buying it next time you go to the store while you just bought a new one. Imagine that, instead of wasting 1-3 minutes of your life thrice, *cough* twice a day, you can divide that time by 2 for the same effect!



We've added some cleaning nano-particles to the fabric, and the latest generation is 90% faster than a conventional toothbrush. That means you can spend 18 seconds brushing and get the exact same effect as in 3 minutes with a regular toothbrush. Something you just couldn't do before now becomes a breeze! All things equal, I'm pretty sure we have a killer product here.


Toothbrushes are cheap and renewed frequently. Let's take the car next.


2% more economic: okaaaay...

10% more economic: that becomes a significant selling factor

50% more economic: You are going to have a hard time convincing both your wife and your wallet not to buy my fuel saving car and buy a conventional one instead.

90% more economic: This car is now pretty much the only game in town, it is considered socially irresponsible to buy the conventional ones, and the legislation will soon evolve towards much drastic carbon emission limits to match this "new normal".


Learnings until now: A 10% improvement in a toothbrush isn't very interesting, while a 10% improvement in a car really is.


Corollary: The more money is at stake, the smaller the improvement percentage has to be for it to become viable/interesting.


If I can improve by 1% a process that costs you 1 Billion dollars, I just saved you 10 Millions dollars, nothing to be frowned at, which explains why banks spend so much money on tuning their algorithms and try to place transactions as fast as possible in high-frequency trading, because a ridiculously small improvement yields huge amounts of money.


So to conclude, if you're not working in Finance, aim for 90% improvements over the current solutions, and if you can make it, you're pretty sure to transform the world in some way, even if it's just toothbrushes. Even if you fail, a 50% better solution is still a very compelling product...


Here's what Larry Page thinks about "10x" innovation: Google's Larry Page on Why Moon Shots Matter | WIRED


Ok, back to sofware now.


You are considering a new feature for your product. That means you are going to spend time, energy and money researching, designing, implementing, documeting,  testing and supporting it, there is no small feature.


How will it change the game for your users? Will it fall into the "negligible improvements" category? In the "mildly interesting" one? Or will it be so good that your users can't understand how they could suffer the previous solution?


Next time you're considering changing anything in your product, don't settle for the safe/modest improvement. Take 10 seconds and do me a favour. Ask yourself: Can I take this opportunity to improve my users lives by at least 2? Chances are it's not that much more expensive than the small insignificant enhancement your were considering...


Conclusion: Your new product developments will be "valuable" when they improve the existing alternative solutions by at least 50% (unless you work in Finance and you can save the 0.5 cents that get lost in the system).



How do you measure it?

Posted by Florian_Cheval Employee Nov 10, 2015

Yesterday, we tried to come up with a working definition of "good software", and we arrived at:


"Good software does what you hope it will. (And is not ugly)"


I said I would talk about how to create it, but we're doing Agile Development here, and our users have asked a pressing question.


Users first, let's answer the question: "Ok, but how do you measure it?"



So let me propose the highly scientific patent-pending empirical method to measure the quality of a piece of software: The Good Software Evaluation.


As we discovered, the quality of a piece of software needs to be evaluated against a particular goal. There is no "good" in general, there's only "good at this".


So the first thing you do is decide upon the goal. That's right, you arbitrarily choose a task in your head and you decide that this is the one you're going to measure against. To be of any use, I advise you choose a task that you think users typically try to complete when they use this piece of software. Of course choosing the right task/goal is where you need to be smart, and you can cheat if you want. I can probably state that all software are *perfect* at some irrelevant task. Yes, let's make that a first law:


1st Law of the Good Software Evaluation: "All software scores perfectly at some irrelevant task"



So choose an irrelevant task, and your Product will score 100%! That should feel good, but will probably not help your users much.


Ok, so you chose a task/goal that you believe relevant for the user(s) of the software. It can be anything really:

  • The user needs to sign up
  • The user needs to fill basic profile information
  • The user creates a project
  • The user finishes a project
  • The user is laughing
  • The user is sleeping
  • The world is a better place
  • A new cat video is posted on youtube
  • Both users get married an happily live ever after


Now, let's start some measuring!

Find a user. Yourself. Your coworkers. Your wife and kids. Your customers. Strange people on the Internet.

Put them in front of your software, and ask them to reach the goal/complete the task.

Start a timer.

If the user cannot complete the task, the software gets 0, you can stop the timer now, let's make that the second law:


2nd Law of the Good Software Evaluation: "If users fail to complete the chosen task, the software scores 0"



Once the user completes the task, stop the timer and mark down the result X. Yes, it took your test subject 103 seconds to sign up. Yes, it took him 3 months and 2 days to finish a project. It only took 2 weeks for these 2 to fly to Vegas and get married.


So, now you compute the grade for this particular test: 1 / ( X + 1 )


If it takes a very very long time, X will be very big, and the score will be close to 0.


If the task has been accomplished "instantaneously", X=0, and the score is equal to 1.


Of course you're going to tell me that I didn't specify in which unit I should count X. In seconds? In hours? In microseconds? The truth is it has no importance, you're not trying to compute an abstract value for your software here, you're actually trying to measure how good it does at accomplishing a particular task for a user *against an alternative solution*. So the unit doesn't matter, choose one and stick to it. If you really don't know, measure in seconds.


Now run the same test with other users, each time you get a grade.


The grade for your software is now the mean of all individual test grades.


3rd Law of the Good Software Evaluation: "The software grade is the mean of individual test grades, where each test grade is equal to 1/(X+1), where X is the time it takes for the user to complete the chosen task"


We're done with the math! Let's look at a few applications and how it relates to my definition.


I'm going to put you in front of my software.

I'm going to ask you to try and do something with it. Create a project, upload a video, write a blog entry, send a message to Wendy, whatever.

As soon as those words are out of my mouth and reach your brain, suddenly you form an expectation that the software can do the very thing I just asked you to do, so I'm going to start the timer.

Now you're going to try and accomplish the task.

Once you're done, I'll stop the timer. Did the software do what you thought it would do? Apparently yes because you just completed the task that I was measuring. Was it easy/intuitive/pleasant? Let's look at the chronometer. Did you really hope it would take approximately this amount of time or less? If the answer is yes, then the software is pretty good. If the answer is no, then you're disappointed, and you think it should be faster/easier.


Ergo: "Good software does what you hope it will" (and is not ugly), and you now have a highly scientific method to measure it


Now we're going to compare 2 Products against one another.

The chosen task is "I want to know what time it is"


On the left my crappy old swatch, on the right my shiny smarphone:Battle.png


Swatch, pull up my sleeve a little, lower my eyes, look at the screen, internalize the numbers, total time: About 1 second.


Smartphone: Take it off my pants, arf, why are these pants so tight, take it off my pants, why does it have to hurt so much to take a phone out of one's pocket? Pfff, finally got it, press the button, look at the screen, be distracted by the reminder, finally see what time it is: About 5 seconds.


And the winner is my good old fashioned faithful watch. In this test, it is a *better* product than my phone to know what time it is.


Tomorrow, we'll talk about an interesting implication.

I was writing a document this morning, and I had to reference the definition of "Good Software", so of course I googled it:


And I was stunned: It was nowhere to be found.


Come on Google! We're in November 2015 and you still don't know what "Good software" is??? How are we supposed to make happy users if we don't know what "good software" is???



I think it's about time we remedy that situation. Let's start with a very common simple definition that one hears frequently: "it just works".


This definition is somewhat helpful but not entirely sufficient either. A "hammer" just works, but if my user needs to drill a small hole in a wall and I give him a hammer, he will complain that my product isn't very good. So to actually measure the "quality" of a piece of software, we need to match it with the "need/problem" that the user is trying to solve. Yes, I'm recklessly discarding all the "internal" qualities of your software about how nicely organized and modular the code is and how lovely the UI art is. I don't care, and your users probably don't either. One can only judge a product by how well it solves a particular need for a particular individual in particular circumstances: You may manufacture and sell the best eletrical drill and carpenters across the world may love it. But if I take it to my shack in the woods where there is no power outlet, it's useless to me in that context.



So that takes us to our second iteration of the definition, and I'm going to authoritatively skip the circumstances part, because:

_It's my blog entry and I can

_I'm going to assume we're going to talk about software that is mostly used under "normal circumstances". You're sitting somewhere comfortably or uncomfortably with a device in front of you, or you're standing, or you're laying down, I don't care. For now, let's assume that you have a piece of software in front of you, you are in the appropriate setting to use it, and you want to know if it's any good or not.

_Finally, you will see that it doesn't change the conclusion by one bit. Even if you're running, diving, naked, in a hurry, sleeping, in the sun or under the rain, agonizing, high, or have lost the use of all limbs, the conclusion will be the same.


So circumstances are important, but we don't care for now. Okay, so second definition then:


"Good software solves a particular user need/problem well".


That's better, but still a bit too vague, the word "well" being too imprecise, let's try and refine it.


In the case of software, speed is often an important factor, right? If you click on the "add to cart" button, and it takes 10 milliseconds which you can't really perceive, you won't even think about it. If it takes 1 second and your brain actually starts to feel the wait, you will go "meah, okay, but definitely not great". If it takes 1 minute, no amount of spinning icons or progress bar indicators on earth will make you think the product is any decent. So to be "good" it has to be "fast".


What else? It needs to be reliable, but also work consistently. Reliable means it must not work extremely well 99 times and destroy your whole world the 100th time you use it. Consistently means it has to produce the same output given the same circumstances, no matter how good the alternative is. If you use a drill to punch a hole, if once in a while it refuses to drill and decides to order a free pizza for you instead of drilling a hole, no matter how much you *love* pizza, you still very much need it to start drilling *now*. Ok, so fast, reliable and consistent the software needs to be.


Next one, I'm sure you've all been bitten by. You search for a piece of software that can do X. After much googling, you finally install the software of the promised land. And then you try to make it do what you thought it could do, but really doesn't after all, no matter how many hours you spent raking the internet. So it has to be "honest" in its depiction of its capabilities, whether that's its UI, its documentation, its marketing website... I could even push it to say that it should be "humble", as you're more likely to "delight" your users by actually doing more than you promised you would... Fast, reliable, consistent, honest, more?


Ooooh, yes there is so much more, the list seems infinite.


anigif_enhanced-buzz-17437-1386861446-16.gif"This is so much more than a bag"


It needs to be polite as Alan Cooper has taught us. It needs to be accurate but "just enuf", because it also needs to know when to approximate/summarize data/information as our poor brains can't deal with copious amounts, so that takes us immediately to the next item: It needs to be "easy".


There are (many) pieces of software out there, that actually have the power to do quite useful things, but are just so darn hard to use. From your user's point of view, there is really little difference between a broken software and a software that he can't use because it's too hard, the end result is exactly the same: He can not use your product.


I'll throw in one more: It needs to be thorough/hard working. Remember the 80/20 rule? "80% of the effects come from 20% of the causes". I hear it applied to software design *all the freaking time*, but people usually don't apply it correctly. It usually goes like this: "Hey, let's just implement this small thing, that way we'll solve 80% of the problem with 20% of the effort". From a user POV, I'm sorry to inform you that a problem 80% solved is *not* a solved problem, so don't expect them to *love* your product, as you're really leaving them to clean the mess you refused to deal with. If you want your users to love your product, you have to be prepared to really "nail" it, the whole nine yards. The 80/20 rule does work for software, but not like you may think it does. Software is about doing stuff for us dumb and lazy humans, right? Let' suppose that to complete a task, I need 10 minutes. If your software doesn't entirely complete that task, but effectively does 80% of the work for me, which means I will only need to "work" for 2 minutes while your software takes care of the other 8 just by pressing on a button, then yes it's pretty darn valuable, what I've seen times and times over is people applying this reasoning to justify poor man's implementations, pretending to solve "80% of the problem". When was the last time you were satisfied when your files were 80% copied? When your shopping cart was 80% processed? When you received 80% of your alarms? I guess it's an argument in favor of small applications that do very few things but do them very well...



It must also be safe and secure. Safe meaning that you won't hurt yourself if you use it, secure means that Mr. Robot will not be able to come and hurt you because you just happened to use that piece of software. Safety and security share one huge characteristic: You will always feel like you pour enormous amounts of money in them, yet it will never be enough, and your users won't give you any love at all for any of that investment, and sometimes will even complain about it! But destroy enough of your user's data, or have their Credit Card data hacked, and your business just might not live another day (well unless you're Sony, in which case people know your products are ridden with security problems already)


It doesn't need to be beautiful. I've seen plenty of good ol' useful working software that weren't fancy looking. Au contraire, I think a stunningly beautiful UI may be distracting or feel alienate some users, but I don't have a final opinion on this. Actually it doesn't need a UI at all. There are many command-line utilities that are pretty darn "good". So "beauty" doesn't really matter, plus it's in the eye of the beholder anyhow. But, it mustn't be ugly either. Ugly is definitely jarring/offputting.



Let's look at our little list again: "Good software" needs to solve a particular problem/need/task for your user, fast, easily, reliably, consistently, politely, accurately, thoroughly, safely and securely, must be honest/accurate in its depiction of its capabilities and must not be ugly.


That's a pretty thorough definition, of course I'm sure I'm forgetting things, but if your product can achieve all of the above, even I might be able to use it!


But let's be honest, that definition is dead ugly and you won't remember it. If you can't remember it, it's no good to you in your day to day job. So can we abstract this definition to something shorter and more useful? Yes, we can, and to get to my final definition, I'll shamely steal from Joel:


Well Joel actually has that whole explanation about matching the mental model, but even that's way too complex for me, so let me distill the essence for you right there:


"Good software" does... what you hope* it will! (and is not ugly)

*Within the limits of reason of course.


I'm sure it's a leaky abstraction, but as most abstractions, if it works 99.99% of the time, it turns out to be pretty darn useful!


Note that I now realize that I could have saved you 3 pages of reading by just defining "fast" and how "fast" is relative to your perception/expectation, boom voila done. Also note that I told you that circumstances didn't matter, and you can verify that by yourself as an assignment for next week.


I'm pretty sure you can generalize this definition to define a "good product", so feel free to replace every occurence of the word "software" with "product" if you're so inclined


Now that we finally have a definition of what "good software" is, next time we'll see how to create it.