Tuesday, August 22, 2017

Learning: Book - First, Break All the Rules

Jotting down some learning /quotes from reading this excellent book on being a good manager by breaking all the conventional rules.

General tips for Managers:

- Treat people as you would like to be treated.They would say don't treat people as you would like to be treated

- Don't treat everyone as one and so to avoid charges of favoritism. Treat them differently as everyone is unique.

- Don't try to make clones.  Don't try to fix people. Make them more of what they already are.

- Don't always promote people out of their role if they are great in what they do. Help them become the 'best'  and 'world class' in what they do.

- Myth - Trust is precious - it must be earned.

- Expect the best from your people and they will give you their best. Trust them.

- Being a great manager doesn't make you a great leader and vice versa. Don't use them interchangeably.

- Managers look inward while Leaders look outward

- As a manager, always remember, you are on a stage every day. You are people are watching you perform.

- Make few promises and keep them all

- If a company is bleeding people, it is bleeding value

- You employees don't work for you. you work for your employees as a manager

-  Managers trump companies. Manager is most important in building a great team. Not pay, benefit, perks or brand name

- Manager is to reach inside each employee and release his unique talent into performance

- Conventional Wisdom: Select a person (based on his experience, determination and intelligence), set expectations (by defining the right steps), motivate the person (by helping him identify and overcome his weakness), develop the person (by helping him learn and get promoted)

- Instead, Select a person (for their talent), set the expectation (by defining the right outcome), motivate (by focusing on their strength) and develop (by help him find the right fit)

- Manager is like a casting director for a movie. He has to select the right employees for the right job.

- As a manager, spend max time with your star performers.

- Never say that i know how you feel because you will never know how exactly someone feels. Each person has its own filter and they see world through those lenses which may be very different from how you see it.

- Conventional wisdom: Investigate your failure and try to invert it

- Instead, build upon your success and study your best performances and amplify that

- Excellence and failure are often surprisingly similar. Average is the anomaly. Example, The difference between the best and the worst is that the best nurses use their emotions to take control
and smooth the patient's world as far as is possible, whereas the worst are
overwhelmed by their emotions. Average nurses? Average nurses protect
themselves by keeping their distance. They are emotionally disengaged

- Manager are akin to sports coaches who turn the talents into great performance by providing the right direction and guidance

- Define the right outcomes and then let each person find his own route toward those outcomes.forcing everyone to follow the same path toward those out comes. Standardizing the ends prevents her from having to standardize the means.

- As a manager, resist the temptation to control their people.Many managers can frequently be seduced by the idea that there is "one best way" and that it can be taught.

- Don't let the creed overshadow the message. Required steps are only necessary if they don't obscure the desired outcome

- Don't try to make your employees more "well rounded". Let them be best in what they are good at.

- Be anything we want to be if we just work hard is a wrong message and this mean everyone has same potential.And if we all have the same potential, then we lose our individuality.

- No matter how well intended, relationships preoccupied with weakness never end well.

- Wrong message: You can become more persuasive, more strategic, or more empathic if you just work at it,"

- Employees can be grouped in 3 buckets - Keepers, Losers and Movers (miscast)

- Spend the Most Time with Your Best People

- Conventional wisdom: No news is good news.
Instead: No news is BAD news. Everyone expects a reaction and acknowledgement so provide instant and regular feedback

- You cannot learn very much about excellence from studying
failure. Unfortunately you don't come any closer to identifying those right ways by eliminating the
wrong ways. Excellence is not the opposite of failure

- Don't use average to estimate the limits of excellence

If a manager is preoccupied by the burden of trans forming strugglers into survivors by helping them squeak above "average," he will have little time left for the truly difficult work of guiding
the good toward the great

- Average thinking actively limits performance with all degenerative diseases, procrastination in the face of poor performance is a fool's remedy.

- Bad performance: "No," it's not a skills/knowledge issue, and "No," it's not a trigger
issue—then by default the nonperformance is probably a talent issue.

- As soon as they realize that a weakness is causing the poor performance, they switch their approach. They know that there are only three possible routes to helping the person succeed.

Employee development

- Devise a support system (Reading glass example). Find a complementary partner. Or find an alternative role

- The first fallacy is that each rung on the ladder represents a slightly more complex version of the previous rung. Consequently, if a person excelled on one rung on the ladder, it is a sure sign that  with just a little more training, he will be able to repeat his success on the rung above. The best managers reject this. They know that one rung doesn't necessarily lead to another..

- Why not carve out alter native career paths by conveying meaningful prestige on every role per
formed at excellence? Why not create heroes in every role?

- Conventional wisdom: "Since Jan has proven herself a solid manager, I am confident that we can teach her the strategic thinking and the vision needed to be a great leader."

- Great managers excel at "holding up the mirror." They excel at giving performance feedback
great managers made a point of giving their feedback in private, one on one. The purpose of feedback is to help each individual to understand and build upon his natural strengths. You cannot dothis in a
group setting.

- Great managers create a safety net. In conventional wisdom's world, taking bold career steps in
order to discover a latent talent or to refine an existing one is almost as fool hardy as volunteering to learn the trapeze without a safety net.

- When an employee is obviously miscast, great managers hold up the mirror. They encourage the employee to use this misstep to learn a little more about his unique combination of talents and non talents. They use language like "This isn't a fit for you, let's talk about why" or "You need
to find a role that plays more to your natural strengths. What do you think that role might be?" They use this language not because it is polite, not because it softens the bad news, but because it is true.

- This is the "love" element of tough love. The most effective managers do genuinely care about each of their people. But they imbue "care" with a distinct meaning. In their minds, to "care" means to set the person up for success

- Great managers disagree. When Gallup asked, "Would you rather get employees what they want, or would you rather get them what is right for them?" the great managers consistently replied, "Get
them what is right for them."

- Everyone breathes different psychological oxygen. What is fulfilling for one person is asphyxiating for another

- Performance appraisals - focus on the future.

- They can help the employee find his path of least resistance toward his goals

- In the world according to great managers, the employee is the star. The manager is the agent. And, as in the world of performing arts, the agent expects a great deal from his stars.

- Instead its power lies in its idiosyncrasy, in the fact that each humans nature is different

- "Don't try to put in what was left out. Try to draw out what was left in."Some managers may try to resist these forces of change, but they will fail

- The Art of Tough Love  "How do great managers terminate someone and still keep the relationship intact?

- The "tough" part is easy to explain. Because great managers use excellence as their frame of reference when assessing performance, Tough love simply implies that they do not compromise on this standard. So in answer to the question "What level of performance is unacceptable?"
these managers reply, "Any level that hovers around average with no trend upward." In answer to the question "How long at that level is too long?" Great managers reply, "Not very long." But I know my people, sometimes better than they know themselves

- Tough Love: When picked the wrong person on the job. Fire him. Do it fast, faster the better. You are not doing a favor to anyone by waiting. I believe that, deep down, the poor performer knows he is struggling before you do. Maybe he can't find the words, or maybe his pride won't
let him say it, but he knows. On some level he wants your help. And so, subconsciously, he puts himself in situations where his weaknesses are exposed. He is daring you, pushing you to fire him. I call this manager assisted career suicide. If you suspect that this is happening, the best
thing you can do is help put him out of his misery.

- In the minds of great managers, consistent poor performance is not primarily a matter of
weakness, stupidity, disobedience, or disrespect. It is a matter of miscasting.

- Manager try to highlight and perfect each person's unique style. They draw his attention to it. They help him understand why it works for him and how to perfect it


- Hire for talent and not just skills and knowledge.

- Hire people whose eye lit up when they talk about their job.

- The Art of Interviewing for Talent "Which are the right questions to ask?"

 - Make sure the Talent interview stand alone

- Ask a few open-ended questions and then try to keep quiet. Listen for specifics


- Talent is recurring pattern of thought, feeling or behavior  that can be productively applied.

- Every role, performed at excellence, requires talent, because it requires certain recurring pattern of thoughts, feeling or behavior.  Great nurses have talent so do janitors. Talent for account is "when the books balance". Love of "precision" is a talent not skill or knowledge or experience

- Talent is not rare.  Manager help identify hidden talents for the right job and further develop them.

- Talent can't be taught. Skill and Knowledge can be.

- You can't rewire someone's brain. People don't change much. Trying to change them is futile. Don't try to fix people.

- While person can't learn a new talent which is required for her job and may become her weakness but she can develop self awareness and self-regulation so she can cope up with that limitation

- Conventional wisdom: Success is 10 % inspiration, 90 % perspiration

- Even if you hire for same level of experience, brain power and will power, you will end with a range of performance because it's the talent that is secret sauce

 - Skills and knowledge are transferable from one person to another. They have limited application when faced with totally new situation, never learnt/faced before.

- Talent are transferable from one situation to another & can help one solve the problem never seen before.

- Each person has a mental filter through which they see the world. Face the same stimuli, people behave differently. It creates your motivations, forges your attitude

- It's wrong to tell that everyone has the same potential if they are willing to open and learn.

- There are striving talent (explain why of a person, what motivates him), thinking talent(explains how of a person, how he makes decisions), and relating talent (explain who of a person. whom he trust, whom he builds relationship with)

- If I find myself telling the same person to look on the bright side' time and time and
time again, I should take a hint. He's not a bright-sider. He's a dark-sider. I should stop wasting my breath and try to find a role where skepticism is key to success."

- Myth - Some roles are easy and don't require talent. Janitor or a Secretary is desperate to get out of their current job and they must be doing this because they are stuck &; it must be demoralizing.  We don't realize that they might love their job and they have talent to be great in what they do and hence moving them out of their job is not helping them.


- Customers really wants - accuracy, availability, partnership and advice

- Accuracy and availability are easy for your competition to replicate and they are important but only prevent customer dissatisfaction. Partnership and advice creates the connection and trust that's important to have long term relationship

Saturday, August 5, 2017

Manifesto for Quality Assurance & Testing

Inspired by legendary Agile manifesto and my own experiences:

1) Having the talent to see through & anticipate failures over finding them later

2) Using machines/tools/humans as slaves to get things done over becoming their slaves

3) Common sense over conventional wisdom

4) Questioning every assumption & requirement with respect over sheep-walking

5) Doubting (with respect) over trusting or hoping for the best.

6) Being reasonable, firm  & constructive over being the opposite

Friday, June 23, 2017

Fighting with my tenant....

Fighting with my tenant to give him his full security deposit back but he insists on getting paid less
Sounds familiar? May be not. At least, doesn't happen often. He is staying in our apartment for last 2 years. No complains. Rent on time always without any reminders.

Well, what more can you ask ?

He has a daughter around the same age as ours (around 4 yrs). Kids know no boundary so she does like to color the walls :) We knew this when we were giving the flat to a family with little kids and mentally prepared that we will anyways have to get the painting done again. While we treat anyone else's property with utmost respect and our daughter is brought up with the same thinking. We had standard 2 month security deposit if there are any major damages.

Well, my tenant, Syed is vacating our apartment as he is moving to a different city. As expected, the walls were bit colorful by now and bathroom had some usual wear and tear

I am a reasonable person. I need to invest my own property as this was our first home.
I told him, i am going to return his security deposit in full and thanked him for staying with us.
He comes back and says, no and i should deduct the damages from the security deposit to which i am surprised but i politely decline.

He insists that i should keep half of the security deposit. We negotiate for sometime that i don't want to deduct more than minimum required for fixing. I reluctantly agree because there is indeed some damage and glad to see that he strongly feels that same way.

Finally, we agree. What's the lesson here ? Nothing that we don't know but we don't practice much.
Be kind and others will be kind to you when you least expect.
In life, when we think of negotiation, we always think that you negotiate for getting "more" out of the situation but i learnt today that you can negotiate to get "less" and still get "more"
Reminder to myself, be nice to people more often without expecting anything in return.

Dealing with frustation and coming out as a winner !!

I ride my bicycle to work everyday and I absolutely love that part of my life as not only i reach my office, which i have to anyways, but also get some good exercise. It was going great for last one and half year till a month back when I had a flat tire when i was on my morning weekend ride, 30 kms from my house. I was surprised as it doesn't happen often in Singapore as roads are so good. I was cool about it as it used to happen a lot more often during my school days while growing up in India. I thought i will dismount and push my bike to find a bicycle repair shop to get it fixed. Little did i know that Murphy's law is going to come into play and I wouldn't find a shop after pushing the bike for around 3-4 kms even though that stretch indeed has quite a few bike shops which i pass by every weekend without even barely noticing them.  Finally, i got a cab driver who was kind enough to try to put my full size bike into the boot space. I got home as it was early in the morning and lot of bicycle repair shops were still closed and i thought of dealing with it a later point.

Now the ordeal begins. In the evening, I looked for nearest bicycle shop and it was 2 km away. I pushed my bike to find out that it was closed. A bit disappointed, this time i tried to be a bit more smarter by finding the next closest shop and called them to check if they are open. Then i pushed my bike for another 2 km to get there and felt victorious that they would now fix my problem and this will be all behind me.

This is Singapore and not India. In India, we don't dispose thing off just like that. If you get a flat tire then you get the puncture done. You do in a for loop every time it happens till the bicycle mechanic gives you an expression that it's enough and it's time to for a new tube. I guess here time is more scarce and you don't fix things instead dispose and just replace it.  Anyways, the girl at the shop looked at my flat tire from behind the counter and said I need to get a new tube. I didn't question her judgement as I  didn't know how bicycle doctors work  and I am anxious to have this fixed so i agreed though not convinced. In less than 3 mins, i was on my way home riding my bike again and feeling like a kid who got his first bike.

I got home, happy that i am all set for a new week to ride my bike to work again. Not bad at all. It happens sometimes but I did pretty well. Had my dinner and after an hour, i went to check my bike again to say good night before going to bed. I went from happy to sad in less than 2 seconds to see that the tire is flat again and the shop is closed by now. How on earth ? Hey, that's okay. Not your day may be. You tried. I will deal with it tomorrow evening and take the bus to work.

I get home next day evening from office with new determination to change things today. Pushed my bike to the same shop which was closed the day before (as that's the nearest one). I tried to tell them my unfortunate series of events and expecting a bit of sympathy but the staff wasn't interested and concluded in less than a second that solution is to change the tube and tire as well. Take it or leave it. Well, I tried persuading them that the tube is brand new and couldn't it be that just the valve that's faulty. You may think that I managed to convince them to have a look at the problem. Well, no. I was given an option, do what they have already concluded or feel free to leave. I thought about it. The tire had worn out as i was using this bike pretty regularly and yes, it could be well the tire  that could be the culprit and i should invest in something that i love. Now what can go wrong if I am changing the tire and tube both. I went for it.  Again, I am on bike and feeling excited again. Deja vu. I am very positive this time. Not efficient but the problem is fixed. Sometime, you need to take the deal even if its not what you wanted.

1 week later, I am on the way to office and my tire goes flat again. Seriously? This is not happening. Are you kidding me ? Awesome !!! What a start of the day. Am i pissed off now ? Yes, a little. I  park my bike near an apartment where it broke down, walked to the nearest bus station and reached office on time to deal with this in the evening. In the evening, reached the spot again, find the nearest shop and this time, found another shop that is 2 kms away. Why it has to be always 2 kms which i feel encouraged to push it as it not far enough to take a cab? Hey, its a good exercise. Walking is as good as cycling, i convince myself. Also, i thought that the last 2 shops were not good since they must have given me bad quality otherwise no way it would have got flat again on the same road i was treading all this while. I reach this shop sweating as weather is not very kind in Singapore for outdoor activities. Again, this mechanic looks to be the cousin of the other mechanic and pays no attention to my story and tells me that tube needs to be changed. There is no other shop in the vicinity. Even if there is, how would  that be different if this is the standard operating procedure here ? Though i was quite frustrated by now, i reluctantly agreed as I have limited options.

I will save you the painful details and let you know that today while returning from work, it happened again. I was heartbroken but hey, i don't give up. This time i searched and found another shop close to my office (again 2 km, i am not making this up). I checked that his reviews were pretty good as compared to the previous shops. I decided that today I am not going to play the victim and going to be cheerful about it. Got to the shop. Surprisingly, i was pretty relaxed today. Hey, i have seen it all. What worse can happen now? Things can only get better from here. For the first time, the mechanic listened to my bicycle medical history and like a seasoned surgeon, thought about it for more than 2 seconds & then concluded that the rim of the tire was probably damaging the new tubes everytime as that's the only thing that didn't change all this while. He placed an additional rubber layer on top of the rim before putting on a new tube. This sounds quite logical. One by one we have eliminated possible point of failure. Again I am on way back home feeling excited again. Couple of mins back i did a check on bike again and its intact so far.

Will it not happen again ? Probably, it will. Am i making progress? Definitely, Yes.  Am i frustrated, Hell, NO. That won't change anything. I will remain positive and keep making progress. I won't give up.  Was i was very smart about this ?  No, but i did learn something new every time and in the future, I wouldn't repeat that again. Do i have a plan if it happens again? Yes, this time, i will take my bike to the manufacturer's showroom which is across the island if the local shops are all selling inferior quality material.

By the way, that's my bike. Yes, i am a little obsessed with it but then a man should have a passion.

I know that this may not mean a lot to you but if you still reading this then the point i want to drive back home is that, unfortunate things happen and sometime they can happen quite often where you would find it hard to believe that it's even happening. The important thing is how you react to it. Sundar Pichai has a beautiful cockroach theory on the same line. Life can throw things at you which can make you quite unhappy or annoyed but how you decide to respond to it is your choice.

If you are wondering, what this post had to do with Testing as i did maintain this as a testing specific blog till date. Then I realized that there are some learning in life which are worth sharing as a reminder to yourself as well as to others who you may read and get inspired. This attitude helps me deal with problems at work everything where things do go wrong which are beyond your control always and we need to just keep fighting till we fix the problem.

Friday, February 14, 2014

Future of Testing


Someone recently asked me, when we have daily builds going into production, and we almost have daily/weekly OS, browser and platform updates that can break our applications and we need to support a multitude of devices (PCs, Tablet, Mobiles) with different form factors, “Can you do all your testing that is required every day?

And the next question, “If you need more time than that, then probably you are too slow to keep up with the pace that our ecosystem is evolving at and why don’t we just enable our developers to test it on fly? Why not automate everything? Can testers really add value anymore?”. 

AI is the new UI. In other words, the future devices might not have a traditional UI to run automation using QTP/Selenium. You future devices will not be just tablets and smartphones but also smart watches, glasses, cars and more. They will take multiple inputs like touch, voice as input.

Now we can look at this chasm and think about returning back or falling down or see the opportunities that lie ahead if we take a big leap forward. My response to these challenges is that “all” the testing can’t be commoditized as testing is an art and science both. You can’t automate everything and replace the intelligence of human testers. No doubt, we (testers) need to be much more agile and change the way we used to think about testing. However my question to him, “Would you go to office in a car that was just unit tested by a developer?”

Key Questions:

1.      De-mystifying the role of traditional testers in years to come and what’s expected from them

2.      Discussion around adapting to embrace daily releases and supporting the consumer devices that run apps. Support trends like Consumerization of IT and BYOD.

3.      New types of testing to test software that run on embedded devices, take voice inputs from humans.  Could we be working in hardware labs where hardware and software be tested together with future devices becoming more life and mission critical?

4.      Why developers would still need to focus on development and they can’t replace testers but can of course help.

Challenges with today’s testing

The most repeated question to development teams in reputed organizations today is, “Why do I, the client, need to spend additional money on testing efforts when I am already paying a premium for high-quality developers?”

Such questions are not without logic. Let us see where the client is coming from.

To the client, an organization taking up the project must have already developed similar projects in the past, have a highly-skilled pool of resources to accomplish the same and able supervisory resources to ensure there are no execution delays. However, as we understand, each project is like a new innings – you cannot simply replicate your past laurels but can draw from your experience.

The purpose of this paper is to precisely help the testing community answer such questions. One can even be faced with such questions from CTOs/CIOs with regards to justifying testing efforts. The entire existence of testing services then faces a question mark – do we have answers that ensure the need for testing in the future?

There are two parts to this answer and a larger implication for subsequent projects.

a)     Functional testing vs. Non-functional testing

The first part is that testing in parlance has ceased to remain merely confined to functional testing. With the increasing tribe of different devices, platforms and applications; testing has become an envelope of services covering functional and several non-functional types of testing. Therefore, for any forthcoming project, it will eventually become commonplace for the client to pay for a bouquet of testing services and not merely functional testing – which is commonly considered a plain-vanilla form of testing meant to unearth development pitfalls. We are at the verge of a paradigm shift where usability testing and user experience has assumed great significance to the extent that usability is considered a must-have and the associated functional testing is a given. This is a departure from the earlier industry trend of considering non-functional testing as a good-to-have but the focus remaining on functional testing as a must-have. Clearly, efforts are more pronounced in value added testing these days as opposed to traditional testing in the past. Therefore, testing services are no longer an added cost but an investment to achieve wow factor with value-added testing services.

b)     Speed of delivery

The second part to the answer goes further in-depth of what functional testing will comprise of. With the faster development sprints and release cycles – testing resources need to be one step ahead of the development folks to firstly detect a flaw and moreover, they need to do that with incessant regularity & speed. We would have to agree that higher the quality of development effort, higher should the testing abilities be for ensuring uncompromising quality. Additionally, customizations and related activities will also assume significance in a lot of projects – wherein testing of functionalities would be on the back-burner.

These answers, however, are not reasons alone to indicate how testing services will be offered but in fact, they point towards a larger implication, which is – testing services will be targeted for reducing release times as we move on and this will mean the need to firstly justify the entire bouquet of testing services that need to be offered to any codebase for providing quality assurance and at the same time, ensure that this is done in the least possible time. In fact, increasingly lesser time.

To add to this race against time, there lie greater pits such as ensuring controlled updates like updating an application with time are not broken. A bigger thing on the update space is uncontrolled updates though – which are passed on by OS manufacturers and come thick and fast these days across devices. The challenge is to therefore ensure that project-sponsors are aware of the need for elaborate testing and testing teams are adept enough to deliver these services in shorter spans of time – not compromising on release quality.

Impact of Future Trend

 Keeping these key elements in mind, we can attempt to identify the gap existing today. The future is already here – so are we ready for daily test cycles for apps, OS updates, daily controlled and uncontrolled updates to production impacting many apps and services or are we ready for services projects who are small in tenure and have early go live in market? The reason is we need to be ready to be fast enough to cater to the pace of the market appetite. We need to be ready to match the pace in which we are doing development now.

It’s not merely a classical time and work mathematics problem. It’s just not possible to shrink timelines on testing effort without touch resources or scope. At least that’s what the age-old scope-budget-time triangle told us in software engineering classes. Instead, the focus is now to strategize. Strategize on testing methodologies & practices that will enable us to adapt to the new era of daily releases, platform ubiquity and backward compatibility. In fact, strategizing on testing services has never been so important – we are now in a phase where the poster boy of testing – “automation testing”, faces a gripping test. Not only it is going to be very expensive to have automation in projects requiring daily releases and updates, it may also not be feasible.

a)     Need to modify the testing approach to accommodate the need of faster releases to production

Let us examine what the gap is today and what can be done to bridge this gap. To articulate, we see widespread prevalence of traditional functional testing services – in several cases with little or no automation inspite of emergence of such offerings as testing as a service and pure-play testing firms offering a bouquet of function & non-functional testing services. The key element here is the adherence to traditional ways of software testing. Despite all efforts by organizations to make their resources up to speed on readiness for adopting cutting-edge trends & services, there is clearly a mismatch in pace at which development & testing is carried out and the expected time to go live in market. We need to change our test strategy to accommodate the need to make the product available in production but maintaining the quality too.

It is essential to understand how this change can then be brought about. The truth is, there is no single way of planning out and working towards adapting our testing methodologies. Each organization is used to a different way of approaching a project, decomposing it into meaningful and testable bits and then running quality tests on the whole.

Minimum Viable Quality (MVQ)

There are many products/services like Bing.com that are shipped rapidly to production. To be in the same phase of such rapid ships to production, we need to twist our testing approaches now. We can’t continue with our traditional testing approaches which used to run for weeks for a build certified to go to production. An approach to certify such rapid builds, we should consider now concept of Minimum Viable Quality (MVQ). 

MVQ promises to help online service teams ship more frequently with less testing, lower (initial) quality, and more efficiently while actually exposing bugs to fewer real users and at a lower overall risk than traditional methods involving stabilization phases and orchestrated releases to production. The reality is that at Microsoft we have always been comfortable shipping unbaked software.  That is why we have dogfood and we have alphas and betas and developer previews.  All of these labels tell the user that they will find bugs but that we feel the quality of the version of the software is just good enough (the minimum) to make it worth their time to use it.  The only time we really ever hit ship quality is the final golden build and even then we've always released products with some known bugs.  After all, what is a Zero Day Patch if not a mulligan on signoff?

The problem you get into with modern services development is that you are never actually done shipping.  There is just the next feature, the next scenario, the next set of performance improvements, and the never ending list of bugs to fix.  A lot of Microsoft testers bring the ship quality mindset to services release instead of looking at each release as just another incremental step where some features are now ready for mass consumption and others may need to remain hidden and exposed only to those internal "dogfood" users or the external early "beta" adopters.  Perhaps the new code is pushed to a subset of users and if it is too buggy a quick fail back to last known good takes the code out of use with minimum negative user impact.

What is the advantage of taking an MVQ approach to services delivery?  The bottom line is that you start to get data about how the code is functioning in production with real users more quickly.  The key aspect is to balance the minimum such that the feature set and the quality of those features within the product is not so low that it is not used by the target user segment.  If it is too low, then you won't discover and learn the harder to find bugs because the code won't be exercised. 

The trick is to realize that it is the science behind this process that needs to change – not the process itself. For instance, features of a mobile OS that need a long time to be released into the market may end up being obsolete or get zero adoption from the app development community. Instead, the feature-set may be broken up into smaller sets of sub-features with periodic go-live to the market and at the same time get the app development community a continuous feed of feature-sets which they can leverage to build additional apps – which eventually also ensures further development when the entire feature set is released.

Reduction in test effort for activities like test environment preparation or test case writing.

A key element of course is how test teams will shape up for such scenarios. Will test team still spend 20% of effort in preparing the Test Cases and test environment? In order to test the builds daily, can we really afford to spend time in upgrading the Test Cases or debugging issues in test environment? Clearly, we would be moving from an era of end-to-end functional testing to an era of greater ownership of application modules, a more granular approach to scheduling of testing – for instance, passing results and unlocking test-scenarios from one testing resource to another in an assembly-line like setup. The eventual approach will depend on how the organization plans to setup testing teams in the near future, evolve its test organization and adopt various testing practices such as crowd-sourcing, testing-as-a-service etc.

Alpha/beta testing or Crowd Sourcing testing

A cursory look at these evolving practices highlight such trends as Testing-in-Production, which essentially translates to alpha/beta testing by releasing an MVQ application into production and awaiting for users to unearth issues with the same. This is widely practiced by Google for instance – whose applications have been known to remain in beta for years before finally coming out of it. A similar trend is, as mentioned earlier, crowd-sourcing. The only difference here being that a group of people who usually excel at testing are the ones the application is exposed to and not to the entire intended application audience. So why do we at all need testers then if we can throw it open to the audience in production or to a competent set of crowd-sourcing folks? The answer lies in controlled adoption of these trends – for instance, an application in beta ends up receiving frequent reports of issues with regards to features within a particular module of the application. This should prompt the project team to test intensively within the module and those affected by flows from this module.

So essentially the horizon of testing has gradually expanded from simply closing testing efforts with UAT to the extent of supporting even in production – and with a variety of variables on top it ranging from ensuring functioning even with controlled/uncontrolled updates, proper support for globalization, adapt to loss of internet connectivity scenarios by ensuring offline availability and even seamless experiences for users accessing the same application from one device to the other.

a)     Complexity of systems has increased. Does it impact our testing?

The testing strategy is not going to be revised just for today’s trend of rapid app development and pushing in to production without spending ‘n’ no. of testing cycles but also for trends which are going to impact largely the way we are doing the testing today. The complexity of applications has increased exponentially. We are not in an era where we developed an application for Windows XP Operating system and which will run only on Internet Explorer 8. Your testing is no longer limited to 1 OS or 2 browsers. The no. of variables have increased which are impacting the functionality of the application.

Let’s take an example of a Banking App designed for Windows 8 Operating system. Few years back, if we had to test a banking application, the use-cases would be very limited. One would verify the core functionality on any one OS plus any one browser. It was obvious to define that this application would work only with 1 particular OS along with one particular browser. The Usability testing was not a key part of the testing strategy.

Figure: 1 Showing how testing permutations and combination of various devices has increased the scope of testing


But, today, the realistic use-cases for such an app other than the pure functionality of the app would be -

-         Can the app run on Windows Phone 8?

-         Can the app handle if Operating System updates are pushed?

-         Can the app be used on multiple touch devices like Surface, iPad, tablets?

-         Can the app work on browsers like Safari, Chrome, Firefox, several version of Internet Explorer?

-         Can the app handle if app updates are pushed along with OS updates?

-         Can the app handle all types of input methods like mouse clicks, touch, keyboard, gestures, voice inputs etc.? Several permutations and combinations can come up out of this.

-         Can the app handle if any platform update is pushed on which the app is designed.


And the list does go on.

Application under test will run on “family of devices”

We also have such apps used by people that can be used when you are working from home, when you are in office network or when you are in public network. For example, on Lync, a user will start taking call using his laptop in the office environment, then he moves towards his car and continues the call over his phone using Lync app and when he reaches home, ultimately switches to his Surface to finish the call. Can our app support this type of usability?

The concentration of testing has moved from functionality to the usability. At any point, the app should not break irrespective of ‘n’ variables impacting it.


Testing in Production.
With so many variables to test with, how will one define the test coverage at the end of the cycle? You can define in your testing scope that we will test the app only for particular use-cases but from usability perspective, in today’s era, you can’t direct user to use the app only with internet explorer because we have done testing with IE only. To handle such situations, the answer is not to increase the test team or increase the test cycles to cover all combinations of devices. But, with use of crowd sourcing and testing in Production (TiP), we can take care of this. If we do our testing more with production data rather than dummy data, we are doing much more real-world testing and can uncover more production environment issues.


Will there be such a thing as Test environment?

Traditionally, we used to invest lot of time in preparing the test environment to make it a replica of production environment. Test leads used to keep one test resource dedicated for preparing the test environment and maintaining the environment. But, now we have our environments on cloud. It has reduced test environment setup time drastically. With this, test teams can dedicate more resources plus time for testing cycles. Also, having environment on cloud makes it more production ready in terms of configuration and database setup. In Microsoft Azure, when testing cycle is completed, same test environment is converted into production and production becomes test environment.


The Case for Human Intelligence

So far, we have talked about how apps are impacted with the presence of so many devices like touch-enabled devices, mobiles etc. But, future devices are not limited to touch devices. We are going to have devices like Smart Glasses, Smart Watches or may be, Artificial intelligence driven Cars. Testing an application on such devices using emulator is not going to be enough. Emulator can’t bring the human piece of testing to such devices. Let’s take an example of Artificial intelligence driven Cars, the car should apply automatic brakes if the sensor senses an obstacle at 30 Feet and should reduce the speed to 20kmph if an obstacle is sensed at 60 Feet. If such scenarios are just unit tested by the developer using emulator, will you feel safe in such car? Will you feel safe if you know that only the code is tested but not the car behavior when it is applied with several types of obstacles to verify the functionality? With Emulator, you can’t validate this and hence, human element of testing can’t be removed, even in future and even with various smart ways of development.


Importance of User Experience Testing

Definitely, testing strategy has to change and become smarter. We need to add a lot of usability verification flavor in our testing approach. Verifying the user experience is not just about checking tool-tips on mouse hover but various factors like app should be minimalistic, individuals should be able to use it with minimum clicks. It should be super responsive and self-explanatory to users. Future generation is going to be different from today and their expectation would be different from an application. We should consider that factor too when we are talking about user experience of an app. Concentration span is very low for future generation and if they do not find the app worthy enough in first 30secs of usage, probably they will never try the app again. And, that would not be just product failure but a failure of joint effort by test and devs. Hence, it is valid to say that verifying an application for user experience is equally important and it should be part of test strategy.


Test Effort Distribution:

In essence, it’s time to stop doing things right and instead do the right things!

Figure: 3 showing how testing strategy is going to change in future as compare to today

Figure: 4 showing how testing effort is going to change in future as compare to today

Taking overall stock of the situation, a few things stand out. Bluntly put, functional testing may witness a fall in test effort but this would eventually be transferred to other types of testing. From the days of writing test cases and executing them, we would be pretty much doing exploratory testing where more of the focus remains on intelligent testing and needs a lot of human interaction.

So while it would be a future where testing services are secured but the service catalog would be a completely re-wash. It is time to move to smart testing, controlling costs but delivering on higher expectations and ensuring the best user experience – not just testing plain-vanilla functionality, and therefore no just traditional testing tools.

Testing-in-Production and Crowd-sourcing are already here as promising ways of delivering the next generation of testing services while optimizing testing efforts with smart resourcing and environments using the power of cloud would pave the way for strategizing the way forward.

Author: Gunjan Jain and Raj Kamal