Thursday, September 23, 2010

Testing the Proximity Sensor of iPhone 4

The proximity sensor problem with iPhone 4 is a topic of much debate on discussion boards, blogs and news sites. The proximity sensor is used by the phone to determine if the user is holding the phone to her ear during a call. The phone uses input from the proximity sensor to decide whether to activate the screen and allow touch input. Many owners of the phone have reported the screen re-enabling while holding the phone to their ear during a call, while others have reported no problems. I am one of the unfortunate owners of the phone that has inadvertently placed a caller on hold or interrupted other callers with touch tones emanating from my end of the call. As of today I am on my second iPhone 4 and disappointed to report my experience has not improved. There are plenty of emotional calls for Apple to quickly address this problem. I want to take a different approach. In this essay, I will provide a discussion about testing approaches and what that means for complex systems. I use the proximity sensor as a real-world example to demonstrate the problem many have experienced and the difficulty involved in testing for it.

Inside iPhone is a complex hardware system arranged in a hierarchy of command and control: a microprocessor, memory, storage, transceivers for wi-fi, cellular, and bluetooth networks. It has touch, light, sound and proximity sensor input. It has external interfaces for the dock, a headset, the SIM card. It has a single display integrated with the touch sensor input. The software distributed through these components is a system of collaborating state machines, each one working continuously to keep the outside world pleased with the experience of interfacing with the phone. It is not just a single human the iPhone must keep satisfied. The cellular networks, wi-fi access points, bluetooth devices, iTunes and other external systems are part of this interactive picture as well. This is oversimplified, but you can begin to appreciate the enormous burden of testing such a small, complex device used by millions of people.

How does a team even start to tackle such a problem? Meyer (2008) presents seven principles in the planning, creation, execution, analysis and assessment of a testing regimen. Meyer writes, above and beyond any other reason for the testing process “is to uncover faults by triggering failures.” The more failures are triggered and fixed before delivery of a product to the end user, the less expensive it will be than to fix them later. Humans are a required yet flawed variable in the planning and execution of test suites for complex systems like iPhone. Identifying all possible triggers for failure can be nearly impossible. Savor (2008) argues that, “The number of invariants to consider [in test design] is typically beyond the comprehension of a human for a practical system.” How do we test the multitude of scenarios and their variations in complex systems without fully comprehending usage patterns and subtle timing requirements for failure in advance?

Meyer (2008) argues that testing time can be more important a criteria than absolute number of tests. When combining time with random testing, also called test escapes, there is a possibility of uncovering more faults than just using a huge, fixed suite of tests continuously repeated without deviation. Test escapes as defined by Chernak (2001) are defects that the fixed testing suite was not able to find, but instead found later by chance, an unassociated test, or by an end-user after the project was delivered to production (e.g. introduction of randomness). Now that we have some background information and terminology, let’s design a test that could make iPhone’s proximity sensor fail to behave correctly.

Consider an obvious test case for the proximity sensor:

  1. Initiate or accept a call.
  2. Hold the phone against ear. Expect the screen to turn off and disable touch input.
  3. Hold the phone away from ear. Expect the screen to turn on and enable touch input.
  4. End call.

This test case can be verified in a few seconds. Do you see a problem with it? It is a valid test, but not a terribly realistic one. The problem with this test case is that it does not reflect what really happens during a call. We do not sit frozen with all of our joints locked into place, refusing to move until the call has completed.

To improve the test case, we add some physical action during the call:

  1. Initiate or accept a call.
  2. Hold the phone against ear. Expect the screen to turn off and disable touch input.
  3. Keep the phone still for 30 seconds.
  4. Change rotation, angle and distance of phone to ear while never exceeding 0.25 inches from the side of the caller’s head. Expect the screen to remain off and touch input remain disabled.
  5. Return to step 3 if call length is less than ten minutes.
  6. Hold the phone away from ear. Expect the screen to turn on and enable touch input.
  7. End call.

Now the test case is reflecting more reality. There are still some problems with it. When I am on a call, I often transfer the phone between ears. Holding a phone to the same ear for a long time gets uncomfortable. During lulls in the conversation, I pull the phone away from my ear to check the battery and signal levels, and then I bring it back to my ear. These two actions need to be added to the test case. Additionally, all of our timing in the test case is fixed. Because of the complex nature of the phone, small variations in timing anywhere can have an impact in successful completion of our test case. Introducing some variability to the test case may raise the chances of finding a failure. In other words, we will purposely create test escapes through random combinations of action and timing.

  1. Initiate or accept a call.
  2. Hold the phone against ear. Expect the screen to turn off and disable touch input.
  3. Keep the phone still for [A] seconds.
  4. Randomly choose step 5, 6 or 7:
  5. Change rotation, angle and distance of phone to ear while never exceeding 0.25 inches from the side of the caller’s head. Expect the screen to remain off and touch input remain disabled.
  6. Pull phone away from ear for [B] seconds and return phone to ear. Expect the screen to turn on and then off at the conclusion of the action.
  7. Move phone to opposite ear. Do no exceed [C] seconds during the transfer. Expect the screen to turn on during the transfer and then off at the conclusion of the transfer.
  8. Return to step 3 if call length is less than [D] minutes.
  9. Hold the phone away from ear. Expect the screen to turn on and enable touch input.
  10. End call.

There are four variables to this test case. It is possible that certain combinations of [A], [B], [C] and [D] will cause the screen to re-enable during a call and cause the test case to fail. Have fun with this one. There are in fact combinations that induce proximity failure on iPhone 4 regardless of the version of iOS, including 4.1.

Finally, an important part of test design is the inclusion of negative test cases. Chernak (2001) writes, “A test case is negative if it exercises abnormal conditions by using either invalid data input or the wrong user action.” For a device like iPhone, tapping the screen constantly while it is disabled, making a call while holding it upside down, or using a faulty docking cable can all be considered negative test cases.

Testing complex systems, regardless of physical size, is an incredibly difficult task. Some of this can be performed by humans and some through automated systems. Finding failures in highly integrated systems requires a combination of fixed test suites, test cases that reflect real usage scenarios, and the introduction of test escapes through creative randomization.

References

Chernak, Y. (2001). Validating and improving test case effectiveness. IEEE Software, January/February 2001.

Meyer, B. (2008). Seven principles of software testing. Computer, August 2008.

Savor, T. (2008). Testing feature-rich reactive systems. IEEE Software, July/August 2008.