...\" @OPENGROUP_COPYRIGHT@ ...\" COPYRIGHT NOTICE ...\" Copyright (c) 1990, 1991, 1992, 1993 Open Software Foundation, Inc. ...\" Copyright (c) 1996, 1997, 1998, 1999, 2000 The Open Group ...\" ALL RIGHTS RESERVED (MOTIF). See the file named COPYRIGHT.MOTIF for ...\" the full copyright text. ...\" ...\" This software is subject to an open license. It may only be ...\" used on, with or for operating systems which are themselves open ...\" source systems. You must contact The Open Group for a license ...\" allowing distribution and sublicensing of this software on, with, ...\" or for operating systems which are not Open Source programs. ...\" ...\" See http://www.opengroup.org/openmotif/license for full ...\" details of the license agreement. Any use, reproduction, or ...\" distribution of the program constitutes recipient's acceptance of ...\" this agreement. ...\" ...\" EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, THE PROGRAM IS ...\" PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY ...\" KIND, EITHER EXPRESS OR IMPLIED INCLUDING, WITHOUT LIMITATION, ANY ...\" WARRANTIES OR CONDITIONS OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY ...\" OR FITNESS FOR A PARTICULAR PURPOSE ...\" ...\" EXCEPT AS EXPRESSLY SET FORTH IN THIS AGREEMENT, NEITHER RECIPIENT ...\" NOR ANY CONTRIBUTORS SHALL HAVE ANY LIABILITY FOR ANY DIRECT, ...\" INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL ...\" DAMAGES (INCLUDING WITHOUT LIMITATION LOST PROFITS), HOWEVER CAUSED ...\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT ...\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ...\" ANY WAY OUT OF THE USE OR DISTRIBUTION OF THE PROGRAM OR THE ...\" EXERCISE OF ANY RIGHTS GRANTED HEREUNDER, EVEN IF ADVISED OF THE ...\" POSSIBILITY OF SUCH DAMAGES. ...\" ...\" ...\" HISTORY # $XConsortium: ch04.sm /main/3 1995/07/13 20:09:16 drk $ ...\" (c) Copyright 1992 OPEN SOFTWARE FOUNDATION, INC. .H 1 "Debugging the Automated Tests" .P Interpreting test results often requires debugging each individual discrepancy until its cause can be determined. The follow sections describe how the automation technology supports debugging. .H 2 "Types of Failures" .P This section describes the types of failures which may be found by automated testing. The next section provides some hints on how to go about debugging those failures. .H 3 "Visual Comparison Failures" .P A common visual comparison failure occurs when the original, recorded image differs in size from the current image under test. Because the automation code does not know how to compare different-sized images, it simply issues a warning. In interactive mode, the original region will be displayed in the dialog box, although the overlay feature is disabled. One can, however, use the X client \*Lxmag\*O and count pixels to help determine what has changed. .P When two images are the same size, the automation code does a pixel-by-pixel comparison of the images. If any one pixel differs between the two images, the automation code prints out a message that a CompareVisual failure has occurred. In interactive mode, the tester can examine both the recorded and current images, and overlay the differences (highlighted in red) on top of either image. .P There are several problems that can produce a \*LVisual Image Not Found\*O error: the recorded image is missing or corrupted; the environment variable \*L$AUTOVPATH\*O is not defined; the directory defined by \*L$AUTOVPATH\*O does not exist, or the visual file in that directory is not readable or is missing. .P This error can also occur if the test or the script was modified between the recording and the playback. For instance, a test is recorded with a script that specifies six visual comparisons. Six images are stored in the \*L.vis\*O file. Later, a tester adds an additional \*LCompareVisual\*O command to the script but fails to rerecord the test. On the next playback, the seventh visual comparison fails because no image is stored with which to compare the current image. .P Either a \*LChecksum\*O or a \*LVisual Image Corrupt\*O error can occur when something in the visuals file has become corrupted. The only way to fix this problem is to rerecord the original, using the original baseline. .H 3 "Fatal Errors" .P The signals are trapped by the automation code, and redirected to a special routine \fBAutoExitSignal\*O which simply prints out a "signal received" message, and the type of signal. Signals so captured are: .sp .in +3 .VL 1i .LI "\*LSIGHUP\*O" hangup .LI "\*LSIGINT\*O" interrupt .LI "\*LSIGQUIT\*O" quit .LI "\*LSIGKILL\*O" kill .LI "\*LSIGSEGV\*O" segmentation violation .LI "\*LSIGFPE\*O" arithmetic (floating point) exception .LI "\*LSIGILL\*O" illegal instruction .LI "\*LSIGTERM\*O" software termination signal .LI "\*LSIGABRT\*O" abort .LI "\*LSIGBUS\*O" bus error .LI "\*LSIGSYS\*O" bad argument to system call .LE .in -3 .sp .fi .P Be certain in reviewing the differences between files that you search for signal messages. .H 3 "Other Failures" .P Many other kinds of failures only show up by examining the contents of the differences file (\*L.err\*O) produced by the RUN script; this differences file is the product of the command \*Ldiff\*O run against the output file stored at record time and the output file for the current run. This section describes some examples of what differences might turn up; very subtle bugs may make their appearance known only through small differences in the output. .P .BL .LI Messages printed by callback routines can indicate more or fewer callbacks are occurring on some action. .LI Differences in size reported by callback routines may show layout differences not captured by CompareVisual, usually because the top-level manager is not photographed. .LI User data printed out may indicate a test was run in the wrong mode. .LI CompareVisual failure messages will turn up in diffs. .LI Running a non-automated \*Lmwm\*O can produce window command failures which show up as differences. This may precipitate CompareVisual failures further along in the test. .LI Key-binding error messages may appear in diffs if the \fBqauser\*O environment in not used. .LI \*LxisMovePointer\*O errors may mean that layout has changed, and the pointer can no longer move to a certain widget. .LI If tests are run in interactive mode, "Accept" messages will appear in diffs but can be ignored. .LE .H 2 "Debugging Tricks and Testing Strategies" .P This section will describe several automation features which can be used to help debug test failures. While nothing is as good as first-hand experience, this section will give you some idea of the tools at your disposal. .P Generally, one will wish to run the entire test suite automatically then debug the failures one at a time. If the failures appear to be due (in whole or in part) to visual differences, it often makes the most sense to run the offending test in interactive mode. The various overlay capabilities of interactive mode have been described earlier. This is the quickest way to assess the nature of the change; once you know what the difference is you can debug the cause. However, there are other modes and features which can be used. .H 3 "The \*L-a\*O Flag" The precursor to the interactive window was the \*L-a\*O flag. When run with this flag, tests print out the x,y location of each pixel that differs from the original. If the differences are really a single or two pixels and you have difficulty seeing the differences in interactive mode, the information provided by \*L-a\*O may be helpful. .H 3 "Window Dumps" Another visual debugging tool which predates the interactive mode window is the window dump feature. Tests run with the \*L-W\*O flag create xwd-format window dumps which can subsequently be viewed with the \*Lxwud\*O client. Your company may have other tools that use xwd-format images; if so, this option may be useful for you. .H 3 "The \*LManual\*O Command" For certain problems, you will need to be able to manipulate the widgets in the test client directly. Because automation takes control of the pointer, we have provided a scripting language command \*LManual\*O which tells the automation code to suspend its normal event handling and return control to the user, until the user clicks the Continue button on the Manual mode DialogBox. At that point the test will resume. This gives you an opportunity to examine closely what happens at very specific points during the test. .P To use \*LManual\*O, simply add the command at the appropriate point in the test's \*L.scr\*O file and remake the \*L.Scr\*O file. When the Manual command is reached, a special dialog box will appear and you may exercise the test client normally. At this point, you can also invoke other useful X debugging clients, such as \*Lxmag\*O, \*Lxwininfo\*O, or the OSF/Motif test tool \*Lxmruler\*O. .H 3 "Delay Mode" If you need the playback slowed enough for a human tester to watch the changes occurring, but don't actually require the suspension of the event loop provided by \*LManual\*O, the \*L-D\*O or \*LDelay\*O mode might be helpful. Run with \*L-D \*Vseconds\*O, the code "sleeps" for the requested number of seconds before executing the next instruction. The Delay flag can also be used when very fast workstations execute instructions before the visuals have "caught up". .H 3 "Trace Mode" The \*L-T\*O or \*LTrace\*O mode is useful in many debugging situations. In Trace mode, the scripting instructions are echoed to the output file as they are executed. This greatly simplies, for instance, the process of determining which button produces an extra Activate callback. It may be worth considering using the Trace mode by default for all recording and playback. This can also help determine cases where failures are merely the result of recorded output files getting out-of-sync with the tests or scripts currently executing. .H 3 "The \*Lxmruler\*O Tool" The tool \*Lxmruler\*O is provided in the directory \*L$TOP/tests/tools\*O. This client allows the user to "measure" widgets on screen which claim to be a certain size.