Exploratory Testing with TestChimp
Why you should do exploratory testing, and What Makes It Hard
Exploratory testing is a hands-on approach to software testing where testers actively explore - by introducing variations of test interactions with the application without predefined scripts, allowing them to discover issues that traditional rigid automated tests might miss.
Importance of Exploratory Testing
Uncovers hidden defects: Exploratory testing can reveal bugs that are not easily detected through rigidly scripted automation tests.
Enhances understanding: It helps testers gain a deeper understanding of the application's functionality, user flows, and potential edge cases.
Adapts to changes: Exploratory testing is flexible and can quickly adapt to changes in the application, making it ideal for agile environments.
Challenges of Exploratory Testing
Requires skilled testers: Effective exploratory testing relies on the experience and intuition of skilled testers who can think critically and creatively.
Time-consuming: Manually exploring different test scenarios, especially for complex applications, can be labour-intensive and time-consuming.
Documentation: Keeping track of what has been tested and what remains to be tested can be challenging without proper documentation and tracking mechanisms.
Different Aspects That Should Be Covered in Exploratory Testing
To ensure comprehensive coverage during exploratory testing, testers should consider the following aspects:
Valid Inputs: Test the application with valid data to ensure it behaves as expected.
Invalid Inputs: Check how the application handles incorrect or unexpected data.
Idempotency of Operations: Verify that repeated operations produce consistent results, especially in APIs and backend services.
Authorised vs. Unauthorised Attempts: Test user journeys with different permission levels to ensure proper access control and security.
Boundary Conditions: Explore edge cases and boundary conditions to identify potential vulnerabilities and stability issues.
Error Handling: Assess how the application responds to errors and unexpected conditions.
Why It's Time-Consuming to Do It Manually Repetitively
Manual exploratory testing involves repeatedly trying out different variations of similar scenarios to uncover issues. This repetitive process can be time-consuming due to:
Extensive Combinations: The need to test multiple combinations of inputs, actions, and user roles.
Frequent Changes: Applications often undergo frequent updates, requiring testers to re-explore and verify functionalities.
Detailed Documentation: Manually documenting each test case and its results adds to the time and effort required.
Human Limitations: Testers may overlook certain scenarios or become fatigued during repetitive testing, leading to potential gaps in coverage.
How TestChimp Helps Build, Track & Automate Test Variations Quickly
TestChimp cuts down time taken in exploratory testing by simplifying the creation of test variations from a base recording, significantly reducing the manual effort required to execute those variations on an ongoing basis.
Introducing variation to a base test can be done in 2 ways broadly:
Doing different things (changing the set of steps involved in the test)
Doing things differently (changing the behaviour of a certain step / steps in the test - without changing the set of steps)
Doing different things
More formally, this refers to changing the set of steps / actions taken in the test, which can be a combination of the following:
Add steps: Do things that wasn’t done in the original test
Remove steps: Avoid doing certain steps of the original test
Repeat steps: Repeat the same step multiple times. Are idempotent outcomes remaining the same despite repetitions?
Reorder steps: Is the system expected to produce the same outcomes if the steps are done in a different order? Introducing variation through re-ordering steps helps verify this.
TestChimp allows users to easily add, remove, edit, re-order and duplicate test steps, making it easy to quickly create variations of the base test that “does different things” to the original.
Adding and Removing Steps
Imagine an e-commerce application where you need to verify the checkout process for different user journeys. In the initial test recording, steps from adding items to the cart to completing the purchase are captured.
Now let's say you wanted to create a variation of the scenario for when users apply a discount code. Traditionally, you would have to re-run the entire test sequence, with the only difference being an additional discount code addition step. With TestChimp, you can simply create a copy of the initial test, and add the discount code application step into the test sequence - without having to re-run the entire scenario manually with a discount code added.
Now imagine you wanted to create another variation for inputting invalid discount codes. If done manually, this would mean yet another duplicated effort of running the entire buy flow user journey, with the only change being applying an invalid discount code. With TestChimp, you can simply create a copy and only change the discount code step to have another variation built quickly.
Re-ordering and Duplicating Steps
One limitation of rigid scripts is that if the user does things in a different order than what was tested, it may lead to potentially breaking experiences to the user. This is why testing variations of the scenario with test step reordering is crucial.
Consider a scenario where you are testing an online booking system. Initially, you record a test where a user selects a flight, adds optional extras, and then proceeds to payment. You want to ensure the system handles different orderings of these steps correctly.
With TestChimp, you can re-order the steps within your recorded test. For example, you might re-order the steps to test adding optional extras before selecting a flight. This helps verify if the system behaves as expected regardless of the order in which actions are performed.
Additionally, you can duplicate the payment step to test idempotency, ensuring that repeated payment attempts do not result in duplicate charges. By reordering and duplicating steps, TestChimp allows you to explore various scenarios and edge cases without having to manually run those variations.
Doing things differently
This refers to trying out different inputs in a step, without changing the step set structure taken during the test. For instance, to test a search box, you may want to try inputting very large values, different character sets, nonsensical strings, numbers, small values etc. to make sure the system works in each such variation.
TestChimp uses the captured full stack recording to load the request payload structure and boilerplate for the involved test steps. Users can configure a value generator for each request field, which can make the requests generated to be dynamic and varied in each run. This allows for testing with varied inputs for the same user journey, without having to repeat the same boilerplate steps manually. TestChimp currently supports 3 value generators:
Variable
Vocabulary
Script
Variable Generator
Imagine a banking application where you need to verify fund transfers between accounts. The source account ID is obtained from a prior API response. When done manually, you would extract the account ID from the response and input it into subsequent requests repeatedly, which is tedious and error-prone.
With TestChimp, you can configure the transfer request to use the account ID variable extracted from the previous API call. During test execution, TestChimp dynamically populates this value, making the test repeatable without manual intervention.
Vocabulary Generator
Consider a search functionality within an e-commerce application that needs to be tested with various search queries. To ensure the search feature is robust and performs well with a wide range of inputs, different search terms of differing lengths, differing character sets / locales should be tried.
Manually entering each search term, submitting the query, and recording the results is a repetitive and time-consuming process. With TestChimp, define a vocabulary of search terms covering the different scenarios. Each test run selects a different search term from the vocabulary, allowing comprehensive coverage of the search functionality without manual input. This allows for running 100s of different variations of the same user journey without requiring 100s of laborious manual test executions and ensuring the feature is robust against the different variations of input.
Script Generator
Imagine a meeting scheduling app that requires entering a start time and a duration. Manually generating and inputting different time windows for each test run is labour-intensive and prone to errors.
With TestChimp, you can write a js script to generate random time windows dynamically. The script can produce a start time and a duration within a specified range, ensuring that each test run uses varied and valid time windows. This allows thorough testing of the scheduling functionality, ensuring it can handle a variety of time inputs correctly and consistently.
TestChimp comes pre-packed with a bunch of scripts for common scenarios such as:
Generate a random number
Generate a random valid US phone number
Generator a random valid US zip code
Time Manipulators:
Yesterday timestamp in YYYY-MM-DD format
Current time in millis
Alternatively, you can simply explain in plain english the required constraints on the expected values, and TestChimp will write a script for you that generates the values within the given constraints.
Conclusion
Exploratory testing is vital for uncovering hidden defects, enhancing understanding of the application, and adapting to changes quickly. However, it is challenging due to the need for skilled testers, the time-consuming nature of manually exploring different scenarios, and the difficulty of maintaining detailed tracking.
TestChimp simplifies exploratory testing by automating the creation of varied test cases through step variation and input data variation. By allowing users to easily add, remove, edit, and duplicate steps, TestChimp ensures comprehensive coverage of different scenarios without repetitive manual effort. Additionally, with dynamic value generators for inputs, TestChimp enables thorough testing with diverse data, making the process more efficient while achieving higher coverage.