top of page

Exploratory Testing with TestChimp



Why you should do exploratory testing, and What Makes It Hard


Exploratory testing is a hands-on approach to software testing where testers actively explore - by introducing variations of test interactions with the application without predefined scripts, allowing them to discover issues that traditional rigid automated tests might miss.


Importance of Exploratory Testing


  • Uncovers hidden defects: Exploratory testing can reveal bugs that are not easily detected through rigidly scripted automation tests.

  • Enhances understanding: It helps testers gain a deeper understanding of the application's functionality, user flows, and potential edge cases.

  • Adapts to changes: Exploratory testing is flexible and can quickly adapt to changes in the application, making it ideal for agile environments.


Challenges of Exploratory Testing


  • Requires skilled testers: Effective exploratory testing relies on the experience and intuition of skilled testers who can think critically and creatively.

  • Time-consuming: Manually exploring different test scenarios, especially for complex applications, can be labour-intensive and time-consuming.

  • Documentation: Keeping track of what has been tested and what remains to be tested can be challenging without proper documentation and tracking mechanisms.


Different Aspects That Should Be Covered in Exploratory Testing


To ensure comprehensive coverage during exploratory testing, testers should consider the following aspects:

  1. Valid Inputs: Test the application with valid data to ensure it behaves as expected.

  2. Invalid Inputs: Check how the application handles incorrect or unexpected data.

  3. Idempotency of Operations: Verify that repeated operations produce consistent results, especially in APIs and backend services.

  4. Authorised vs. Unauthorised Attempts: Test user journeys with different permission levels to ensure proper access control and security.

  5. Boundary Conditions: Explore edge cases and boundary conditions to identify potential vulnerabilities and stability issues.

  6. Error Handling: Assess how the application responds to errors and unexpected conditions.


Why It's Time-Consuming to Do It Manually Repetitively


Manual exploratory testing involves repeatedly trying out different variations of similar scenarios to uncover issues. This repetitive process can be time-consuming due to:

  • Extensive Combinations: The need to test multiple combinations of inputs, actions, and user roles.

  • Frequent Changes: Applications often undergo frequent updates, requiring testers to re-explore and verify functionalities.

  • Detailed Documentation: Manually documenting each test case and its results adds to the time and effort required.

  • Human Limitations: Testers may overlook certain scenarios or become fatigued during repetitive testing, leading to potential gaps in coverage.


How TestChimp Helps Build, Track & Automate Test Variations Quickly


TestChimp cuts down time taken in exploratory testing by simplifying the creation of test variations from a base recording, significantly reducing the manual effort required to execute those variations on an ongoing basis. 


Introducing variation to a base test can be done in 2 ways broadly:

  1. Doing different things (changing the set of steps involved in the test)

  2. Doing things differently (changing the behaviour of a certain step / steps in the test - without changing the set of steps)


Doing different things


More formally, this refers to changing the set of steps / actions taken in the test, which can be a combination of the following:

  • Add steps: Do things that wasn’t done in the original test

  • Remove steps: Avoid doing certain steps of the original test

  • Repeat steps: Repeat the same step multiple times. Are idempotent outcomes remaining the same despite repetitions?

  • Reorder steps: Is the system expected to produce the same outcomes if the steps are done in a different order? Introducing variation through re-ordering steps helps verify this.

TestChimp allows users to easily add, remove, edit, re-order and duplicate test steps, making it easy to quickly create variations of the base test that “does different things” to the original.


Adding and Removing Steps

Imagine an e-commerce application where you need to verify the checkout process for different user journeys. In the initial test recording, steps from adding items to the cart to completing the purchase are captured.


Now let's say you wanted to create a variation of the scenario for when users apply a discount code. Traditionally, you would have to re-run the entire test sequence, with the only difference being an additional discount code addition step. With TestChimp, you can simply create a copy of the initial test, and add the discount code application step into the test sequence - without having to re-run the entire scenario manually with a discount code added.


Now imagine you wanted to create another variation for inputting invalid discount codes. If done manually, this would mean yet another duplicated effort of running the entire buy flow user journey, with the only change being applying an invalid discount code. With TestChimp, you can simply create a copy and only change the discount code step to have another variation built quickly.


Re-ordering and Duplicating Steps

One limitation of rigid scripts is that if the user does things in a different order than what was tested, it may lead to potentially breaking experiences to the user. This is why testing variations of the scenario with test step reordering is crucial.


Consider a scenario where you are testing an online booking system. Initially, you record a test where a user selects a flight, adds optional extras, and then proceeds to payment. You want to ensure the system handles different orderings of these steps correctly.


With TestChimp, you can re-order the steps within your recorded test. For example, you might re-order the steps to test adding optional extras before selecting a flight. This helps verify if the system behaves as expected regardless of the order in which actions are performed.


Additionally, you can duplicate the payment step to test idempotency, ensuring that repeated payment attempts do not result in duplicate charges. By reordering and duplicating steps, TestChimp allows you to explore various scenarios and edge cases without having to manually run those variations.


Doing things differently


This refers to trying out different inputs in a step, without changing the step set structure taken during the test. For instance, to test a search box, you may want to try inputting very large values, different character sets, nonsensical strings, numbers, small values etc. to make sure the system works in each such variation.


TestChimp uses the captured full stack recording to load the request payload structure and boilerplate for the involved test steps. Users can configure a value generator for each request field, which can make the requests generated to be dynamic and varied in each run. This allows for testing with varied inputs for the same user journey, without having to repeat the same boilerplate steps manually. TestChimp currently supports 3 value generators:


  1. Variable

  2. Vocabulary

  3. Script


Variable Generator

Imagine a banking application where you need to verify fund transfers between accounts. The source account ID is obtained from a prior API response. When done manually, you would extract the account ID from the response and input it into subsequent requests repeatedly, which is tedious and error-prone.


With TestChimp, you can configure the transfer request to use the account ID variable extracted from the previous API call. During test execution, TestChimp dynamically populates this value, making the test repeatable without manual intervention.


Vocabulary Generator 

Consider a search functionality within an e-commerce application that needs to be tested with various search queries. To ensure the search feature is robust and performs well with a wide range of inputs, different search terms of differing lengths, differing character sets / locales should be tried.


Manually entering each search term, submitting the query, and recording the results is a repetitive and time-consuming process. With TestChimp, define a vocabulary of search terms covering the different scenarios. Each test run selects a different search term from the vocabulary, allowing comprehensive coverage of the search functionality without manual input. This allows for running 100s of different variations of the same user journey without requiring 100s of laborious manual test executions and ensuring the feature is robust against the different variations of input.


Script Generator 

Imagine a meeting scheduling app that requires entering a start time and a duration. Manually generating and inputting different time windows for each test run is labour-intensive and prone to errors.


With TestChimp, you can write a js script to generate random time windows dynamically. The script can produce a start time and a duration within a specified range, ensuring that each test run uses varied and valid time windows. This allows thorough testing of the scheduling functionality, ensuring it can handle a variety of time inputs correctly and consistently.


TestChimp comes pre-packed with a bunch of scripts for common scenarios such as:

  • Generate a random number

  • Generate a random valid US phone number

  • Generator a random valid US zip code

  • Time Manipulators:

    • Yesterday timestamp in YYYY-MM-DD format

    • Current time in millis


Alternatively, you can simply explain in plain english the required constraints on the expected values, and TestChimp will write a script for you that generates the values within the given constraints.


Conclusion


Exploratory testing is vital for uncovering hidden defects, enhancing understanding of the application, and adapting to changes quickly. However, it is challenging due to the need for skilled testers, the time-consuming nature of manually exploring different scenarios, and the difficulty of maintaining detailed tracking.


TestChimp simplifies exploratory testing by automating the creation of varied test cases through step variation and input data variation. By allowing users to easily add, remove, edit, and duplicate steps, TestChimp ensures comprehensive coverage of different scenarios without repetitive manual effort. Additionally, with dynamic value generators for inputs, TestChimp enables thorough testing with diverse data, making the process more efficient while achieving higher coverage.

bottom of page