Test File Format - OpenTester

OpenTester stores all tests as human-readable JSON files in tests/e2e/. This guide explains the test format so you can understand, modify, and create tests manually.

File Location & Naming

Tests are stored in: tests/e2e/ Example paths:

tests/e2e/login_flow.json
tests/e2e/user_profile_navigation.json
tests/e2e/checkout_complete.json

Use descriptive, kebab-case names that describe what the test does.

Basic Structure

Here’s a minimal test:

{
  "name": "Simple app launch",
  "description": "Verify the app launches and home screen is visible",
  "preconditions": {
    "appLaunch": true,
    "clearData": false
  },
  "steps": [
    {
      "action": "wait",
      "timeout": 3000
    }
  ],
  "assertions": [
    {
      "type": "text visibility",
      "target": "Welcome"
    }
  ],
  "config": {
    "useAIFallback": true
  }
}

Full Test Example

Here’s a realistic, complete test:

{
  "name": "User login flow",
  "description": "Test successful user login with valid credentials",
  "preconditions": {
    "appLaunch": true,
    "clearData": true,
    "clearCache": true
  },
  "steps": [
    {
      "action": "wait",
      "timeout": 2000
    },
    {
      "action": "tap",
      "target": "email input field",
      "timeout": 5000
    },
    {
      "action": "type",
      "value": "testuser@example.com",
      "timeout": 5000
    },
    {
      "action": "tap",
      "target": "password input field"
    },
    {
      "action": "type",
      "value": "SecurePassword123!"
    },
    {
      "action": "tap",
      "target": "Login button"
    },
    {
      "action": "wait",
      "timeout": 3000
    }
  ],
  "assertions": [
    {
      "type": "text visibility",
      "target": "Dashboard",
      "timeout": 5000
    },
    {
      "type": "element presence",
      "target": "Profile button",
      "timeout": 3000
    }
  ],
  "config": {
    "useAIFallback": true,
    "screenshotOnFailure": true,
    "retryOnFailure": 1
  }
}

Section Breakdown

name & description

{
  "name": "User login flow",
  "description": "Test successful user login with valid credentials"
}

name: Short title for the test (appears in test lists)
description: Longer explanation of what the test does (for documentation)

preconditions

{
  "preconditions": {
    "appLaunch": true,
    "clearData": false,
    "clearCache": false
  }
}

Setup actions before the test runs:

Field	Type	Default	Description
`appLaunch`	boolean	`true`	Whether to launch the app at the start
`clearData`	boolean	`false`	Clear app data (like signing out users)
`clearCache`	boolean	`false`	Clear app cache files

steps

The sequence of actions to perform:

{
  "steps": [
    {
      "action": "tap",
      "target": "Login button",
      "timeout": 5000
    }
  ]
}

Step Actions

tap

{
  "action": "tap",
  "target": "Login button",
  "timeout": 5000
}

Taps/clicks an element
target: Text description of the element (vision-based detection)
timeout: Maximum milliseconds to wait for element

type

{
  "action": "type",
  "value": "user@example.com",
  "timeout": 3000
}

Types text into a focused input field
value: Text to type
timeout: Maximum milliseconds to wait

scroll

{
  "action": "scroll",
  "direction": "down",
  "amount": 3,
  "timeout": 2000
}

Scrolls the screen
direction: "up", "down", "left", or "right"
amount: Number of scroll swipes
timeout: Maximum milliseconds to wait

wait

{
  "action": "wait",
  "timeout": 2000
}

Pauses execution
timeout: Milliseconds to wait

swipe

{
  "action": "swipe",
  "direction": "right",
  "speed": "fast",
  "timeout": 2000
}

Performs a swipe gesture
direction: "up", "down", "left", or "right"
speed: "slow" or "fast"

long_press

{
  "action": "long_press",
  "target": "Message item",
  "duration": 1000,
  "timeout": 5000
}

Long-press/hold on an element
target: Element to press
duration: Milliseconds to hold
timeout: Max wait for element

assertions

Verifications that confirm expected behavior:

{
  "assertions": [
    {
      "type": "text visibility",
      "target": "Welcome, User",
      "timeout": 5000
    }
  ]
}

Assertion Types

text visibility

{
  "type": "text visibility",
  "target": "Expected text",
  "timeout": 5000
}

Checks that specific text is visible on screen

element presence

{
  "type": "element presence",
  "target": "Sign out button",
  "timeout": 5000
}

Checks that an element exists (regardless of visibility)

element absence

{
  "type": "element absence",
  "target": "Loading spinner",
  "timeout": 5000
}

Checks that an element is NOT present

url contains

{
  "type": "url contains",
  "target": "/dashboard",
  "timeout": 3000
}

Checks that the current URL contains a string (web apps)

config

Test configuration options:

{
  "config": {
    "useAIFallback": true,
    "screenshotOnFailure": true,
    "retryOnFailure": 1
  }
}

Field	Type	Default	Description
`useAIFallback`	boolean	`true`	Use AI vision if element not found
`screenshotOnFailure`	boolean	`true`	Capture screenshot when test fails
`retryOnFailure`	number	`0`	Retry count if test fails

Real-World Examples

Example 1: Search Flow

{
  "name": "App search flow",
  "description": "Search for a product and view results",
  "preconditions": {
    "appLaunch": true,
    "clearData": false
  },
  "steps": [
    {
      "action": "wait",
      "timeout": 1000
    },
    {
      "action": "tap",
      "target": "Search icon"
    },
    {
      "action": "type",
      "value": "blue shoes"
    },
    {
      "action": "wait",
      "timeout": 2000
    }
  ],
  "assertions": [
    {
      "type": "text visibility",
      "target": "Results for"
    },
    {
      "type": "element presence",
      "target": "Product list"
    }
  ],
  "config": {
    "useAIFallback": true
  }
}

Example 2: Onboarding Flow

{
  "name": "App onboarding",
  "description": "Complete first-time user onboarding",
  "preconditions": {
    "appLaunch": true,
    "clearData": true
  },
  "steps": [
    {
      "action": "wait",
      "timeout": 2000
    },
    {
      "action": "tap",
      "target": "Get Started button"
    },
    {
      "action": "wait",
      "timeout": 1000
    },
    {
      "action": "tap",
      "target": "Next button"
    },
    {
      "action": "wait",
      "timeout": 1000
    },
    {
      "action": "tap",
      "target": "Next button"
    },
    {
      "action": "wait",
      "timeout": 1000
    },
    {
      "action": "tap",
      "target": "Done button"
    }
  ],
  "assertions": [
    {
      "type": "text visibility",
      "target": "Home"
    }
  ],
  "config": {
    "useAIFallback": true
  }
}

{
  "name": "Bottom navigation",
  "description": "Test navigation between main app screens",
  "preconditions": {
    "appLaunch": true,
    "clearData": false
  },
  "steps": [
    {
      "action": "wait",
      "timeout": 1500
    },
    {
      "action": "tap",
      "target": "Profile tab"
    },
    {
      "action": "wait",
      "timeout": 1000
    },
    {
      "action": "tap",
      "target": "Settings tab"
    },
    {
      "action": "wait",
      "timeout": 1000
    },
    {
      "action": "tap",
      "target": "Home tab"
    }
  ],
  "assertions": [
    {
      "type": "text visibility",
      "target": "Welcome"
    }
  ],
  "config": {
    "useAIFallback": true
  }
}

Best Practices

Naming Targets

OpenTester uses vision-based element detection, so be descriptive: ✅ Good target names:

"Login button"
"Email input field"
"Username text at top of screen"
"Settings gear icon"

❌ Poor target names:

"button"
"element"
"something"

Timeouts

Set appropriate timeouts based on action:

Quick actions (tap): 3000-5000 ms
Network requests: 5000-10000 ms
App transitions: 2000-3000 ms

Step Best Practices

{
  "steps": [
    {
      "action": "tap",
      "target": "Login button"
    },
    {
      "action": "wait",
      "timeout": 2000
    },
    {
      "action": "tap",
      "target": "Profile button"
    }
  ]
}

Add waits after actions that load new screens
Use meaningful timeout values
Keep steps sequential and simple
Avoid testing too many things in one test

Assertion Best Practices

{
  "assertions": [
    {
      "type": "text visibility",
      "target": "Success message",
      "timeout": 5000
    }
  ]
}

Include multiple assertions to verify complete state
Use specific text rather than generic messages
Include timeouts for network-dependent assertions

Creating Tests Manually

You can create tests by hand if needed:

Create a .json file in tests/e2e/
Use the structure from this guide
Run the test with OpenTester

Example:

# Create the file
cat > tests/e2e/my_test.json << 'EOF'
{
  "name": "My test",
  ...
}
EOF

# Then validate it
# (Through Claude Code or Gemini CLI)

Schema Validation

OpenTester validates test files on load. Invalid tests will show errors. Common issues:

Missing name field
Invalid action names
Missing target on tap/scroll actions
Assertion type not recognized

Make sure your JSON is valid and matches the examples in this guide.

Modifying Existing Tests

You can edit tests directly in your editor:

Open the test file in tests/e2e/
Modify steps or assertions
Save and validate with OpenTester
Commit to version control

This is useful for:

Fixing element targets (if UI changed)
Adding new assertions
Adjusting timeouts
Updating test descriptions

​File Location & Naming

​Basic Structure

​Full Test Example

​Section Breakdown

​name & description

​preconditions

​steps

​Step Actions

​assertions

​Assertion Types

​config

​Real-World Examples

​Example 1: Search Flow

​Example 2: Onboarding Flow

​Example 3: Navigation Test

​Best Practices

​Naming Targets

​Timeouts

​Step Best Practices

​Assertion Best Practices

​Creating Tests Manually

​Schema Validation

​Modifying Existing Tests