Skip to Content

Testing

Three tools, run_acceptance, create_test_accounts and remove_test_accounts, let the agent run business tests on a test copy of the app and try each role with throw-away accounts.

An app with an API is done when the tests in spec/acceptance.json pass on the deployed app, not when its files are written. For a hardware shop, say: a cashier sells 3 items and stock drops by 3, selling more than is left is refused with 409, warehouse staff can’t see revenue.

run_acceptance

Runs the tests in spec/acceptance.json on a test copy of the deployed app.

  • Parameters: project; only: the test ids to run, all by default. A test that needs a value another test saves pulls that test in too.
  • Returns: passed, total, each test’s result with its first failing step; unprovenCapabilities (declared capabilities such as cron or email that no test proves yet) and missingCapabilities (capabilities the code doesn’t have, with the block to add).
  • When the agent calls it: after every deploy, until every test passes.

The test copy runs the deployed code with its own database, wiped before every run (then loaded with spec/sample-data.json when there is one). Live data is never touched. Each test creates what it needs through the API, a product and a customer for instance, before it acts.

Each step is one of:

  • An API call as a role (as) or as a visitor, checking status and JSON fields. Fields are dotted paths: items.0.qty, items.length, or an array item picked by a field: items[id=setup].done reads done of the first item whose id is setup.
  • sql: read or change rows in the test copy’s database, such as backdating a date instead of waiting.
  • cron: run a scheduled job now and wait for it.
  • outbox: check that the app sent an e-mail or Telegram message with that text. For mail to a test account, contains looks in the subject and the body (the first ~600 characters, links written with their address), so a test can prove the link, the order number or the total in the mail.
  • parallel: fire several requests at once, to prove two cashiers can’t sell the same last item.
  • page: open a page in a real browser as a role; fails on a crash, a blank page or missing text. A page that is blank on the first read is opened again about 2 seconds later before it counts as a failure, since charts can still be drawing.

deploy also runs the tests for about 30 seconds and returns them under tests. run_acceptance runs whatever didn’t fit. A cut-short run is still saved: get_project says how many passed and how many weren’t reached. Until spec/acceptance.json exists, manh.run makes a few smoke tests from your files: each role opens the home page, the other pages open, every list endpoint answers 200 or 403, never 500.

A run that cannot start, for instance a test signing in as a role (as) that app/auth.ts no longer has, is reported with its reason by get_project and by the deploy’s handover. A run cut off before it saved anything is reported as such by get_project, which asks for run_acceptance instead of saying “never run”.

Changed the values of an enum (defineEnum, enum in a table file)? Fix the rows of spec/sample-data.json and the bodies of spec/acceptance.json that still use the old ones too. check warns about values the enum no longer has.

create_test_accounts

Makes one throw-away sign-in per role on the running app, for trying each role’s flow by hand.

  • Parameters: project; roles: the roles to create, every role in app/auth.ts by default.
  • Returns: an e-mail and password per role and how to sign in (POST /api/auth/login for a token). Calling again gives new passwords.
  • When the agent calls it: to try the manager’s or accountant’s screens on the live app before you sign up yourself.

Test accounts use the reserved domain @test.manh.invalid, never a real mailbox. Mail sent there never leaves; it only shows in the app’s logs. Needs a deployed app with app/auth.ts.

remove_test_accounts

Deletes every test account from the users table.

  • Parameters: project.
  • Returns: how many accounts were deleted; accounts that other rows still point to are deactivated (can’t sign in) instead, with a list so the agent can clean those rows up.
  • When the agent calls it: before handing the app over.

The first admin account is always yours: open the app in the dashboard preview and sign up. A test account never counts as the first admin.

Next steps

Last updated