Testing
Three tools, run_acceptance, create_test_accounts and remove_test_accounts, let the agent run business tests on a test copy of the app and try each role with throw-away accounts.
An app with an API is done when the tests in spec/acceptance.json pass on the deployed app, not when its files are written. For a hardware shop, say: a cashier sells 3 items and stock drops by 3, selling more than is left is refused with 409, warehouse staff can’t see revenue.
run_acceptance
Runs the tests in spec/acceptance.json on a test copy of the deployed app.
- Parameters:
project;only: the test ids to run, all by default. A test that needs a value another test saves pulls that test in too. - Returns:
passed,total, each test’s result with its first failing step;unprovenCapabilities(declared capabilities such as cron or email that no test proves yet) andmissingCapabilities(capabilities the code doesn’t have, with the block to add). - When the agent calls it: after every deploy, until every test passes.
The test copy runs the deployed code with its own database, wiped before every run (then loaded with spec/sample-data.json when there is one). Live data is never touched. Each test creates what it needs through the API, a product and a customer for instance, before it acts.
Each step is one of:
- An API call as a role (
as) or as a visitor, checkingstatusand JSON fields. Fields are dotted paths:items.0.qty,items.length, or an array item picked by a field:items[id=setup].donereadsdoneof the first item whoseidissetup. sql: read or change rows in the test copy’s database, such as backdating a date instead of waiting.cron: run a scheduled job now and wait for it.outbox: check that the app sent an e-mail or Telegram message with that text. For mail to a test account,containslooks in the subject and the body (the first ~600 characters, links written with their address), so a test can prove the link, the order number or the total in the mail.parallel: fire several requests at once, to prove two cashiers can’t sell the same last item.page: open a page in a real browser as a role; fails on a crash, a blank page or missing text. A page that is blank on the first read is opened again about 2 seconds later before it counts as a failure, since charts can still be drawing.
deploy also runs the tests for about 30 seconds and returns them under tests. run_acceptance runs whatever didn’t fit. A cut-short run is still saved: get_project says how many passed and how many weren’t reached. Until spec/acceptance.json exists, manh.run makes a few smoke tests from your files: each role opens the home page, the other pages open, every list endpoint answers 200 or 403, never 500.
A run that cannot start, for instance a test signing in as a role (as) that app/auth.ts no longer has, is reported with its reason by get_project and by the deploy’s handover. A run cut off before it saved anything is reported as such by get_project, which asks for run_acceptance instead of saying “never run”.
Changed the values of an enum (defineEnum, enum in a table file)? Fix the rows of spec/sample-data.json and the bodies of spec/acceptance.json that still use the old ones too. check warns about values the enum no longer has.
create_test_accounts
Makes one throw-away sign-in per role on the running app, for trying each role’s flow by hand.
- Parameters:
project;roles: the roles to create, every role inapp/auth.tsby default. - Returns: an e-mail and password per role and how to sign in (
POST /api/auth/loginfor atoken). Calling again gives new passwords. - When the agent calls it: to try the manager’s or accountant’s screens on the live app before you sign up yourself.
Test accounts use the reserved domain @test.manh.invalid, never a real mailbox. Mail sent there never leaves; it only shows in the app’s logs. Needs a deployed app with app/auth.ts.
remove_test_accounts
Deletes every test account from the users table.
- Parameters:
project. - Returns: how many accounts were deleted; accounts that other rows still point to are deactivated (can’t sign in) instead, with a list so the agent can clean those rows up.
- When the agent calls it: before handing the app over.
The first admin account is always yours: open the app in the dashboard preview and sign up. A test account never counts as the first admin.
Next steps
- deploy: every deploy runs the tests and returns
handover. - Database: read the live data after trying flows by hand.
- Sign-in and permissions: roles in
app/auth.ts.