In short:
ak-ab-test-setuphelps you write the hypothesis, pick the metrics, compute the sample size, split traffic and lock the stopping rule before the test runs. It also helps you read the results when the test is done. Measurement and copy for the variants go to the matching skills you have installed.
Disclosure: I maintain AgentKit and have an affiliate relationship with it: the AgentKit link in this post is an affiliate link; buying through it gets you 30% off.
Summary
Use it when you want to try a change on a page or feature, but do not want to run a "let's just try it" test. It is not for hypotheses while debugging.
flowchart LR
A["Page or feature to improve"] --> B["ak-ab-test-setup"]
B --> C["Test plan"]
C --> D["Installed copywriting skill: copy for variants"]
C --> E["Installed analytics skill: measurement"]
D --> F["Run the test with your tool"]
E --> F
F --> G["ak-ab-test-setup: read the results"]Four concepts to know:
| Concept | Question | Example |
|---|---|---|
| Hypothesis | What do we change, why, and what result do we expect for whom? | Because the heatmap shows new visitors barely notice the sign-up button, a bigger, higher-contrast button will raise the click rate by 15% or more for new visitors |
| Three kinds of metrics | Which number decides, which explains, which must not get worse? | The primary metric is the button click rate; secondary metrics are time to click and scroll depth; guardrails are the bounce rate and conversion at the next step |
| Sample size and minimum lift | How many people per variant to detect the lift you care about? | Baseline rate 5%, wanting to detect a 20% lift: about 7k per variant, at 95% confidence and 80% power |
| Stopping rule | When may you stop the test? | A fixed-sample test runs to its full sample; a sequential test follows its declared stopping boundary; hitting a harm condition stops it |
Each test changes only one thing, so you know for sure what made the difference. If baseline figures such as the current conversion rate or traffic are missing, it asks for them.
Sample prompt for a CTA button test:
/ak-ab-test-setup The sign-up button on the home page of <product name>.
Today: click rate about 5%, the page gets <visits> visits a day.
Want to try: a bigger button, higher-contrast color, for new visitors.
Reason: the heatmap shows few people notice the button.
Constraint: must not lower the sign-up completion rate.This skill has no flags.
Below is the detailed version: the run flow, 5 situations with prompts, the change history by ak release, and common pitfalls. The detailed version and the interactive page are for subscribers. Your agent can also search them through the portal's MCP.
Opens when your account has subscriber access
This guide is for subscribers
Once your account has access, the full piece, the interactive visual and the agent call all open.
- Sign in or create a modernEngineer accountReading access belongs to your account email.
- Choose a plan on the pricing page1, 3 or 12 months, paid by VietQR bank transfer. Access opens about a minute after the money arrives.
- Or redeem a Facebook subscriber codePeople supporting the page on Facebook get an activation code by message each month.
Already a subscriber? Sign in to read.
Use this piece in your agent
Get an API keyCopy the text below and paste it into Claude Code, Codex or whichever agent you use. The agent reads the piece and checks it against the repo you have open.
Read "ak-ab-test-setup: plan an A/B test before you run it" on modernEngineer and apply it to this repo.
1. Call the content_get tool of the modernengineer MCP server with locale "en", slug "ak-ab-test-setup". If the modernengineer MCP server is not connected, stop and walk me through connecting it from https://modernengineer.dev/en/mcp.
2. Summarize the main steps of the piece and point out which apply to this repo.
3. Propose concrete changes and ask me before editing any file.Connect MCP in Claude Code (once)
claude mcp add --transport http modernengineer \
https://api.modernengineer.dev/mcp \
--header "Authorization: Bearer <API key>"Reading the full piece over MCP needs an API key from a subscriber account or one that bought this guide. Without access, the agent only gets the opening.
- Version
- v1.0.0
- Updated
- 22/10/2026
- Licence
- All rights reserved
- Slug
- ak-ab-test-setup
