# Hass Bench

https://hass-bench.skillsafe.ai/

Paste a Home Assistant automation or script as YAML. The page reads it in your browser,
runs forty deterministic checks over it, dry-runs it against a world state you set, and
then offers four model lanes over the same document.

Everything above the run button is free, needs no account, and never uploads anything.

## The thing this exists for

Home Assistant loads YAML through a loader derived from PyYAML's SafeLoader, and PyYAML
implements YAML **1.1** scalar resolution. Under those rules the unquoted plain scalar
`on` is the boolean `true`. So this trigger:

    trigger:
      - platform: state
        entity_id: binary_sensor.front_door
        to: on

does not test for the string `"on"` that a binary sensor actually reports. It tests for
`True`, never matches, and the automation silently never runs. Nothing appears in the
log, because nothing is wrong with the file. It is valid configuration that is simply
never true.

A JavaScript YAML library implementing YAML 1.2 resolves `on` to the string `"on"` and
reports no problem at all, which is why this app ships its own reader rather than
vendoring one. The same resolver rule turns an unquoted `at: 7:30:00` into the integer
`27000`, because PyYAML's integer resolver accepts base-60 digits separated by colons.
Note the precision: `7:30:00` resolves; `07:30:00` does not, because the sexagesimal
pattern requires a leading digit of 1 to 9. And bare `y` and `n` are NOT booleans in
PyYAML, whatever the YAML 1.1 spec says, so this app does not claim they are.

## What runs in the browser, free

- **A YAML reader** written against Home Assistant's loader. Every node carries its line
  number. Block and flow collections, block scalars with chomping, multi-document files,
  the `!secret` and `!include` tags, duplicate keys, tab indentation, and PyYAML's exact
  implicit resolvers for null, bool, int (binary, octal, decimal, hex, sexagesimal) and
  float. Verified structurally identical to `yaml.safe_load` on the bundled examples.
- **Forty deterministic checks** in three severities. `blocker` means it will not load,
  or will provably never fire. `warn` means there is a realistic state of the world in
  which it does the wrong thing. `note` means it works and there is a better way.
  Covered: the YAML 1.1 scalar traps above; the 2024 schema renames (`trigger:` to
  `triggers:`, `platform:` to `trigger:`, `service:` to `action:`, `service_template`,
  `data_template`); missing or empty trigger and action blocks; missing `id` and `alias`;
  run-mode problems, including a `delay` or an unbounded wait under `mode: single`;
  `numeric_state` with no bound; a state trigger with `to` equal to `from`; `for:` without
  `to:`; dense `time_pattern` schedules; webhook defaults changed in 2024.4; zone triggers
  with no event; `wait_for_trigger` and `wait_template` with no timeout; `choose` with no
  default; a `while` loop with no pause; a `notify` call with no message; brightness out
  of range; invalid entity ids and unknown domains; `device_id` targets; `entity_id` under
  `data:` instead of `target:`; unbalanced Jinja; `states.domain.object.state` dotted
  access; `| int` with no default; `is_state()` with one argument; self-trigger loops; and
  credentials written inline.
- **A dry-run simulator.** Set the state of every entity the automation names, plus the
  time, the sun and the weekday, and see which trigger fires, which condition blocks, and
  what the actions would do. Actions change the simulated world as they run, so
  `light.turn_on` really does set that light to `on` for every later step. The clock
  advances through each `delay`. `choose` takes the branch it would really take.
- **A Jinja subset evaluator.** `states()`, `is_state()`, `is_state_attr()`,
  `state_attr()`, `has_value()`, `now()`, `int`, `float`, `round`, `default`, comparisons,
  boolean operators, inline conditionals, and Jinja tests. It is a recursive-descent
  parser: there is no `eval`, no `Function` constructor, and no string execution anywhere
  in this app. Anything it cannot evaluate - a `{% %}` block, a filter it does not
  implement, a device or zone trigger - is listed as NOT EVALUATED rather than assumed
  true.

## The four lanes

Every lane takes the same pasted automation and sends the same digest, facts and dry-run
transcript. They differ only in what they are asked to produce.

| `task` | What it returns | Verdicts |
| --- | --- | --- |
| `review` | Trigger by trigger, condition by condition, action by action, plus what the run mode does to re-triggers, plus loop risk | `safe-to-reload`, `reload-with-changes`, `do-not-reload` |
| `explain` | Plain English for someone who did not write it, and the circumstances where it does something surprising | `clear`, `has-surprises`, `unpredictable` |
| `harden` | The whole automation rewritten in the current schema with every YAML 1.1 trap quoted, plus a numbered change list | `already-modern`, `rewritten`, `rewritten-with-assumptions` |
| `card` | Lovelace card and view YAML for the entities this automation touches | `card-ready`, `card-with-placeholders`, `not-enough-entities` |

## What the page checks about the model's answer

Three things, all in the browser, all free:

1. **Reconciliation.** Every deterministic finding is sent as a fact the reply must
   address, and the result panel shows one row per finding with what the reply said about
   it. A finding the reply skipped is shown as not addressed. The browser read your
   actual bytes; where they disagree, the browser wins.
2. **Line audit.** Any line number the reply cites that is outside the document is named
   as fabricated.
3. **Rewrite audit.** The `harden` lane's YAML is re-read by the same reader and rule
   engine before it is displayed, so a rewrite that reintroduced the very bug the lane
   exists to remove is named rather than handed over as an improvement. The `card` lane's
   entities are checked back against the entities the document actually names.

## Privacy

The document is read in your browser. When you run a lane, what is sent is a
line-numbered digest with credential-shaped values already replaced - a value under a key
named like a token or password, a webhook URL, or a long opaque string. The dry-run
transcript is sent with it. Nothing else. This app never connects to a Home Assistant
instance, has no credentials for one, and cannot change anything in your house. Every
artifact it produces is something you copy and paste yourself, after reading it.

A credential found in a paste is reported before anything else, because by the time it is
in the box it is also in the clipboard and in every backup since. Redaction before
sending does not un-disclose it; rotate it.

## API

`https://api.skillsafe.ai/v1/app-api` - see https://hass-bench.skillsafe.ai/api.html for
the full tutorial in cURL, Python, JavaScript, Go, Java, Ruby, PHP and C#. The run body
is the input object itself: `task`, `automation_digest` (or `raw_yaml` plus `parse_error`),
`facts`, `dry_run`, `context`, `ha_version`. There is no `X-App-Slug` header, and wrapping
the body in an `input` key returns 200 while hiding `task` from the model.

## Sources

Derived from four agent skills, and credited to all of them:

- `@bradsjm/home-assistant-automation-scripts` - automations, scripts and blueprints:
  triggers, conditions, actions, run modes, troubleshooting.
- `@homeassistant-ai/home-assistant-best-practices`
- `@claude-office-skills/home-assistant-automation`
- `@bradsjm/home-assistant-dashboards-cards` - Lovelace views, cards and resources.

An independent derived work. Not affiliated with those skills' authors, with the Home
Assistant project, with Nabu Casa, or with any integration named on the page.