Hugging Face's logo

autotrust
/
JEV-9B

Text Classification
Transformers
Safetensors
English
qwen3_5_text
text-generation
system-one
system-two
blocks-of-experts
typed-decisions
decision-model
calibrated-probabilities
knowledge-distillation
jev
noul
choice
score
lora
qwen3_5
dual-head
vllm
multimodal
vision
computer-use
robotics
Eval Results (legacy)
Model card Files
xet
 Community
2
cloudyu commited on
Commit
b63f651
·
verified ·
1 Parent(s): 4ab5dfb

JEV-9B can see: robot-arm and computer-use demos, vl/ (multimodal serving), image evaluations

Browse files
Files changed (24) hide show
  1. .gitattributes +1 -0
  2. README.md +66 -0
  3. reports/vl/agentrewardbench.json +7 -0
  4. reports/vl/computer_use.json +4249 -0
  5. reports/vl/microlens.json +71 -0
  6. reports/vl/mmrb2.json +26 -0
  7. reports/vl/quickdraw.json +8 -0
  8. reports/vl/robot_arm.json +1464 -0
  9. reports/vl/synthetic_images.json +6 -0
  10. reports/vl/text_parity.json +12 -0
  11. reports/vl/vl_rewardbench.json +14 -0
  12. videos/computer_use_mail.mp4 +0 -0
  13. videos/computer_use_settings.mp4 +0 -0
  14. videos/computer_use_shop.mp4 +3 -0
  15. videos/robot_arm_pick_place.mp4 +0 -0
  16. vl/adapter_vllm/adapter_config.json +58 -0
  17. vl/adapter_vllm/adapter_model.safetensors +3 -0
  18. vl/adapter_vllm/decision_head.json +99 -0
  19. vl/calibration.json +44 -0
  20. vl/demos/computer_use.py +178 -0
  21. vl/demos/robot_arm.py +323 -0
  22. vl/demos/webapp.html +119 -0
  23. vl/serve.sh +19 -0
  24. vl/serve_decide.py +444 -0
.gitattributes CHANGED
@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
 
 
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
37
+ videos/computer_use_shop.mp4 filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -25,6 +25,10 @@ tags:
25
  - text-generation
26
  - dual-head
27
  - vllm
 
 
 
 
28
  metrics:
29
  - kl
30
  - auroc
@@ -85,6 +89,64 @@ routed per request. The fastest model of the family: it answers a single decisio
85
  hosted API takes. Its successor, [autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B), is closer to Jev,
86
  transfers better to unseen tasks and has a stronger System 2.**
87
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
88
  ## At a glance
89
 
90
  **Integrated System 1 + System 2, first generation.** JEV-9B is AutoTrust's first open model to serve both modes of
@@ -699,6 +761,10 @@ adapter_vllm/ the same adapter for vLLM: backbone LoRA
699
  tokenizer.json · tokenizer_config.json · chat_template.jinja
700
  reports/ evaluation reports: test-set evaluation, bundle checks, HumanEval per-problem
701
  results, vLLM measurements, real-world tests, training-milestone reviews
 
 
 
 
702
  ```
703
 
704
  ## License and acknowledgements
 
25
  - text-generation
26
  - dual-head
27
  - vllm
28
+ - multimodal
29
+ - vision
30
+ - computer-use
31
+ - robotics
32
  metrics:
33
  - kl
34
  - auroc
 
89
  hosted API takes. Its successor, [autotrust/JEV-27B](https://huggingface.co/autotrust/JEV-27B), is closer to Jev,
90
  transfers better to unseen tasks and has a stronger System 2.**
91
 
92
+ ## New (3 October 2026): JEV-9B can see — robot arm and computer use
93
+
94
+ JEV-9B now takes images. Every step below is **one System 1 decision**: camera image or screenshot in, a probability
95
+ for every action out, in a single forward pass (about 0.2 s on one GPU). Run it with `bash vl/serve.sh` (see
96
+ [Images: quick start](#images-quick-start)).
97
+
98
+ **Robot arm: pick and place from camera images.** The arm sees a top camera image; at every step System 1 answers two
99
+ questions (is the target left or right of the gripper, above or below it), and the arm moves accordingly, halving its
100
+ step whenever an answer flips. It grasps the cube, carries it and drops it in the tray (MuJoCo simulation).
101
+
102
+ <video src="https://huggingface.co/autotrust/JEV-9B/resolve/main/videos/robot_arm_pick_place.mp4" controls autoplay loop muted playsinline width="100%"></video>
103
+
104
+ On 20 random scenes it completed the task 10 times; every cube it grasped ended in the tray, and every miss was a grasp
105
+ 3–5 cm off target. About 165 ms per decision. Asking it to choose one of 8 motor commands directly did not work: this
106
+ model is a fast visual judge, not an end-to-end controller.
107
+
108
+ **Computer use: screenshot → which element to click.** A real browser (headless Chromium). Every clickable element gets
109
+ a numbered box; System 1 picks the next click (or "the task is complete"), the browser clicks it, and the loop repeats.
110
+
111
+ <video src="https://huggingface.co/autotrust/JEV-9B/resolve/main/videos/computer_use_shop.mp4" controls autoplay loop muted playsinline width="100%"></video>
112
+
113
+ <video src="https://huggingface.co/autotrust/JEV-9B/resolve/main/videos/computer_use_settings.mp4" controls autoplay loop muted playsinline width="100%"></video>
114
+
115
+ <video src="https://huggingface.co/autotrust/JEV-9B/resolve/main/videos/computer_use_mail.mp4" controls autoplay loop muted playsinline width="100%"></video>
116
+
117
+ **95% of 60 random multi-step tasks completed** (shop, settings, mail; 3–7 clicks each), about 0.2 s per click. The
118
+ colour swatches and switches carry no text, so those clicks are decided from the screenshot alone. With the numbered
119
+ boxes only (no element text) it completed 37%. The failures skipped a step (the colour) and then checked out an empty cart.
120
+
121
+ Code for both demos: [`vl/demos/`](vl/demos). Image judging, briefly: VL-RewardBench 74.3%, AgentRewardBench AUROC 0.91,
122
+ zero-shot short-video recommendation from covers AUC 0.72 (details in [`reports/vl/`](reports/vl)).
123
+
124
+ ### Images: quick start
125
+
126
+ ```bash
127
+ hf download autotrust/JEV-9B --include "vl/*" --local-dir JEV-9B
128
+ bash JEV-9B/vl/serve.sh # downloads Qwen/Qwen3.5-9B (with its vision encoder) and serves both systems on :8000
129
+ ```
130
+
131
+ ```python
132
+ import base64, requests
133
+
134
+ def image(path):
135
+ return {"image": "data:image/png;base64," + base64.b64encode(open(path, "rb").read()).decode()}
136
+
137
+ r = requests.post("http://localhost:8000/v1/decide", json={
138
+ "kind": "choice",
139
+ "state": ["Top camera image:", image("scene.png"), "\nTask: put the red cube in the tray."],
140
+ "question": "Is the red cube to the left or to the right of the gripper?",
141
+ "options": ["left", "right"]}).json()
142
+ print(dict(zip(r["options"], r["probabilities"])))
143
+ ```
144
+
145
+ How it works: JEV-9B's language weights are bit-identical to Qwen3.5-9B's, so `vl/serve.sh` serves the unmodified
146
+ multimodal Qwen3.5-9B with JEV-9B's System 1 adapter (`vl/adapter_vllm`, the same weights with the layer names moved).
147
+ Text decisions match the text-only model (300 test decisions: largest probability difference 0.011). System 2 also reads
148
+ images. Keep `--max-num-seqs 8` (set in `serve.sh`); decisions over images are zero-shot.
149
+
150
  ## At a glance
151
 
152
  **Integrated System 1 + System 2, first generation.** JEV-9B is AutoTrust's first open model to serve both modes of
 
761
  tokenizer.json · tokenizer_config.json · chat_template.jinja
762
  reports/ evaluation reports: test-set evaluation, bundle checks, HumanEval per-problem
763
  results, vLLM measurements, real-world tests, training-milestone reviews
764
+ vl/ vision: serve.sh (multimodal Qwen3.5-9B + System 1), serve_decide.py (POST /v1/decide),
765
+ adapter_vllm/ (layer names for the multimodal model), calibration.json, demos/
766
+ videos/ robot-arm and computer-use demo videos
767
+ reports/vl/ image evaluations and demo results
768
  ```
769
 
770
  ## License and acknowledgements
reports/vl/agentrewardbench.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "n": 1302,
3
+ "auroc": 0.9079788939829442,
4
+ "precision_at_0.5": 0.7507598784194529,
5
+ "recall_at_0.5": 0.6938202247191011,
6
+ "positive_rate": 0.27342549923195086
7
+ }
reports/vl/computer_use.json ADDED
@@ -0,0 +1,4249 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "summary": {
3
+ "marks/mail": {
4
+ "episodes": 20,
5
+ "success": 0.15,
6
+ "mean_steps": 3.45,
7
+ "median_ms": 448.2111930847168
8
+ },
9
+ "marks/shop": {
10
+ "episodes": 20,
11
+ "success": 0.0,
12
+ "mean_steps": 5.75,
13
+ "median_ms": 436.57422065734863
14
+ },
15
+ "marks/settings": {
16
+ "episodes": 20,
17
+ "success": 0.95,
18
+ "mean_steps": 4.5,
19
+ "median_ms": 457.08513259887695
20
+ },
21
+ "marks/all": {
22
+ "episodes": 60,
23
+ "success": 0.36666666666666664,
24
+ "mean_steps": 4.566666666666666,
25
+ "median_ms": 441.36738777160645
26
+ },
27
+ "marks+text/mail": {
28
+ "episodes": 20,
29
+ "success": 1.0,
30
+ "mean_steps": 3.0,
31
+ "median_ms": 420.0015068054199
32
+ },
33
+ "marks+text/shop": {
34
+ "episodes": 20,
35
+ "success": 0.85,
36
+ "mean_steps": 6.85,
37
+ "median_ms": 356.5859794616699
38
+ },
39
+ "marks+text/settings": {
40
+ "episodes": 20,
41
+ "success": 1.0,
42
+ "mean_steps": 4.0,
43
+ "median_ms": 335.61229705810547
44
+ },
45
+ "marks+text/all": {
46
+ "episodes": 60,
47
+ "success": 0.95,
48
+ "mean_steps": 4.616666666666666,
49
+ "median_ms": 354.40516471862793
50
+ }
51
+ },
52
+ "episodes": [
53
+ {
54
+ "app": "mail",
55
+ "seed": 1,
56
+ "variant": "marks",
57
+ "task": "Archive the email from Grace Lee about the server outage.",
58
+ "success": false,
59
+ "goal_met": false,
60
+ "stopped": true,
61
+ "steps": 3,
62
+ "trace": [
63
+ {
64
+ "options": 10,
65
+ "choice": "click element 8",
66
+ "p": 0.9982746437151456
67
+ },
68
+ {
69
+ "options": 9,
70
+ "choice": "click element 6",
71
+ "p": 0.8334909382092105
72
+ },
73
+ {
74
+ "options": 9,
75
+ "choice": "the task is complete",
76
+ "p": 0.5058816504214596
77
+ }
78
+ ]
79
+ },
80
+ {
81
+ "app": "mail",
82
+ "seed": 7,
83
+ "variant": "marks",
84
+ "task": "Archive the email from Farid Khan about the server outage.",
85
+ "success": false,
86
+ "goal_met": false,
87
+ "stopped": true,
88
+ "steps": 2,
89
+ "trace": [
90
+ {
91
+ "options": 10,
92
+ "choice": "click element 6",
93
+ "p": 0.9647011684460824
94
+ },
95
+ {
96
+ "options": 9,
97
+ "choice": "the task is complete",
98
+ "p": 0.7792887183749148
99
+ }
100
+ ]
101
+ },
102
+ {
103
+ "app": "mail",
104
+ "seed": 13,
105
+ "variant": "marks",
106
+ "task": "Archive the email from Grace Lee about the design review.",
107
+ "success": true,
108
+ "goal_met": true,
109
+ "stopped": true,
110
+ "steps": 3,
111
+ "trace": [
112
+ {
113
+ "options": 10,
114
+ "choice": "click element 9",
115
+ "p": 0.9937341264133329
116
+ },
117
+ {
118
+ "options": 9,
119
+ "choice": "click element 6",
120
+ "p": 0.9863845529692548
121
+ },
122
+ {
123
+ "options": 9,
124
+ "choice": "the task is complete",
125
+ "p": 0.44492366517381127
126
+ }
127
+ ]
128
+ },
129
+ {
130
+ "app": "mail",
131
+ "seed": 19,
132
+ "variant": "marks",
133
+ "task": "Archive the email from Carla Diaz about the design review.",
134
+ "success": false,
135
+ "goal_met": false,
136
+ "stopped": true,
137
+ "steps": 6,
138
+ "trace": [
139
+ {
140
+ "options": 10,
141
+ "choice": "click element 5",
142
+ "p": 0.9807737343210758
143
+ },
144
+ {
145
+ "options": 9,
146
+ "choice": "click element 6",
147
+ "p": 0.8174530530135385
148
+ },
149
+ {
150
+ "options": 9,
151
+ "choice": "click element 5",
152
+ "p": 0.5244242389788385
153
+ },
154
+ {
155
+ "options": 9,
156
+ "choice": "click element 6",
157
+ "p": 0.31159580177347423
158
+ },
159
+ {
160
+ "options": 8,
161
+ "choice": "click element 5",
162
+ "p": 0.6709079633984095
163
+ },
164
+ {
165
+ "options": 9,
166
+ "choice": "the task is complete",
167
+ "p": 0.4141379128556906
168
+ }
169
+ ]
170
+ },
171
+ {
172
+ "app": "shop",
173
+ "seed": 5,
174
+ "variant": "marks",
175
+ "task": "Buy a blue Backpack in size XL: add it to the cart and check out.",
176
+ "success": false,
177
+ "goal_met": false,
178
+ "stopped": true,
179
+ "steps": 6,
180
+ "trace": [
181
+ {
182
+ "options": 10,
183
+ "choice": "click element 8",
184
+ "p": 0.9723648021740522
185
+ },
186
+ {
187
+ "options": 15,
188
+ "choice": "click element 6",
189
+ "p": 0.8600005236688053
190
+ },
191
+ {
192
+ "options": 15,
193
+ "choice": "click element 13",
194
+ "p": 0.9764484781782136
195
+ },
196
+ {
197
+ "options": 15,
198
+ "choice": "click element 14",
199
+ "p": 0.9973305801715106
200
+ },
201
+ {
202
+ "options": 15,
203
+ "choice": "click element 4",
204
+ "p": 0.583533532377031
205
+ },
206
+ {
207
+ "options": 6,
208
+ "choice": "the task is complete",
209
+ "p": 0.9714770425759613
210
+ }
211
+ ]
212
+ },
213
+ {
214
+ "app": "shop",
215
+ "seed": 11,
216
+ "variant": "marks",
217
+ "task": "Buy a green Hoodie in size M: add it to the cart and check out.",
218
+ "success": false,
219
+ "goal_met": false,
220
+ "stopped": true,
221
+ "steps": 5,
222
+ "trace": [
223
+ {
224
+ "options": 10,
225
+ "choice": "click element 4",
226
+ "p": 0.8722076667590815
227
+ },
228
+ {
229
+ "options": 15,
230
+ "choice": "click element 7",
231
+ "p": 0.9549730511314003
232
+ },
233
+ {
234
+ "options": 15,
235
+ "choice": "click element 11",
236
+ "p": 0.899875124333359
237
+ },
238
+ {
239
+ "options": 15,
240
+ "choice": "click element 14",
241
+ "p": 0.9783519677258821
242
+ },
243
+ {
244
+ "options": 15,
245
+ "choice": "the task is complete",
246
+ "p": 0.26210784293130024
247
+ }
248
+ ]
249
+ },
250
+ {
251
+ "app": "shop",
252
+ "seed": 17,
253
+ "variant": "marks",
254
+ "task": "Buy a yellow Backpack in size S: add it to the cart and check out.",
255
+ "success": false,
256
+ "goal_met": false,
257
+ "stopped": true,
258
+ "steps": 5,
259
+ "trace": [
260
+ {
261
+ "options": 10,
262
+ "choice": "click element 4",
263
+ "p": 0.8748457149745421
264
+ },
265
+ {
266
+ "options": 15,
267
+ "choice": "click element 9",
268
+ "p": 0.9919989012545333
269
+ },
270
+ {
271
+ "options": 15,
272
+ "choice": "click element 10",
273
+ "p": 0.6902823083403318
274
+ },
275
+ {
276
+ "options": 15,
277
+ "choice": "click element 14",
278
+ "p": 0.9802913310756076
279
+ },
280
+ {
281
+ "options": 15,
282
+ "choice": "the task is complete",
283
+ "p": 0.2689643969993545
284
+ }
285
+ ]
286
+ },
287
+ {
288
+ "app": "settings",
289
+ "seed": 3,
290
+ "variant": "marks",
291
+ "task": "Turn on \"Public profile\" in the Profile settings and save the change.",
292
+ "success": false,
293
+ "goal_met": false,
294
+ "stopped": false,
295
+ "steps": 10,
296
+ "trace": [
297
+ {
298
+ "options": 7,
299
+ "choice": "click element 6",
300
+ "p": 0.9244319010091888
301
+ },
302
+ {
303
+ "options": 11,
304
+ "choice": "click element 3",
305
+ "p": 0.21800954835263237
306
+ },
307
+ {
308
+ "options": 11,
309
+ "choice": "click element 9",
310
+ "p": 0.27040654496582883
311
+ },
312
+ {
313
+ "options": 11,
314
+ "choice": "click element 6",
315
+ "p": 0.34239870331676225
316
+ },
317
+ {
318
+ "options": 11,
319
+ "choice": "click element 6",
320
+ "p": 0.3316417510622889
321
+ },
322
+ {
323
+ "options": 11,
324
+ "choice": "click element 6",
325
+ "p": 0.3253367122452282
326
+ },
327
+ {
328
+ "options": 11,
329
+ "choice": "click element 6",
330
+ "p": 0.29282394222947133
331
+ },
332
+ {
333
+ "options": 11,
334
+ "choice": "click element 6",
335
+ "p": 0.2897465697254864
336
+ },
337
+ {
338
+ "options": 11,
339
+ "choice": "click element 3",
340
+ "p": 0.3013946459625229
341
+ },
342
+ {
343
+ "options": 11,
344
+ "choice": "click element 6",
345
+ "p": 0.27051882083252277
346
+ }
347
+ ]
348
+ },
349
+ {
350
+ "app": "settings",
351
+ "seed": 9,
352
+ "variant": "marks",
353
+ "task": "Turn on \"Show online status\" in the Profile settings and save the change.",
354
+ "success": true,
355
+ "goal_met": true,
356
+ "stopped": true,
357
+ "steps": 5,
358
+ "trace": [
359
+ {
360
+ "options": 7,
361
+ "choice": "click element 5",
362
+ "p": 0.9670326405483354
363
+ },
364
+ {
365
+ "options": 11,
366
+ "choice": "click element 4",
367
+ "p": 0.6468518133087894
368
+ },
369
+ {
370
+ "options": 11,
371
+ "choice": "click element 7",
372
+ "p": 0.9933768305569631
373
+ },
374
+ {
375
+ "options": 11,
376
+ "choice": "click element 9",
377
+ "p": 0.9699413010725476
378
+ },
379
+ {
380
+ "options": 11,
381
+ "choice": "the task is complete",
382
+ "p": 0.9097613437721809
383
+ }
384
+ ]
385
+ },
386
+ {
387
+ "app": "settings",
388
+ "seed": 15,
389
+ "variant": "marks",
390
+ "task": "Turn on \"Login alerts\" in the Security settings and save the change.",
391
+ "success": true,
392
+ "goal_met": true,
393
+ "stopped": true,
394
+ "steps": 4,
395
+ "trace": [
396
+ {
397
+ "options": 7,
398
+ "choice": "click element 6",
399
+ "p": 0.996253790857984
400
+ },
401
+ {
402
+ "options": 11,
403
+ "choice": "click element 8",
404
+ "p": 0.99995244088394
405
+ },
406
+ {
407
+ "options": 11,
408
+ "choice": "click element 9",
409
+ "p": 0.9998217960544984
410
+ },
411
+ {
412
+ "options": 11,
413
+ "choice": "the task is complete",
414
+ "p": 0.9865564735196983
415
+ }
416
+ ]
417
+ },
418
+ {
419
+ "app": "mail",
420
+ "seed": 1,
421
+ "variant": "marks+text",
422
+ "task": "Archive the email from Grace Lee about the server outage.",
423
+ "success": true,
424
+ "goal_met": true,
425
+ "stopped": true,
426
+ "steps": 3,
427
+ "trace": [
428
+ {
429
+ "options": 10,
430
+ "choice": "click element 7: Grace Lee server outage (list row)",
431
+ "p": 0.9997272732546577
432
+ },
433
+ {
434
+ "options": 9,
435
+ "choice": "click element 6: Archive (button)",
436
+ "p": 0.9949619613442648
437
+ },
438
+ {
439
+ "options": 9,
440
+ "choice": "the task is complete",
441
+ "p": 0.9737861919434078
442
+ }
443
+ ]
444
+ },
445
+ {
446
+ "app": "mail",
447
+ "seed": 7,
448
+ "variant": "marks+text",
449
+ "task": "Archive the email from Farid Khan about the server outage.",
450
+ "success": true,
451
+ "goal_met": true,
452
+ "stopped": true,
453
+ "steps": 3,
454
+ "trace": [
455
+ {
456
+ "options": 10,
457
+ "choice": "click element 5: Farid Khan server outage (list row)",
458
+ "p": 0.99891955649885
459
+ },
460
+ {
461
+ "options": 9,
462
+ "choice": "click element 6: Archive (button)",
463
+ "p": 0.9961048712722775
464
+ },
465
+ {
466
+ "options": 9,
467
+ "choice": "the task is complete",
468
+ "p": 0.9948701459532536
469
+ }
470
+ ]
471
+ },
472
+ {
473
+ "app": "mail",
474
+ "seed": 13,
475
+ "variant": "marks+text",
476
+ "task": "Archive the email from Grace Lee about the design review.",
477
+ "success": true,
478
+ "goal_met": true,
479
+ "stopped": true,
480
+ "steps": 3,
481
+ "trace": [
482
+ {
483
+ "options": 10,
484
+ "choice": "click element 9: Grace Lee design review (list row)",
485
+ "p": 0.9999329216037259
486
+ },
487
+ {
488
+ "options": 9,
489
+ "choice": "click element 6: Archive (button)",
490
+ "p": 0.9999737385115206
491
+ },
492
+ {
493
+ "options": 9,
494
+ "choice": "the task is complete",
495
+ "p": 0.992147055643575
496
+ }
497
+ ]
498
+ },
499
+ {
500
+ "app": "mail",
501
+ "seed": 19,
502
+ "variant": "marks+text",
503
+ "task": "Archive the email from Carla Diaz about the design review.",
504
+ "success": true,
505
+ "goal_met": true,
506
+ "stopped": true,
507
+ "steps": 3,
508
+ "trace": [
509
+ {
510
+ "options": 10,
511
+ "choice": "click element 4: Carla Diaz design review (list row)",
512
+ "p": 0.9950059236871672
513
+ },
514
+ {
515
+ "options": 9,
516
+ "choice": "click element 6: Archive (button)",
517
+ "p": 0.9964700166912593
518
+ },
519
+ {
520
+ "options": 9,
521
+ "choice": "the task is complete",
522
+ "p": 0.9784770032552219
523
+ }
524
+ ]
525
+ },
526
+ {
527
+ "app": "shop",
528
+ "seed": 5,
529
+ "variant": "marks+text",
530
+ "task": "Buy a blue Backpack in size XL: add it to the cart and check out.",
531
+ "success": true,
532
+ "goal_met": true,
533
+ "stopped": true,
534
+ "steps": 7,
535
+ "trace": [
536
+ {
537
+ "options": 10,
538
+ "choice": "click element 8: Backpack $47 (product card)",
539
+ "p": 0.993364117705504
540
+ },
541
+ {
542
+ "options": 15,
543
+ "choice": "click element 6: colour swatch",
544
+ "p": 0.9274856450395298
545
+ },
546
+ {
547
+ "options": 15,
548
+ "choice": "click element 13: XL (button)",
549
+ "p": 0.9928069873869816
550
+ },
551
+ {
552
+ "options": 15,
553
+ "choice": "click element 14: Add to cart (button)",
554
+ "p": 0.9989047164598033
555
+ },
556
+ {
557
+ "options": 15,
558
+ "choice": "click element 4: Cart (1) (button)",
559
+ "p": 0.9219735598194037
560
+ },
561
+ {
562
+ "options": 6,
563
+ "choice": "click element 4: Checkout (button)",
564
+ "p": 0.9573855700355163
565
+ },
566
+ {
567
+ "options": 6,
568
+ "choice": "the task is complete",
569
+ "p": 0.9922097493907391
570
+ }
571
+ ]
572
+ },
573
+ {
574
+ "app": "shop",
575
+ "seed": 11,
576
+ "variant": "marks+text",
577
+ "task": "Buy a green Hoodie in size M: add it to the cart and check out.",
578
+ "success": true,
579
+ "goal_met": true,
580
+ "stopped": true,
581
+ "steps": 7,
582
+ "trace": [
583
+ {
584
+ "options": 10,
585
+ "choice": "click element 4: Hoodie $19 (product card)",
586
+ "p": 0.9412609201093222
587
+ },
588
+ {
589
+ "options": 15,
590
+ "choice": "click element 7: colour swatch",
591
+ "p": 0.9564180663469437
592
+ },
593
+ {
594
+ "options": 15,
595
+ "choice": "click element 11: M (button)",
596
+ "p": 0.986536596936008
597
+ },
598
+ {
599
+ "options": 15,
600
+ "choice": "click element 14: Add to cart (button)",
601
+ "p": 0.9985445443898117
602
+ },
603
+ {
604
+ "options": 15,
605
+ "choice": "click element 4: Cart (1) (button)",
606
+ "p": 0.8282340607456052
607
+ },
608
+ {
609
+ "options": 6,
610
+ "choice": "click element 4: Checkout (button)",
611
+ "p": 0.9654997367268956
612
+ },
613
+ {
614
+ "options": 6,
615
+ "choice": "the task is complete",
616
+ "p": 0.9842322022806581
617
+ }
618
+ ]
619
+ },
620
+ {
621
+ "app": "shop",
622
+ "seed": 17,
623
+ "variant": "marks+text",
624
+ "task": "Buy a yellow Backpack in size S: add it to the cart and check out.",
625
+ "success": true,
626
+ "goal_met": true,
627
+ "stopped": true,
628
+ "steps": 7,
629
+ "trace": [
630
+ {
631
+ "options": 10,
632
+ "choice": "click element 4: Backpack $19 (product card)",
633
+ "p": 0.940359339582156
634
+ },
635
+ {
636
+ "options": 15,
637
+ "choice": "click element 9: colour swatch",
638
+ "p": 0.9858881742243698
639
+ },
640
+ {
641
+ "options": 15,
642
+ "choice": "click element 10: S (button)",
643
+ "p": 0.8551272500749205
644
+ },
645
+ {
646
+ "options": 15,
647
+ "choice": "click element 14: Add to cart (button)",
648
+ "p": 0.9984293372563396
649
+ },
650
+ {
651
+ "options": 15,
652
+ "choice": "click element 4: Cart (1) (button)",
653
+ "p": 0.8050696131921257
654
+ },
655
+ {
656
+ "options": 6,
657
+ "choice": "click element 4: Checkout (button)",
658
+ "p": 0.9021917776190918
659
+ },
660
+ {
661
+ "options": 6,
662
+ "choice": "the task is complete",
663
+ "p": 0.9881904436979773
664
+ }
665
+ ]
666
+ },
667
+ {
668
+ "app": "settings",
669
+ "seed": 3,
670
+ "variant": "marks+text",
671
+ "task": "Turn on \"Public profile\" in the Profile settings and save the change.",
672
+ "success": true,
673
+ "goal_met": true,
674
+ "stopped": true,
675
+ "steps": 4,
676
+ "trace": [
677
+ {
678
+ "options": 7,
679
+ "choice": "click element 5: Profile (button)",
680
+ "p": 0.9959615170011822
681
+ },
682
+ {
683
+ "options": 11,
684
+ "choice": "click element 8: switch",
685
+ "p": 0.9999599811214911
686
+ },
687
+ {
688
+ "options": 11,
689
+ "choice": "click element 9: Save changes (button)",
690
+ "p": 0.9994108816211869
691
+ },
692
+ {
693
+ "options": 11,
694
+ "choice": "the task is complete",
695
+ "p": 0.9936511423877045
696
+ }
697
+ ]
698
+ },
699
+ {
700
+ "app": "settings",
701
+ "seed": 9,
702
+ "variant": "marks+text",
703
+ "task": "Turn on \"Show online status\" in the Profile settings and save the change.",
704
+ "success": true,
705
+ "goal_met": true,
706
+ "stopped": true,
707
+ "steps": 4,
708
+ "trace": [
709
+ {
710
+ "options": 7,
711
+ "choice": "click element 4: Profile (button)",
712
+ "p": 0.9988477194352602
713
+ },
714
+ {
715
+ "options": 11,
716
+ "choice": "click element 7: switch",
717
+ "p": 0.9985245328592022
718
+ },
719
+ {
720
+ "options": 11,
721
+ "choice": "click element 9: Save changes (button)",
722
+ "p": 0.9992239296747358
723
+ },
724
+ {
725
+ "options": 11,
726
+ "choice": "the task is complete",
727
+ "p": 0.9941867260593812
728
+ }
729
+ ]
730
+ },
731
+ {
732
+ "app": "settings",
733
+ "seed": 15,
734
+ "variant": "marks+text",
735
+ "task": "Turn on \"Login alerts\" in the Security settings and save the change.",
736
+ "success": true,
737
+ "goal_met": true,
738
+ "stopped": true,
739
+ "steps": 4,
740
+ "trace": [
741
+ {
742
+ "options": 7,
743
+ "choice": "click element 6: Security (button)",
744
+ "p": 0.9999235275879494
745
+ },
746
+ {
747
+ "options": 11,
748
+ "choice": "click element 8: switch",
749
+ "p": 0.9999808420275468
750
+ },
751
+ {
752
+ "options": 11,
753
+ "choice": "click element 9: Save changes (button)",
754
+ "p": 0.9997413447716699
755
+ },
756
+ {
757
+ "options": 11,
758
+ "choice": "the task is complete",
759
+ "p": 0.9963593628898741
760
+ }
761
+ ]
762
+ },
763
+ {
764
+ "app": "mail",
765
+ "seed": 2,
766
+ "variant": "marks",
767
+ "task": "Archive the email from Carla Diaz about the design review.",
768
+ "success": false,
769
+ "goal_met": false,
770
+ "stopped": true,
771
+ "steps": 2,
772
+ "trace": [
773
+ {
774
+ "options": 10,
775
+ "choice": "click element 6",
776
+ "p": 0.992892747318574
777
+ },
778
+ {
779
+ "options": 9,
780
+ "choice": "the task is complete",
781
+ "p": 0.9394490421410343
782
+ }
783
+ ]
784
+ },
785
+ {
786
+ "app": "mail",
787
+ "seed": 8,
788
+ "variant": "marks",
789
+ "task": "Archive the email from David Kim about the team offsite.",
790
+ "success": false,
791
+ "goal_met": false,
792
+ "stopped": true,
793
+ "steps": 4,
794
+ "trace": [
795
+ {
796
+ "options": 10,
797
+ "choice": "click element 5",
798
+ "p": 0.9839992752867965
799
+ },
800
+ {
801
+ "options": 9,
802
+ "choice": "click element 6",
803
+ "p": 0.8287444053272831
804
+ },
805
+ {
806
+ "options": 9,
807
+ "choice": "click element 5",
808
+ "p": 0.5330357886108769
809
+ },
810
+ {
811
+ "options": 9,
812
+ "choice": "the task is complete",
813
+ "p": 0.3273539011637661
814
+ }
815
+ ]
816
+ },
817
+ {
818
+ "app": "mail",
819
+ "seed": 14,
820
+ "variant": "marks",
821
+ "task": "Archive the email from Farid Khan about the design review.",
822
+ "success": false,
823
+ "goal_met": false,
824
+ "stopped": true,
825
+ "steps": 3,
826
+ "trace": [
827
+ {
828
+ "options": 10,
829
+ "choice": "click element 8",
830
+ "p": 0.9967033674828369
831
+ },
832
+ {
833
+ "options": 9,
834
+ "choice": "click element 6",
835
+ "p": 0.7805851948452558
836
+ },
837
+ {
838
+ "options": 9,
839
+ "choice": "the task is complete",
840
+ "p": 0.4830101845743075
841
+ }
842
+ ]
843
+ },
844
+ {
845
+ "app": "mail",
846
+ "seed": 20,
847
+ "variant": "marks",
848
+ "task": "Archive the email from David Kim about the quarterly report.",
849
+ "success": false,
850
+ "goal_met": false,
851
+ "stopped": true,
852
+ "steps": 3,
853
+ "trace": [
854
+ {
855
+ "options": 10,
856
+ "choice": "click element 8",
857
+ "p": 0.9987644959897256
858
+ },
859
+ {
860
+ "options": 9,
861
+ "choice": "click element 6",
862
+ "p": 0.8527813615448195
863
+ },
864
+ {
865
+ "options": 9,
866
+ "choice": "the task is complete",
867
+ "p": 0.5125817489314975
868
+ }
869
+ ]
870
+ },
871
+ {
872
+ "app": "shop",
873
+ "seed": 6,
874
+ "variant": "marks",
875
+ "task": "Buy a yellow Hoodie in size L: add it to the cart and check out.",
876
+ "success": false,
877
+ "goal_met": false,
878
+ "stopped": true,
879
+ "steps": 6,
880
+ "trace": [
881
+ {
882
+ "options": 10,
883
+ "choice": "click element 7",
884
+ "p": 0.9285634084521504
885
+ },
886
+ {
887
+ "options": 15,
888
+ "choice": "click element 9",
889
+ "p": 0.7314717723970512
890
+ },
891
+ {
892
+ "options": 15,
893
+ "choice": "click element 12",
894
+ "p": 0.9084396079269625
895
+ },
896
+ {
897
+ "options": 15,
898
+ "choice": "click element 14",
899
+ "p": 0.992894054982253
900
+ },
901
+ {
902
+ "options": 15,
903
+ "choice": "click element 4",
904
+ "p": 0.5219137971277054
905
+ },
906
+ {
907
+ "options": 6,
908
+ "choice": "the task is complete",
909
+ "p": 0.9685345805656184
910
+ }
911
+ ]
912
+ },
913
+ {
914
+ "app": "shop",
915
+ "seed": 12,
916
+ "variant": "marks",
917
+ "task": "Buy a yellow T-shirt in size S: add it to the cart and check out.",
918
+ "success": false,
919
+ "goal_met": false,
920
+ "stopped": true,
921
+ "steps": 5,
922
+ "trace": [
923
+ {
924
+ "options": 10,
925
+ "choice": "click element 9",
926
+ "p": 0.9023016621302632
927
+ },
928
+ {
929
+ "options": 15,
930
+ "choice": "click element 10",
931
+ "p": 0.6798205530212345
932
+ },
933
+ {
934
+ "options": 15,
935
+ "choice": "click element 14",
936
+ "p": 0.982139760810323
937
+ },
938
+ {
939
+ "options": 15,
940
+ "choice": "click element 4",
941
+ "p": 0.43474362963224583
942
+ },
943
+ {
944
+ "options": 6,
945
+ "choice": "the task is complete",
946
+ "p": 0.6348496367171638
947
+ }
948
+ ]
949
+ },
950
+ {
951
+ "app": "shop",
952
+ "seed": 18,
953
+ "variant": "marks",
954
+ "task": "Buy a yellow Hoodie in size M: add it to the cart and check out.",
955
+ "success": false,
956
+ "goal_met": false,
957
+ "stopped": true,
958
+ "steps": 6,
959
+ "trace": [
960
+ {
961
+ "options": 10,
962
+ "choice": "click element 4",
963
+ "p": 0.8831294091331314
964
+ },
965
+ {
966
+ "options": 15,
967
+ "choice": "click element 9",
968
+ "p": 0.9897705699477023
969
+ },
970
+ {
971
+ "options": 15,
972
+ "choice": "click element 11",
973
+ "p": 0.9485748263462652
974
+ },
975
+ {
976
+ "options": 15,
977
+ "choice": "click element 14",
978
+ "p": 0.980252965938635
979
+ },
980
+ {
981
+ "options": 15,
982
+ "choice": "click element 2",
983
+ "p": 0.3433746666724213
984
+ },
985
+ {
986
+ "options": 15,
987
+ "choice": "the task is complete",
988
+ "p": 0.2845906266826623
989
+ }
990
+ ]
991
+ },
992
+ {
993
+ "app": "settings",
994
+ "seed": 4,
995
+ "variant": "marks",
996
+ "task": "Turn on \"Dark mode\" in the Appearance settings and save the change.",
997
+ "success": true,
998
+ "goal_met": true,
999
+ "stopped": true,
1000
+ "steps": 4,
1001
+ "trace": [
1002
+ {
1003
+ "options": 7,
1004
+ "choice": "click element 5",
1005
+ "p": 0.7659056181153656
1006
+ },
1007
+ {
1008
+ "options": 11,
1009
+ "choice": "click element 7",
1010
+ "p": 0.9981562579322069
1011
+ },
1012
+ {
1013
+ "options": 11,
1014
+ "choice": "click element 9",
1015
+ "p": 0.9995431369873375
1016
+ },
1017
+ {
1018
+ "options": 11,
1019
+ "choice": "the task is complete",
1020
+ "p": 0.9829790685579896
1021
+ }
1022
+ ]
1023
+ },
1024
+ {
1025
+ "app": "settings",
1026
+ "seed": 10,
1027
+ "variant": "marks",
1028
+ "task": "Turn on \"Login alerts\" in the Security settings and save the change.",
1029
+ "success": true,
1030
+ "goal_met": true,
1031
+ "stopped": true,
1032
+ "steps": 4,
1033
+ "trace": [
1034
+ {
1035
+ "options": 7,
1036
+ "choice": "click element 5",
1037
+ "p": 0.8607466026492488
1038
+ },
1039
+ {
1040
+ "options": 11,
1041
+ "choice": "click element 8",
1042
+ "p": 0.9997785980513878
1043
+ },
1044
+ {
1045
+ "options": 11,
1046
+ "choice": "click element 9",
1047
+ "p": 0.9995801557152993
1048
+ },
1049
+ {
1050
+ "options": 11,
1051
+ "choice": "the task is complete",
1052
+ "p": 0.9777011584656636
1053
+ }
1054
+ ]
1055
+ },
1056
+ {
1057
+ "app": "settings",
1058
+ "seed": 16,
1059
+ "variant": "marks",
1060
+ "task": "Turn on \"Dark mode\" in the Appearance settings and save the change.",
1061
+ "success": true,
1062
+ "goal_met": true,
1063
+ "stopped": true,
1064
+ "steps": 4,
1065
+ "trace": [
1066
+ {
1067
+ "options": 7,
1068
+ "choice": "click element 3",
1069
+ "p": 0.9212433445285446
1070
+ },
1071
+ {
1072
+ "options": 11,
1073
+ "choice": "click element 7",
1074
+ "p": 0.9935464056890795
1075
+ },
1076
+ {
1077
+ "options": 11,
1078
+ "choice": "click element 9",
1079
+ "p": 0.9993049765619494
1080
+ },
1081
+ {
1082
+ "options": 11,
1083
+ "choice": "the task is complete",
1084
+ "p": 0.9591851744653948
1085
+ }
1086
+ ]
1087
+ },
1088
+ {
1089
+ "app": "mail",
1090
+ "seed": 2,
1091
+ "variant": "marks+text",
1092
+ "task": "Archive the email from Carla Diaz about the design review.",
1093
+ "success": true,
1094
+ "goal_met": true,
1095
+ "stopped": true,
1096
+ "steps": 3,
1097
+ "trace": [
1098
+ {
1099
+ "options": 10,
1100
+ "choice": "click element 5: Carla Diaz design review (list row)",
1101
+ "p": 0.9977086597928134
1102
+ },
1103
+ {
1104
+ "options": 9,
1105
+ "choice": "click element 6: Archive (button)",
1106
+ "p": 0.9903121014052187
1107
+ },
1108
+ {
1109
+ "options": 9,
1110
+ "choice": "the task is complete",
1111
+ "p": 0.9814345325268685
1112
+ }
1113
+ ]
1114
+ },
1115
+ {
1116
+ "app": "mail",
1117
+ "seed": 8,
1118
+ "variant": "marks+text",
1119
+ "task": "Archive the email from David Kim about the team offsite.",
1120
+ "success": true,
1121
+ "goal_met": true,
1122
+ "stopped": true,
1123
+ "steps": 3,
1124
+ "trace": [
1125
+ {
1126
+ "options": 10,
1127
+ "choice": "click element 4: David Kim team offsite (list row)",
1128
+ "p": 0.9894351748126062
1129
+ },
1130
+ {
1131
+ "options": 9,
1132
+ "choice": "click element 6: Archive (button)",
1133
+ "p": 0.9964010863647544
1134
+ },
1135
+ {
1136
+ "options": 9,
1137
+ "choice": "the task is complete",
1138
+ "p": 0.983771267828729
1139
+ }
1140
+ ]
1141
+ },
1142
+ {
1143
+ "app": "mail",
1144
+ "seed": 14,
1145
+ "variant": "marks+text",
1146
+ "task": "Archive the email from Farid Khan about the design review.",
1147
+ "success": true,
1148
+ "goal_met": true,
1149
+ "stopped": true,
1150
+ "steps": 3,
1151
+ "trace": [
1152
+ {
1153
+ "options": 10,
1154
+ "choice": "click element 7: Farid Khan design review (list row)",
1155
+ "p": 0.9993521420451742
1156
+ },
1157
+ {
1158
+ "options": 9,
1159
+ "choice": "click element 6: Archive (button)",
1160
+ "p": 0.9770269820561703
1161
+ },
1162
+ {
1163
+ "options": 9,
1164
+ "choice": "the task is complete",
1165
+ "p": 0.9777662221765737
1166
+ }
1167
+ ]
1168
+ },
1169
+ {
1170
+ "app": "mail",
1171
+ "seed": 20,
1172
+ "variant": "marks+text",
1173
+ "task": "Archive the email from David Kim about the quarterly report.",
1174
+ "success": true,
1175
+ "goal_met": true,
1176
+ "stopped": true,
1177
+ "steps": 3,
1178
+ "trace": [
1179
+ {
1180
+ "options": 10,
1181
+ "choice": "click element 7: David Kim quarterly report (list row)",
1182
+ "p": 0.9990144165005647
1183
+ },
1184
+ {
1185
+ "options": 9,
1186
+ "choice": "click element 6: Archive (button)",
1187
+ "p": 0.9926864103491031
1188
+ },
1189
+ {
1190
+ "options": 9,
1191
+ "choice": "the task is complete",
1192
+ "p": 0.9756077897569488
1193
+ }
1194
+ ]
1195
+ },
1196
+ {
1197
+ "app": "shop",
1198
+ "seed": 6,
1199
+ "variant": "marks+text",
1200
+ "task": "Buy a yellow Hoodie in size L: add it to the cart and check out.",
1201
+ "success": true,
1202
+ "goal_met": true,
1203
+ "stopped": true,
1204
+ "steps": 7,
1205
+ "trace": [
1206
+ {
1207
+ "options": 10,
1208
+ "choice": "click element 7: Hoodie $40 (product card)",
1209
+ "p": 0.9813036312355834
1210
+ },
1211
+ {
1212
+ "options": 15,
1213
+ "choice": "click element 9: colour swatch",
1214
+ "p": 0.9610590109418928
1215
+ },
1216
+ {
1217
+ "options": 15,
1218
+ "choice": "click element 12: L (button)",
1219
+ "p": 0.9815786175202621
1220
+ },
1221
+ {
1222
+ "options": 15,
1223
+ "choice": "click element 14: Add to cart (button)",
1224
+ "p": 0.9988241701268467
1225
+ },
1226
+ {
1227
+ "options": 15,
1228
+ "choice": "click element 4: Cart (1) (button)",
1229
+ "p": 0.8970628239849784
1230
+ },
1231
+ {
1232
+ "options": 6,
1233
+ "choice": "click element 4: Checkout (button)",
1234
+ "p": 0.8904910506224382
1235
+ },
1236
+ {
1237
+ "options": 6,
1238
+ "choice": "the task is complete",
1239
+ "p": 0.9872025732578935
1240
+ }
1241
+ ]
1242
+ },
1243
+ {
1244
+ "app": "shop",
1245
+ "seed": 12,
1246
+ "variant": "marks+text",
1247
+ "task": "Buy a yellow T-shirt in size S: add it to the cart and check out.",
1248
+ "success": false,
1249
+ "goal_met": false,
1250
+ "stopped": true,
1251
+ "steps": 6,
1252
+ "trace": [
1253
+ {
1254
+ "options": 10,
1255
+ "choice": "click element 9: T-shirt $54 (product card)",
1256
+ "p": 0.9804849887896055
1257
+ },
1258
+ {
1259
+ "options": 15,
1260
+ "choice": "click element 14: Add to cart (button)",
1261
+ "p": 0.5761758000316538
1262
+ },
1263
+ {
1264
+ "options": 15,
1265
+ "choice": "click element 10: S (button)",
1266
+ "p": 0.5022904409646454
1267
+ },
1268
+ {
1269
+ "options": 15,
1270
+ "choice": "click element 4: Cart (0) (button)",
1271
+ "p": 0.8642461308245414
1272
+ },
1273
+ {
1274
+ "options": 6,
1275
+ "choice": "click element 4: Checkout (button)",
1276
+ "p": 0.8316142885191892
1277
+ },
1278
+ {
1279
+ "options": 6,
1280
+ "choice": "the task is complete",
1281
+ "p": 0.7651186738172107
1282
+ }
1283
+ ]
1284
+ },
1285
+ {
1286
+ "app": "shop",
1287
+ "seed": 18,
1288
+ "variant": "marks+text",
1289
+ "task": "Buy a yellow Hoodie in size M: add it to the cart and check out.",
1290
+ "success": true,
1291
+ "goal_met": true,
1292
+ "stopped": true,
1293
+ "steps": 7,
1294
+ "trace": [
1295
+ {
1296
+ "options": 10,
1297
+ "choice": "click element 4: Hoodie $19 (product card)",
1298
+ "p": 0.9522250943022443
1299
+ },
1300
+ {
1301
+ "options": 15,
1302
+ "choice": "click element 9: colour swatch",
1303
+ "p": 0.9738524366972843
1304
+ },
1305
+ {
1306
+ "options": 15,
1307
+ "choice": "click element 11: M (button)",
1308
+ "p": 0.9863360416857281
1309
+ },
1310
+ {
1311
+ "options": 15,
1312
+ "choice": "click element 14: Add to cart (button)",
1313
+ "p": 0.9986619272427975
1314
+ },
1315
+ {
1316
+ "options": 15,
1317
+ "choice": "click element 4: Cart (1) (button)",
1318
+ "p": 0.8094502494360777
1319
+ },
1320
+ {
1321
+ "options": 6,
1322
+ "choice": "click element 4: Checkout (button)",
1323
+ "p": 0.9487938856798926
1324
+ },
1325
+ {
1326
+ "options": 6,
1327
+ "choice": "the task is complete",
1328
+ "p": 0.9828495924812901
1329
+ }
1330
+ ]
1331
+ },
1332
+ {
1333
+ "app": "settings",
1334
+ "seed": 4,
1335
+ "variant": "marks+text",
1336
+ "task": "Turn on \"Dark mode\" in the Appearance settings and save the change.",
1337
+ "success": true,
1338
+ "goal_met": true,
1339
+ "stopped": true,
1340
+ "steps": 4,
1341
+ "trace": [
1342
+ {
1343
+ "options": 7,
1344
+ "choice": "click element 5: Appearance (button)",
1345
+ "p": 0.9999708741560777
1346
+ },
1347
+ {
1348
+ "options": 11,
1349
+ "choice": "click element 7: switch",
1350
+ "p": 0.9980580764701288
1351
+ },
1352
+ {
1353
+ "options": 11,
1354
+ "choice": "click element 9: Save changes (button)",
1355
+ "p": 0.9999883319061353
1356
+ },
1357
+ {
1358
+ "options": 11,
1359
+ "choice": "the task is complete",
1360
+ "p": 0.9967894681511111
1361
+ }
1362
+ ]
1363
+ },
1364
+ {
1365
+ "app": "settings",
1366
+ "seed": 10,
1367
+ "variant": "marks+text",
1368
+ "task": "Turn on \"Login alerts\" in the Security settings and save the change.",
1369
+ "success": true,
1370
+ "goal_met": true,
1371
+ "stopped": true,
1372
+ "steps": 4,
1373
+ "trace": [
1374
+ {
1375
+ "options": 7,
1376
+ "choice": "click element 5: Security (button)",
1377
+ "p": 0.9995923520219391
1378
+ },
1379
+ {
1380
+ "options": 11,
1381
+ "choice": "click element 8: switch",
1382
+ "p": 0.999982835500853
1383
+ },
1384
+ {
1385
+ "options": 11,
1386
+ "choice": "click element 9: Save changes (button)",
1387
+ "p": 0.9995185658916087
1388
+ },
1389
+ {
1390
+ "options": 11,
1391
+ "choice": "the task is complete",
1392
+ "p": 0.9951004316276988
1393
+ }
1394
+ ]
1395
+ },
1396
+ {
1397
+ "app": "settings",
1398
+ "seed": 16,
1399
+ "variant": "marks+text",
1400
+ "task": "Turn on \"Dark mode\" in the Appearance settings and save the change.",
1401
+ "success": true,
1402
+ "goal_met": true,
1403
+ "stopped": true,
1404
+ "steps": 4,
1405
+ "trace": [
1406
+ {
1407
+ "options": 7,
1408
+ "choice": "click element 3: Appearance (button)",
1409
+ "p": 0.9999751438353036
1410
+ },
1411
+ {
1412
+ "options": 11,
1413
+ "choice": "click element 7: switch",
1414
+ "p": 0.9986287386223566
1415
+ },
1416
+ {
1417
+ "options": 11,
1418
+ "choice": "click element 9: Save changes (button)",
1419
+ "p": 0.9999953952381304
1420
+ },
1421
+ {
1422
+ "options": 11,
1423
+ "choice": "the task is complete",
1424
+ "p": 0.997078758999307
1425
+ }
1426
+ ]
1427
+ },
1428
+ {
1429
+ "app": "mail",
1430
+ "seed": 3,
1431
+ "variant": "marks",
1432
+ "task": "Archive the email from David Kim about the quarterly report.",
1433
+ "success": false,
1434
+ "goal_met": false,
1435
+ "stopped": true,
1436
+ "steps": 6,
1437
+ "trace": [
1438
+ {
1439
+ "options": 10,
1440
+ "choice": "click element 5",
1441
+ "p": 0.9838817435045476
1442
+ },
1443
+ {
1444
+ "options": 9,
1445
+ "choice": "click element 6",
1446
+ "p": 0.8455759183384586
1447
+ },
1448
+ {
1449
+ "options": 9,
1450
+ "choice": "click element 5",
1451
+ "p": 0.5875776523184338
1452
+ },
1453
+ {
1454
+ "options": 9,
1455
+ "choice": "click element 6",
1456
+ "p": 0.3086781194757333
1457
+ },
1458
+ {
1459
+ "options": 8,
1460
+ "choice": "click element 5",
1461
+ "p": 0.6766474044327385
1462
+ },
1463
+ {
1464
+ "options": 9,
1465
+ "choice": "the task is complete",
1466
+ "p": 0.3183251198465185
1467
+ }
1468
+ ]
1469
+ },
1470
+ {
1471
+ "app": "mail",
1472
+ "seed": 9,
1473
+ "variant": "marks",
1474
+ "task": "Archive the email from Bob Martin about the team offsite.",
1475
+ "success": true,
1476
+ "goal_met": true,
1477
+ "stopped": true,
1478
+ "steps": 3,
1479
+ "trace": [
1480
+ {
1481
+ "options": 10,
1482
+ "choice": "click element 9",
1483
+ "p": 0.997289240137233
1484
+ },
1485
+ {
1486
+ "options": 9,
1487
+ "choice": "click element 6",
1488
+ "p": 0.9837272087328917
1489
+ },
1490
+ {
1491
+ "options": 9,
1492
+ "choice": "the task is complete",
1493
+ "p": 0.4647874571488452
1494
+ }
1495
+ ]
1496
+ },
1497
+ {
1498
+ "app": "mail",
1499
+ "seed": 15,
1500
+ "variant": "marks",
1501
+ "task": "Archive the email from Emma Rossi about the invoice for March.",
1502
+ "success": false,
1503
+ "goal_met": false,
1504
+ "stopped": true,
1505
+ "steps": 2,
1506
+ "trace": [
1507
+ {
1508
+ "options": 10,
1509
+ "choice": "click element 6",
1510
+ "p": 0.9702812148517301
1511
+ },
1512
+ {
1513
+ "options": 9,
1514
+ "choice": "the task is complete",
1515
+ "p": 0.9403280793404984
1516
+ }
1517
+ ]
1518
+ },
1519
+ {
1520
+ "app": "shop",
1521
+ "seed": 1,
1522
+ "variant": "marks",
1523
+ "task": "Buy a yellow T-shirt in size S: add it to the cart and check out.",
1524
+ "success": false,
1525
+ "goal_met": false,
1526
+ "stopped": true,
1527
+ "steps": 6,
1528
+ "trace": [
1529
+ {
1530
+ "options": 10,
1531
+ "choice": "click element 7",
1532
+ "p": 0.9101713573009392
1533
+ },
1534
+ {
1535
+ "options": 15,
1536
+ "choice": "click element 9",
1537
+ "p": 0.9692643661636647
1538
+ },
1539
+ {
1540
+ "options": 15,
1541
+ "choice": "click element 10",
1542
+ "p": 0.8454289470261946
1543
+ },
1544
+ {
1545
+ "options": 15,
1546
+ "choice": "click element 14",
1547
+ "p": 0.9888392092463819
1548
+ },
1549
+ {
1550
+ "options": 15,
1551
+ "choice": "click element 4",
1552
+ "p": 0.43631729559271687
1553
+ },
1554
+ {
1555
+ "options": 6,
1556
+ "choice": "the task is complete",
1557
+ "p": 0.9751536258590529
1558
+ }
1559
+ ]
1560
+ },
1561
+ {
1562
+ "app": "shop",
1563
+ "seed": 7,
1564
+ "variant": "marks",
1565
+ "task": "Buy a yellow Hoodie in size L: add it to the cart and check out.",
1566
+ "success": false,
1567
+ "goal_met": false,
1568
+ "stopped": true,
1569
+ "steps": 6,
1570
+ "trace": [
1571
+ {
1572
+ "options": 10,
1573
+ "choice": "click element 5",
1574
+ "p": 0.9289429773474432
1575
+ },
1576
+ {
1577
+ "options": 15,
1578
+ "choice": "click element 9",
1579
+ "p": 0.9932170754901455
1580
+ },
1581
+ {
1582
+ "options": 15,
1583
+ "choice": "click element 12",
1584
+ "p": 0.8321332823471849
1585
+ },
1586
+ {
1587
+ "options": 15,
1588
+ "choice": "click element 14",
1589
+ "p": 0.9840190780467732
1590
+ },
1591
+ {
1592
+ "options": 15,
1593
+ "choice": "click element 4",
1594
+ "p": 0.48966729581964946
1595
+ },
1596
+ {
1597
+ "options": 6,
1598
+ "choice": "the task is complete",
1599
+ "p": 0.90910168336184
1600
+ }
1601
+ ]
1602
+ },
1603
+ {
1604
+ "app": "shop",
1605
+ "seed": 13,
1606
+ "variant": "marks",
1607
+ "task": "Buy a red Hoodie in size L: add it to the cart and check out.",
1608
+ "success": false,
1609
+ "goal_met": false,
1610
+ "stopped": true,
1611
+ "steps": 6,
1612
+ "trace": [
1613
+ {
1614
+ "options": 10,
1615
+ "choice": "click element 9",
1616
+ "p": 0.991030158484895
1617
+ },
1618
+ {
1619
+ "options": 15,
1620
+ "choice": "click element 5",
1621
+ "p": 0.7043170750165028
1622
+ },
1623
+ {
1624
+ "options": 15,
1625
+ "choice": "click element 12",
1626
+ "p": 0.9062719628987717
1627
+ },
1628
+ {
1629
+ "options": 15,
1630
+ "choice": "click element 14",
1631
+ "p": 0.9913608684740531
1632
+ },
1633
+ {
1634
+ "options": 15,
1635
+ "choice": "click element 4",
1636
+ "p": 0.5227950049510057
1637
+ },
1638
+ {
1639
+ "options": 6,
1640
+ "choice": "the task is complete",
1641
+ "p": 0.944637088321295
1642
+ }
1643
+ ]
1644
+ },
1645
+ {
1646
+ "app": "shop",
1647
+ "seed": 19,
1648
+ "variant": "marks",
1649
+ "task": "Buy a blue Hoodie in size M: add it to the cart and check out.",
1650
+ "success": false,
1651
+ "goal_met": false,
1652
+ "stopped": true,
1653
+ "steps": 6,
1654
+ "trace": [
1655
+ {
1656
+ "options": 10,
1657
+ "choice": "click element 4",
1658
+ "p": 0.9077247245545136
1659
+ },
1660
+ {
1661
+ "options": 15,
1662
+ "choice": "click element 6",
1663
+ "p": 0.9898188184798979
1664
+ },
1665
+ {
1666
+ "options": 15,
1667
+ "choice": "click element 11",
1668
+ "p": 0.9464649526716705
1669
+ },
1670
+ {
1671
+ "options": 15,
1672
+ "choice": "click element 14",
1673
+ "p": 0.9843282251462333
1674
+ },
1675
+ {
1676
+ "options": 15,
1677
+ "choice": "click element 4",
1678
+ "p": 0.2766443618754146
1679
+ },
1680
+ {
1681
+ "options": 6,
1682
+ "choice": "the task is complete",
1683
+ "p": 0.8037782839935003
1684
+ }
1685
+ ]
1686
+ },
1687
+ {
1688
+ "app": "settings",
1689
+ "seed": 5,
1690
+ "variant": "marks",
1691
+ "task": "Turn on \"Push notifications\" in the Notifications settings and save the change.",
1692
+ "success": true,
1693
+ "goal_met": true,
1694
+ "stopped": true,
1695
+ "steps": 4,
1696
+ "trace": [
1697
+ {
1698
+ "options": 7,
1699
+ "choice": "click element 6",
1700
+ "p": 0.9958939165113844
1701
+ },
1702
+ {
1703
+ "options": 11,
1704
+ "choice": "click element 8",
1705
+ "p": 0.999960974536647
1706
+ },
1707
+ {
1708
+ "options": 11,
1709
+ "choice": "click element 9",
1710
+ "p": 0.9990585217282378
1711
+ },
1712
+ {
1713
+ "options": 11,
1714
+ "choice": "the task is complete",
1715
+ "p": 0.9884240638076942
1716
+ }
1717
+ ]
1718
+ },
1719
+ {
1720
+ "app": "settings",
1721
+ "seed": 11,
1722
+ "variant": "marks",
1723
+ "task": "Turn on \"Push notifications\" in the Notifications settings and save the change.",
1724
+ "success": true,
1725
+ "goal_met": true,
1726
+ "stopped": true,
1727
+ "steps": 4,
1728
+ "trace": [
1729
+ {
1730
+ "options": 7,
1731
+ "choice": "click element 5",
1732
+ "p": 0.6573439674745547
1733
+ },
1734
+ {
1735
+ "options": 11,
1736
+ "choice": "click element 8",
1737
+ "p": 0.9904128810866905
1738
+ },
1739
+ {
1740
+ "options": 11,
1741
+ "choice": "click element 9",
1742
+ "p": 0.998773792864025
1743
+ },
1744
+ {
1745
+ "options": 11,
1746
+ "choice": "the task is complete",
1747
+ "p": 0.9648748061389091
1748
+ }
1749
+ ]
1750
+ },
1751
+ {
1752
+ "app": "settings",
1753
+ "seed": 17,
1754
+ "variant": "marks",
1755
+ "task": "Turn on \"Email digests\" in the Notifications settings and save the change.",
1756
+ "success": true,
1757
+ "goal_met": true,
1758
+ "stopped": true,
1759
+ "steps": 4,
1760
+ "trace": [
1761
+ {
1762
+ "options": 7,
1763
+ "choice": "click element 6",
1764
+ "p": 0.9977362582394523
1765
+ },
1766
+ {
1767
+ "options": 11,
1768
+ "choice": "click element 7",
1769
+ "p": 0.999788174419673
1770
+ },
1771
+ {
1772
+ "options": 11,
1773
+ "choice": "click element 9",
1774
+ "p": 0.9944024512942901
1775
+ },
1776
+ {
1777
+ "options": 11,
1778
+ "choice": "the task is complete",
1779
+ "p": 0.9861272891224006
1780
+ }
1781
+ ]
1782
+ },
1783
+ {
1784
+ "app": "mail",
1785
+ "seed": 3,
1786
+ "variant": "marks+text",
1787
+ "task": "Archive the email from David Kim about the quarterly report.",
1788
+ "success": true,
1789
+ "goal_met": true,
1790
+ "stopped": true,
1791
+ "steps": 3,
1792
+ "trace": [
1793
+ {
1794
+ "options": 10,
1795
+ "choice": "click element 4: David Kim quarterly report (list row)",
1796
+ "p": 0.9952371496910286
1797
+ },
1798
+ {
1799
+ "options": 9,
1800
+ "choice": "click element 6: Archive (button)",
1801
+ "p": 0.998120127796241
1802
+ },
1803
+ {
1804
+ "options": 9,
1805
+ "choice": "the task is complete",
1806
+ "p": 0.9744201429328296
1807
+ }
1808
+ ]
1809
+ },
1810
+ {
1811
+ "app": "mail",
1812
+ "seed": 9,
1813
+ "variant": "marks+text",
1814
+ "task": "Archive the email from Bob Martin about the team offsite.",
1815
+ "success": true,
1816
+ "goal_met": true,
1817
+ "stopped": true,
1818
+ "steps": 3,
1819
+ "trace": [
1820
+ {
1821
+ "options": 10,
1822
+ "choice": "click element 9: Bob Martin team offsite (list row)",
1823
+ "p": 0.9999868081391723
1824
+ },
1825
+ {
1826
+ "options": 9,
1827
+ "choice": "click element 6: Archive (button)",
1828
+ "p": 0.9999322150619298
1829
+ },
1830
+ {
1831
+ "options": 9,
1832
+ "choice": "the task is complete",
1833
+ "p": 0.992569728998587
1834
+ }
1835
+ ]
1836
+ },
1837
+ {
1838
+ "app": "mail",
1839
+ "seed": 15,
1840
+ "variant": "marks+text",
1841
+ "task": "Archive the email from Emma Rossi about the invoice for March.",
1842
+ "success": true,
1843
+ "goal_met": true,
1844
+ "stopped": true,
1845
+ "steps": 3,
1846
+ "trace": [
1847
+ {
1848
+ "options": 10,
1849
+ "choice": "click element 5: Emma Rossi invoice for March (list row)",
1850
+ "p": 0.9992222579588744
1851
+ },
1852
+ {
1853
+ "options": 9,
1854
+ "choice": "click element 6: Archive (button)",
1855
+ "p": 0.9975109962949797
1856
+ },
1857
+ {
1858
+ "options": 9,
1859
+ "choice": "the task is complete",
1860
+ "p": 0.982054501907486
1861
+ }
1862
+ ]
1863
+ },
1864
+ {
1865
+ "app": "shop",
1866
+ "seed": 1,
1867
+ "variant": "marks+text",
1868
+ "task": "Buy a yellow T-shirt in size S: add it to the cart and check out.",
1869
+ "success": true,
1870
+ "goal_met": true,
1871
+ "stopped": true,
1872
+ "steps": 7,
1873
+ "trace": [
1874
+ {
1875
+ "options": 10,
1876
+ "choice": "click element 7: T-shirt $40 (product card)",
1877
+ "p": 0.9740591912318491
1878
+ },
1879
+ {
1880
+ "options": 15,
1881
+ "choice": "click element 9: colour swatch",
1882
+ "p": 0.9759231406621297
1883
+ },
1884
+ {
1885
+ "options": 15,
1886
+ "choice": "click element 10: S (button)",
1887
+ "p": 0.8900758446119122
1888
+ },
1889
+ {
1890
+ "options": 15,
1891
+ "choice": "click element 14: Add to cart (button)",
1892
+ "p": 0.9985380209641412
1893
+ },
1894
+ {
1895
+ "options": 15,
1896
+ "choice": "click element 4: Cart (1) (button)",
1897
+ "p": 0.8978607987249052
1898
+ },
1899
+ {
1900
+ "options": 6,
1901
+ "choice": "click element 4: Checkout (button)",
1902
+ "p": 0.8704514891952391
1903
+ },
1904
+ {
1905
+ "options": 6,
1906
+ "choice": "the task is complete",
1907
+ "p": 0.987484227256529
1908
+ }
1909
+ ]
1910
+ },
1911
+ {
1912
+ "app": "shop",
1913
+ "seed": 7,
1914
+ "variant": "marks+text",
1915
+ "task": "Buy a yellow Hoodie in size L: add it to the cart and check out.",
1916
+ "success": true,
1917
+ "goal_met": true,
1918
+ "stopped": true,
1919
+ "steps": 7,
1920
+ "trace": [
1921
+ {
1922
+ "options": 10,
1923
+ "choice": "click element 5: Hoodie $26 (product card)",
1924
+ "p": 0.963667775145069
1925
+ },
1926
+ {
1927
+ "options": 15,
1928
+ "choice": "click element 9: colour swatch",
1929
+ "p": 0.9712447647070995
1930
+ },
1931
+ {
1932
+ "options": 15,
1933
+ "choice": "click element 12: L (button)",
1934
+ "p": 0.978977233548299
1935
+ },
1936
+ {
1937
+ "options": 15,
1938
+ "choice": "click element 14: Add to cart (button)",
1939
+ "p": 0.9987039650705727
1940
+ },
1941
+ {
1942
+ "options": 15,
1943
+ "choice": "click element 4: Cart (1) (button)",
1944
+ "p": 0.9021430832310717
1945
+ },
1946
+ {
1947
+ "options": 6,
1948
+ "choice": "click element 4: Checkout (button)",
1949
+ "p": 0.9423090118612728
1950
+ },
1951
+ {
1952
+ "options": 6,
1953
+ "choice": "the task is complete",
1954
+ "p": 0.9820948481733074
1955
+ }
1956
+ ]
1957
+ },
1958
+ {
1959
+ "app": "shop",
1960
+ "seed": 13,
1961
+ "variant": "marks+text",
1962
+ "task": "Buy a red Hoodie in size L: add it to the cart and check out.",
1963
+ "success": true,
1964
+ "goal_met": true,
1965
+ "stopped": true,
1966
+ "steps": 7,
1967
+ "trace": [
1968
+ {
1969
+ "options": 10,
1970
+ "choice": "click element 9: Hoodie $54 (product card)",
1971
+ "p": 0.9976998363005869
1972
+ },
1973
+ {
1974
+ "options": 15,
1975
+ "choice": "click element 5: colour swatch",
1976
+ "p": 0.5975121320987309
1977
+ },
1978
+ {
1979
+ "options": 15,
1980
+ "choice": "click element 12: L (button)",
1981
+ "p": 0.9644448183475047
1982
+ },
1983
+ {
1984
+ "options": 15,
1985
+ "choice": "click element 14: Add to cart (button)",
1986
+ "p": 0.9975853359368868
1987
+ },
1988
+ {
1989
+ "options": 15,
1990
+ "choice": "click element 4: Cart (1) (button)",
1991
+ "p": 0.9238046809385214
1992
+ },
1993
+ {
1994
+ "options": 6,
1995
+ "choice": "click element 4: Checkout (button)",
1996
+ "p": 0.9350430068110196
1997
+ },
1998
+ {
1999
+ "options": 6,
2000
+ "choice": "the task is complete",
2001
+ "p": 0.9885171237123968
2002
+ }
2003
+ ]
2004
+ },
2005
+ {
2006
+ "app": "shop",
2007
+ "seed": 19,
2008
+ "variant": "marks+text",
2009
+ "task": "Buy a blue Hoodie in size M: add it to the cart and check out.",
2010
+ "success": true,
2011
+ "goal_met": true,
2012
+ "stopped": true,
2013
+ "steps": 7,
2014
+ "trace": [
2015
+ {
2016
+ "options": 10,
2017
+ "choice": "click element 4: Hoodie $19 (product card)",
2018
+ "p": 0.9614889198814917
2019
+ },
2020
+ {
2021
+ "options": 15,
2022
+ "choice": "click element 6: colour swatch",
2023
+ "p": 0.9536291966235857
2024
+ },
2025
+ {
2026
+ "options": 15,
2027
+ "choice": "click element 11: M (button)",
2028
+ "p": 0.9790026806067829
2029
+ },
2030
+ {
2031
+ "options": 15,
2032
+ "choice": "click element 14: Add to cart (button)",
2033
+ "p": 0.9986270119611157
2034
+ },
2035
+ {
2036
+ "options": 15,
2037
+ "choice": "click element 4: Cart (1) (button)",
2038
+ "p": 0.8663043379054834
2039
+ },
2040
+ {
2041
+ "options": 6,
2042
+ "choice": "click element 4: Checkout (button)",
2043
+ "p": 0.9573331464379526
2044
+ },
2045
+ {
2046
+ "options": 6,
2047
+ "choice": "the task is complete",
2048
+ "p": 0.9867372354449497
2049
+ }
2050
+ ]
2051
+ },
2052
+ {
2053
+ "app": "settings",
2054
+ "seed": 5,
2055
+ "variant": "marks+text",
2056
+ "task": "Turn on \"Push notifications\" in the Notifications settings and save the change.",
2057
+ "success": true,
2058
+ "goal_met": true,
2059
+ "stopped": true,
2060
+ "steps": 4,
2061
+ "trace": [
2062
+ {
2063
+ "options": 7,
2064
+ "choice": "click element 6: Notifications (button)",
2065
+ "p": 0.9999851359456106
2066
+ },
2067
+ {
2068
+ "options": 11,
2069
+ "choice": "click element 8: switch",
2070
+ "p": 0.9999529034342831
2071
+ },
2072
+ {
2073
+ "options": 11,
2074
+ "choice": "click element 9: Save changes (button)",
2075
+ "p": 0.999393528660747
2076
+ },
2077
+ {
2078
+ "options": 11,
2079
+ "choice": "the task is complete",
2080
+ "p": 0.9975966277343157
2081
+ }
2082
+ ]
2083
+ },
2084
+ {
2085
+ "app": "settings",
2086
+ "seed": 11,
2087
+ "variant": "marks+text",
2088
+ "task": "Turn on \"Push notifications\" in the Notifications settings and save the change.",
2089
+ "success": true,
2090
+ "goal_met": true,
2091
+ "stopped": true,
2092
+ "steps": 4,
2093
+ "trace": [
2094
+ {
2095
+ "options": 7,
2096
+ "choice": "click element 5: Notifications (button)",
2097
+ "p": 0.9998418248744801
2098
+ },
2099
+ {
2100
+ "options": 11,
2101
+ "choice": "click element 8: switch",
2102
+ "p": 0.999958143903509
2103
+ },
2104
+ {
2105
+ "options": 11,
2106
+ "choice": "click element 9: Save changes (button)",
2107
+ "p": 0.999683199898457
2108
+ },
2109
+ {
2110
+ "options": 11,
2111
+ "choice": "the task is complete",
2112
+ "p": 0.9943518558112904
2113
+ }
2114
+ ]
2115
+ },
2116
+ {
2117
+ "app": "settings",
2118
+ "seed": 17,
2119
+ "variant": "marks+text",
2120
+ "task": "Turn on \"Email digests\" in the Notifications settings and save the change.",
2121
+ "success": true,
2122
+ "goal_met": true,
2123
+ "stopped": true,
2124
+ "steps": 4,
2125
+ "trace": [
2126
+ {
2127
+ "options": 7,
2128
+ "choice": "click element 6: Notifications (button)",
2129
+ "p": 0.9999797971102274
2130
+ },
2131
+ {
2132
+ "options": 11,
2133
+ "choice": "click element 7: switch",
2134
+ "p": 0.9994324394495737
2135
+ },
2136
+ {
2137
+ "options": 11,
2138
+ "choice": "click element 9: Save changes (button)",
2139
+ "p": 0.997233540081523
2140
+ },
2141
+ {
2142
+ "options": 11,
2143
+ "choice": "the task is complete",
2144
+ "p": 0.9967343460531148
2145
+ }
2146
+ ]
2147
+ },
2148
+ {
2149
+ "app": "mail",
2150
+ "seed": 4,
2151
+ "variant": "marks",
2152
+ "task": "Archive the email from Emma Rossi about the lunch on Friday.",
2153
+ "success": false,
2154
+ "goal_met": false,
2155
+ "stopped": true,
2156
+ "steps": 2,
2157
+ "trace": [
2158
+ {
2159
+ "options": 10,
2160
+ "choice": "click element 6",
2161
+ "p": 0.9855487551663377
2162
+ },
2163
+ {
2164
+ "options": 9,
2165
+ "choice": "the task is complete",
2166
+ "p": 0.9308825906967287
2167
+ }
2168
+ ]
2169
+ },
2170
+ {
2171
+ "app": "mail",
2172
+ "seed": 10,
2173
+ "variant": "marks",
2174
+ "task": "Archive the email from Farid Khan about the travel booking.",
2175
+ "success": false,
2176
+ "goal_met": false,
2177
+ "stopped": true,
2178
+ "steps": 2,
2179
+ "trace": [
2180
+ {
2181
+ "options": 10,
2182
+ "choice": "click element 6",
2183
+ "p": 0.9852244373508147
2184
+ },
2185
+ {
2186
+ "options": 9,
2187
+ "choice": "the task is complete",
2188
+ "p": 0.9140660760871496
2189
+ }
2190
+ ]
2191
+ },
2192
+ {
2193
+ "app": "mail",
2194
+ "seed": 16,
2195
+ "variant": "marks",
2196
+ "task": "Archive the email from Farid Khan about the lunch on Friday.",
2197
+ "success": false,
2198
+ "goal_met": false,
2199
+ "stopped": true,
2200
+ "steps": 4,
2201
+ "trace": [
2202
+ {
2203
+ "options": 10,
2204
+ "choice": "click element 9",
2205
+ "p": 0.9972137776650049
2206
+ },
2207
+ {
2208
+ "options": 9,
2209
+ "choice": "click element 6",
2210
+ "p": 0.7362238314361507
2211
+ },
2212
+ {
2213
+ "options": 9,
2214
+ "choice": "click element 8",
2215
+ "p": 0.9532336855993342
2216
+ },
2217
+ {
2218
+ "options": 9,
2219
+ "choice": "the task is complete",
2220
+ "p": 0.897518995028419
2221
+ }
2222
+ ]
2223
+ },
2224
+ {
2225
+ "app": "shop",
2226
+ "seed": 2,
2227
+ "variant": "marks",
2228
+ "task": "Buy a black Jacket in size S: add it to the cart and check out.",
2229
+ "success": false,
2230
+ "goal_met": false,
2231
+ "stopped": true,
2232
+ "steps": 6,
2233
+ "trace": [
2234
+ {
2235
+ "options": 10,
2236
+ "choice": "click element 5",
2237
+ "p": 0.9609309782002599
2238
+ },
2239
+ {
2240
+ "options": 15,
2241
+ "choice": "click element 8",
2242
+ "p": 0.9396790619223374
2243
+ },
2244
+ {
2245
+ "options": 15,
2246
+ "choice": "click element 10",
2247
+ "p": 0.9018170161205531
2248
+ },
2249
+ {
2250
+ "options": 15,
2251
+ "choice": "click element 14",
2252
+ "p": 0.996378990977478
2253
+ },
2254
+ {
2255
+ "options": 15,
2256
+ "choice": "click element 4",
2257
+ "p": 0.5631464953077281
2258
+ },
2259
+ {
2260
+ "options": 6,
2261
+ "choice": "the task is complete",
2262
+ "p": 0.9319969987750208
2263
+ }
2264
+ ]
2265
+ },
2266
+ {
2267
+ "app": "shop",
2268
+ "seed": 8,
2269
+ "variant": "marks",
2270
+ "task": "Buy a black Sneakers in size XL: add it to the cart and check out.",
2271
+ "success": false,
2272
+ "goal_met": false,
2273
+ "stopped": true,
2274
+ "steps": 6,
2275
+ "trace": [
2276
+ {
2277
+ "options": 10,
2278
+ "choice": "click element 4",
2279
+ "p": 0.9290852645605687
2280
+ },
2281
+ {
2282
+ "options": 15,
2283
+ "choice": "click element 8",
2284
+ "p": 0.9835174004714023
2285
+ },
2286
+ {
2287
+ "options": 15,
2288
+ "choice": "click element 13",
2289
+ "p": 0.9886847375645783
2290
+ },
2291
+ {
2292
+ "options": 15,
2293
+ "choice": "click element 14",
2294
+ "p": 0.9974020135975942
2295
+ },
2296
+ {
2297
+ "options": 15,
2298
+ "choice": "click element 2",
2299
+ "p": 0.32275330837769406
2300
+ },
2301
+ {
2302
+ "options": 15,
2303
+ "choice": "the task is complete",
2304
+ "p": 0.3005283192869037
2305
+ }
2306
+ ]
2307
+ },
2308
+ {
2309
+ "app": "shop",
2310
+ "seed": 14,
2311
+ "variant": "marks",
2312
+ "task": "Buy a yellow T-shirt in size M: add it to the cart and check out.",
2313
+ "success": false,
2314
+ "goal_met": false,
2315
+ "stopped": true,
2316
+ "steps": 6,
2317
+ "trace": [
2318
+ {
2319
+ "options": 10,
2320
+ "choice": "click element 7",
2321
+ "p": 0.8957954100251765
2322
+ },
2323
+ {
2324
+ "options": 15,
2325
+ "choice": "click element 9",
2326
+ "p": 0.9527137105108581
2327
+ },
2328
+ {
2329
+ "options": 15,
2330
+ "choice": "click element 11",
2331
+ "p": 0.8418209101275489
2332
+ },
2333
+ {
2334
+ "options": 15,
2335
+ "choice": "click element 14",
2336
+ "p": 0.985780767306956
2337
+ },
2338
+ {
2339
+ "options": 15,
2340
+ "choice": "click element 4",
2341
+ "p": 0.5725218740452435
2342
+ },
2343
+ {
2344
+ "options": 6,
2345
+ "choice": "the task is complete",
2346
+ "p": 0.9731734853402912
2347
+ }
2348
+ ]
2349
+ },
2350
+ {
2351
+ "app": "shop",
2352
+ "seed": 20,
2353
+ "variant": "marks",
2354
+ "task": "Buy a green T-shirt in size L: add it to the cart and check out.",
2355
+ "success": false,
2356
+ "goal_met": false,
2357
+ "stopped": true,
2358
+ "steps": 5,
2359
+ "trace": [
2360
+ {
2361
+ "options": 10,
2362
+ "choice": "click element 7",
2363
+ "p": 0.8981146342911692
2364
+ },
2365
+ {
2366
+ "options": 15,
2367
+ "choice": "click element 12",
2368
+ "p": 0.973576385408416
2369
+ },
2370
+ {
2371
+ "options": 15,
2372
+ "choice": "click element 14",
2373
+ "p": 0.9947437700951561
2374
+ },
2375
+ {
2376
+ "options": 15,
2377
+ "choice": "click element 4",
2378
+ "p": 0.5379663256325706
2379
+ },
2380
+ {
2381
+ "options": 6,
2382
+ "choice": "the task is complete",
2383
+ "p": 0.6553060678980768
2384
+ }
2385
+ ]
2386
+ },
2387
+ {
2388
+ "app": "settings",
2389
+ "seed": 6,
2390
+ "variant": "marks",
2391
+ "task": "Turn on \"Email digests\" in the Notifications settings and save the change.",
2392
+ "success": true,
2393
+ "goal_met": true,
2394
+ "stopped": true,
2395
+ "steps": 4,
2396
+ "trace": [
2397
+ {
2398
+ "options": 7,
2399
+ "choice": "click element 6",
2400
+ "p": 0.9946522153525018
2401
+ },
2402
+ {
2403
+ "options": 11,
2404
+ "choice": "click element 7",
2405
+ "p": 0.9997800537380285
2406
+ },
2407
+ {
2408
+ "options": 11,
2409
+ "choice": "click element 9",
2410
+ "p": 0.9931225113501516
2411
+ },
2412
+ {
2413
+ "options": 11,
2414
+ "choice": "the task is complete",
2415
+ "p": 0.983254758203928
2416
+ }
2417
+ ]
2418
+ },
2419
+ {
2420
+ "app": "settings",
2421
+ "seed": 12,
2422
+ "variant": "marks",
2423
+ "task": "Turn on \"Compact layout\" in the Appearance settings and save the change.",
2424
+ "success": true,
2425
+ "goal_met": true,
2426
+ "stopped": true,
2427
+ "steps": 4,
2428
+ "trace": [
2429
+ {
2430
+ "options": 7,
2431
+ "choice": "click element 4",
2432
+ "p": 0.6771187385863631
2433
+ },
2434
+ {
2435
+ "options": 11,
2436
+ "choice": "click element 8",
2437
+ "p": 0.9997470954486974
2438
+ },
2439
+ {
2440
+ "options": 11,
2441
+ "choice": "click element 9",
2442
+ "p": 0.999907418123659
2443
+ },
2444
+ {
2445
+ "options": 11,
2446
+ "choice": "the task is complete",
2447
+ "p": 0.9835333339517102
2448
+ }
2449
+ ]
2450
+ },
2451
+ {
2452
+ "app": "settings",
2453
+ "seed": 18,
2454
+ "variant": "marks",
2455
+ "task": "Turn on \"Show online status\" in the Profile settings and save the change.",
2456
+ "success": true,
2457
+ "goal_met": true,
2458
+ "stopped": true,
2459
+ "steps": 6,
2460
+ "trace": [
2461
+ {
2462
+ "options": 7,
2463
+ "choice": "click element 6",
2464
+ "p": 0.860364468713165
2465
+ },
2466
+ {
2467
+ "options": 11,
2468
+ "choice": "click element 3",
2469
+ "p": 0.30531098696231596
2470
+ },
2471
+ {
2472
+ "options": 11,
2473
+ "choice": "click element 5",
2474
+ "p": 0.25359959705762425
2475
+ },
2476
+ {
2477
+ "options": 11,
2478
+ "choice": "click element 7",
2479
+ "p": 0.9981164071772921
2480
+ },
2481
+ {
2482
+ "options": 11,
2483
+ "choice": "click element 9",
2484
+ "p": 0.9948318206465058
2485
+ },
2486
+ {
2487
+ "options": 11,
2488
+ "choice": "the task is complete",
2489
+ "p": 0.9580818329237141
2490
+ }
2491
+ ]
2492
+ },
2493
+ {
2494
+ "app": "mail",
2495
+ "seed": 4,
2496
+ "variant": "marks+text",
2497
+ "task": "Archive the email from Emma Rossi about the lunch on Friday.",
2498
+ "success": true,
2499
+ "goal_met": true,
2500
+ "stopped": true,
2501
+ "steps": 3,
2502
+ "trace": [
2503
+ {
2504
+ "options": 10,
2505
+ "choice": "click element 5: Emma Rossi lunch on Friday (list row)",
2506
+ "p": 0.9975740282709057
2507
+ },
2508
+ {
2509
+ "options": 9,
2510
+ "choice": "click element 6: Archive (button)",
2511
+ "p": 0.9986279950212591
2512
+ },
2513
+ {
2514
+ "options": 9,
2515
+ "choice": "the task is complete",
2516
+ "p": 0.9721280369484723
2517
+ }
2518
+ ]
2519
+ },
2520
+ {
2521
+ "app": "mail",
2522
+ "seed": 10,
2523
+ "variant": "marks+text",
2524
+ "task": "Archive the email from Farid Khan about the travel booking.",
2525
+ "success": true,
2526
+ "goal_met": true,
2527
+ "stopped": true,
2528
+ "steps": 3,
2529
+ "trace": [
2530
+ {
2531
+ "options": 10,
2532
+ "choice": "click element 5: Farid Khan travel booking (list row)",
2533
+ "p": 0.9982699772427862
2534
+ },
2535
+ {
2536
+ "options": 9,
2537
+ "choice": "click element 6: Archive (button)",
2538
+ "p": 0.9983118964905628
2539
+ },
2540
+ {
2541
+ "options": 9,
2542
+ "choice": "the task is complete",
2543
+ "p": 0.9905520244311105
2544
+ }
2545
+ ]
2546
+ },
2547
+ {
2548
+ "app": "mail",
2549
+ "seed": 16,
2550
+ "variant": "marks+text",
2551
+ "task": "Archive the email from Farid Khan about the lunch on Friday.",
2552
+ "success": true,
2553
+ "goal_met": true,
2554
+ "stopped": true,
2555
+ "steps": 3,
2556
+ "trace": [
2557
+ {
2558
+ "options": 10,
2559
+ "choice": "click element 8: Farid Khan lunch on Friday (list row)",
2560
+ "p": 0.9999809451500565
2561
+ },
2562
+ {
2563
+ "options": 9,
2564
+ "choice": "click element 6: Archive (button)",
2565
+ "p": 0.9997292910799996
2566
+ },
2567
+ {
2568
+ "options": 9,
2569
+ "choice": "the task is complete",
2570
+ "p": 0.9870400226989652
2571
+ }
2572
+ ]
2573
+ },
2574
+ {
2575
+ "app": "shop",
2576
+ "seed": 2,
2577
+ "variant": "marks+text",
2578
+ "task": "Buy a black Jacket in size S: add it to the cart and check out.",
2579
+ "success": true,
2580
+ "goal_met": true,
2581
+ "stopped": true,
2582
+ "steps": 7,
2583
+ "trace": [
2584
+ {
2585
+ "options": 10,
2586
+ "choice": "click element 5: Jacket $26 (product card)",
2587
+ "p": 0.9827851210771855
2588
+ },
2589
+ {
2590
+ "options": 15,
2591
+ "choice": "click element 8: colour swatch",
2592
+ "p": 0.9512028750586858
2593
+ },
2594
+ {
2595
+ "options": 15,
2596
+ "choice": "click element 10: S (button)",
2597
+ "p": 0.9172164523902843
2598
+ },
2599
+ {
2600
+ "options": 15,
2601
+ "choice": "click element 14: Add to cart (button)",
2602
+ "p": 0.9991719465167891
2603
+ },
2604
+ {
2605
+ "options": 15,
2606
+ "choice": "click element 4: Cart (1) (button)",
2607
+ "p": 0.9407980052505666
2608
+ },
2609
+ {
2610
+ "options": 6,
2611
+ "choice": "click element 4: Checkout (button)",
2612
+ "p": 0.9076550297300644
2613
+ },
2614
+ {
2615
+ "options": 6,
2616
+ "choice": "the task is complete",
2617
+ "p": 0.9833290856419374
2618
+ }
2619
+ ]
2620
+ },
2621
+ {
2622
+ "app": "shop",
2623
+ "seed": 8,
2624
+ "variant": "marks+text",
2625
+ "task": "Buy a black Sneakers in size XL: add it to the cart and check out.",
2626
+ "success": true,
2627
+ "goal_met": true,
2628
+ "stopped": true,
2629
+ "steps": 7,
2630
+ "trace": [
2631
+ {
2632
+ "options": 10,
2633
+ "choice": "click element 4: Sneakers $19 (product card)",
2634
+ "p": 0.9676992827619597
2635
+ },
2636
+ {
2637
+ "options": 15,
2638
+ "choice": "click element 8: colour swatch",
2639
+ "p": 0.9819713432247609
2640
+ },
2641
+ {
2642
+ "options": 15,
2643
+ "choice": "click element 13: XL (button)",
2644
+ "p": 0.9958746162915724
2645
+ },
2646
+ {
2647
+ "options": 15,
2648
+ "choice": "click element 14: Add to cart (button)",
2649
+ "p": 0.9994346913700353
2650
+ },
2651
+ {
2652
+ "options": 15,
2653
+ "choice": "click element 4: Cart (1) (button)",
2654
+ "p": 0.8058434759165681
2655
+ },
2656
+ {
2657
+ "options": 6,
2658
+ "choice": "click element 4: Checkout (button)",
2659
+ "p": 0.9128849961763449
2660
+ },
2661
+ {
2662
+ "options": 6,
2663
+ "choice": "the task is complete",
2664
+ "p": 0.984380423345111
2665
+ }
2666
+ ]
2667
+ },
2668
+ {
2669
+ "app": "shop",
2670
+ "seed": 14,
2671
+ "variant": "marks+text",
2672
+ "task": "Buy a yellow T-shirt in size M: add it to the cart and check out.",
2673
+ "success": true,
2674
+ "goal_met": true,
2675
+ "stopped": true,
2676
+ "steps": 7,
2677
+ "trace": [
2678
+ {
2679
+ "options": 10,
2680
+ "choice": "click element 7: T-shirt $40 (product card)",
2681
+ "p": 0.97411548061535
2682
+ },
2683
+ {
2684
+ "options": 15,
2685
+ "choice": "click element 9: colour swatch",
2686
+ "p": 0.9593834606086283
2687
+ },
2688
+ {
2689
+ "options": 15,
2690
+ "choice": "click element 11: M (button)",
2691
+ "p": 0.9833986409904353
2692
+ },
2693
+ {
2694
+ "options": 15,
2695
+ "choice": "click element 14: Add to cart (button)",
2696
+ "p": 0.9985064778608521
2697
+ },
2698
+ {
2699
+ "options": 15,
2700
+ "choice": "click element 4: Cart (1) (button)",
2701
+ "p": 0.9205587157263847
2702
+ },
2703
+ {
2704
+ "options": 6,
2705
+ "choice": "click element 4: Checkout (button)",
2706
+ "p": 0.863131126413095
2707
+ },
2708
+ {
2709
+ "options": 6,
2710
+ "choice": "the task is complete",
2711
+ "p": 0.9878627456829393
2712
+ }
2713
+ ]
2714
+ },
2715
+ {
2716
+ "app": "shop",
2717
+ "seed": 20,
2718
+ "variant": "marks+text",
2719
+ "task": "Buy a green T-shirt in size L: add it to the cart and check out.",
2720
+ "success": false,
2721
+ "goal_met": false,
2722
+ "stopped": true,
2723
+ "steps": 6,
2724
+ "trace": [
2725
+ {
2726
+ "options": 10,
2727
+ "choice": "click element 7: T-shirt $40 (product card)",
2728
+ "p": 0.970578779632661
2729
+ },
2730
+ {
2731
+ "options": 15,
2732
+ "choice": "click element 12: L (button)",
2733
+ "p": 0.5990679873207809
2734
+ },
2735
+ {
2736
+ "options": 15,
2737
+ "choice": "click element 14: Add to cart (button)",
2738
+ "p": 0.9956647741439458
2739
+ },
2740
+ {
2741
+ "options": 15,
2742
+ "choice": "click element 4: Cart (0) (button)",
2743
+ "p": 0.8757491522466574
2744
+ },
2745
+ {
2746
+ "options": 6,
2747
+ "choice": "click element 4: Checkout (button)",
2748
+ "p": 0.7774576449337983
2749
+ },
2750
+ {
2751
+ "options": 6,
2752
+ "choice": "the task is complete",
2753
+ "p": 0.7690601700268177
2754
+ }
2755
+ ]
2756
+ },
2757
+ {
2758
+ "app": "settings",
2759
+ "seed": 6,
2760
+ "variant": "marks+text",
2761
+ "task": "Turn on \"Email digests\" in the Notifications settings and save the change.",
2762
+ "success": true,
2763
+ "goal_met": true,
2764
+ "stopped": true,
2765
+ "steps": 4,
2766
+ "trace": [
2767
+ {
2768
+ "options": 7,
2769
+ "choice": "click element 6: Notifications (button)",
2770
+ "p": 0.9999842016504991
2771
+ },
2772
+ {
2773
+ "options": 11,
2774
+ "choice": "click element 7: switch",
2775
+ "p": 0.9994150620395968
2776
+ },
2777
+ {
2778
+ "options": 11,
2779
+ "choice": "click element 9: Save changes (button)",
2780
+ "p": 0.9972876521304855
2781
+ },
2782
+ {
2783
+ "options": 11,
2784
+ "choice": "the task is complete",
2785
+ "p": 0.9960997796797132
2786
+ }
2787
+ ]
2788
+ },
2789
+ {
2790
+ "app": "settings",
2791
+ "seed": 12,
2792
+ "variant": "marks+text",
2793
+ "task": "Turn on \"Compact layout\" in the Appearance settings and save the change.",
2794
+ "success": true,
2795
+ "goal_met": true,
2796
+ "stopped": true,
2797
+ "steps": 4,
2798
+ "trace": [
2799
+ {
2800
+ "options": 7,
2801
+ "choice": "click element 4: Appearance (button)",
2802
+ "p": 0.9999691692678567
2803
+ },
2804
+ {
2805
+ "options": 11,
2806
+ "choice": "click element 8: switch",
2807
+ "p": 0.9999730053229482
2808
+ },
2809
+ {
2810
+ "options": 11,
2811
+ "choice": "click element 9: Save changes (button)",
2812
+ "p": 0.9999830929907714
2813
+ },
2814
+ {
2815
+ "options": 11,
2816
+ "choice": "the task is complete",
2817
+ "p": 0.9968240033922481
2818
+ }
2819
+ ]
2820
+ },
2821
+ {
2822
+ "app": "settings",
2823
+ "seed": 18,
2824
+ "variant": "marks+text",
2825
+ "task": "Turn on \"Show online status\" in the Profile settings and save the change.",
2826
+ "success": true,
2827
+ "goal_met": true,
2828
+ "stopped": true,
2829
+ "steps": 4,
2830
+ "trace": [
2831
+ {
2832
+ "options": 7,
2833
+ "choice": "click element 5: Profile (button)",
2834
+ "p": 0.9985919139831643
2835
+ },
2836
+ {
2837
+ "options": 11,
2838
+ "choice": "click element 7: switch",
2839
+ "p": 0.9986426020816851
2840
+ },
2841
+ {
2842
+ "options": 11,
2843
+ "choice": "click element 9: Save changes (button)",
2844
+ "p": 0.9983380512498816
2845
+ },
2846
+ {
2847
+ "options": 11,
2848
+ "choice": "the task is complete",
2849
+ "p": 0.9942144276734939
2850
+ }
2851
+ ]
2852
+ },
2853
+ {
2854
+ "app": "mail",
2855
+ "seed": 5,
2856
+ "variant": "marks",
2857
+ "task": "Archive the email from Grace Lee about the design review.",
2858
+ "success": false,
2859
+ "goal_met": false,
2860
+ "stopped": true,
2861
+ "steps": 4,
2862
+ "trace": [
2863
+ {
2864
+ "options": 10,
2865
+ "choice": "click element 9",
2866
+ "p": 0.8899010117753333
2867
+ },
2868
+ {
2869
+ "options": 9,
2870
+ "choice": "click element 6",
2871
+ "p": 0.8066999118688997
2872
+ },
2873
+ {
2874
+ "options": 9,
2875
+ "choice": "click element 8",
2876
+ "p": 0.9120995511344607
2877
+ },
2878
+ {
2879
+ "options": 9,
2880
+ "choice": "the task is complete",
2881
+ "p": 0.8881741550281158
2882
+ }
2883
+ ]
2884
+ },
2885
+ {
2886
+ "app": "mail",
2887
+ "seed": 11,
2888
+ "variant": "marks",
2889
+ "task": "Archive the email from Alice Chen about the quarterly report.",
2890
+ "success": false,
2891
+ "goal_met": false,
2892
+ "stopped": true,
2893
+ "steps": 4,
2894
+ "trace": [
2895
+ {
2896
+ "options": 10,
2897
+ "choice": "click element 5",
2898
+ "p": 0.9831324322046033
2899
+ },
2900
+ {
2901
+ "options": 9,
2902
+ "choice": "click element 6",
2903
+ "p": 0.8472186834319649
2904
+ },
2905
+ {
2906
+ "options": 9,
2907
+ "choice": "click element 5",
2908
+ "p": 0.5508126979678155
2909
+ },
2910
+ {
2911
+ "options": 9,
2912
+ "choice": "the task is complete",
2913
+ "p": 0.33350175447877733
2914
+ }
2915
+ ]
2916
+ },
2917
+ {
2918
+ "app": "mail",
2919
+ "seed": 17,
2920
+ "variant": "marks",
2921
+ "task": "Archive the email from Emma Rossi about the contract renewal.",
2922
+ "success": false,
2923
+ "goal_met": false,
2924
+ "stopped": true,
2925
+ "steps": 6,
2926
+ "trace": [
2927
+ {
2928
+ "options": 10,
2929
+ "choice": "click element 5",
2930
+ "p": 0.9822709369473034
2931
+ },
2932
+ {
2933
+ "options": 9,
2934
+ "choice": "click element 6",
2935
+ "p": 0.8238683071196944
2936
+ },
2937
+ {
2938
+ "options": 9,
2939
+ "choice": "click element 5",
2940
+ "p": 0.5910924206545309
2941
+ },
2942
+ {
2943
+ "options": 9,
2944
+ "choice": "click element 6",
2945
+ "p": 0.25963709770423166
2946
+ },
2947
+ {
2948
+ "options": 8,
2949
+ "choice": "click element 5",
2950
+ "p": 0.691520205330588
2951
+ },
2952
+ {
2953
+ "options": 9,
2954
+ "choice": "the task is complete",
2955
+ "p": 0.2882675288787037
2956
+ }
2957
+ ]
2958
+ },
2959
+ {
2960
+ "app": "shop",
2961
+ "seed": 3,
2962
+ "variant": "marks",
2963
+ "task": "Buy a red T-shirt in size XL: add it to the cart and check out.",
2964
+ "success": false,
2965
+ "goal_met": false,
2966
+ "stopped": true,
2967
+ "steps": 6,
2968
+ "trace": [
2969
+ {
2970
+ "options": 10,
2971
+ "choice": "click element 4",
2972
+ "p": 0.9375362542911772
2973
+ },
2974
+ {
2975
+ "options": 15,
2976
+ "choice": "click element 5",
2977
+ "p": 0.8732786746968813
2978
+ },
2979
+ {
2980
+ "options": 15,
2981
+ "choice": "click element 13",
2982
+ "p": 0.9800002435274163
2983
+ },
2984
+ {
2985
+ "options": 15,
2986
+ "choice": "click element 14",
2987
+ "p": 0.9969592123057359
2988
+ },
2989
+ {
2990
+ "options": 15,
2991
+ "choice": "click element 4",
2992
+ "p": 0.23922567319207153
2993
+ },
2994
+ {
2995
+ "options": 6,
2996
+ "choice": "the task is complete",
2997
+ "p": 0.45299021554689833
2998
+ }
2999
+ ]
3000
+ },
3001
+ {
3002
+ "app": "shop",
3003
+ "seed": 9,
3004
+ "variant": "marks",
3005
+ "task": "Buy a black Hoodie in size L: add it to the cart and check out.",
3006
+ "success": false,
3007
+ "goal_met": false,
3008
+ "stopped": true,
3009
+ "steps": 6,
3010
+ "trace": [
3011
+ {
3012
+ "options": 10,
3013
+ "choice": "click element 9",
3014
+ "p": 0.9741231718074939
3015
+ },
3016
+ {
3017
+ "options": 15,
3018
+ "choice": "click element 8",
3019
+ "p": 0.9961483976287776
3020
+ },
3021
+ {
3022
+ "options": 15,
3023
+ "choice": "click element 12",
3024
+ "p": 0.9567583973213926
3025
+ },
3026
+ {
3027
+ "options": 15,
3028
+ "choice": "click element 14",
3029
+ "p": 0.9915358825629454
3030
+ },
3031
+ {
3032
+ "options": 15,
3033
+ "choice": "click element 4",
3034
+ "p": 0.5907729898341846
3035
+ },
3036
+ {
3037
+ "options": 6,
3038
+ "choice": "the task is complete",
3039
+ "p": 0.9752848062801821
3040
+ }
3041
+ ]
3042
+ },
3043
+ {
3044
+ "app": "shop",
3045
+ "seed": 15,
3046
+ "variant": "marks",
3047
+ "task": "Buy a red Cap in size S: add it to the cart and check out.",
3048
+ "success": false,
3049
+ "goal_met": false,
3050
+ "stopped": true,
3051
+ "steps": 5,
3052
+ "trace": [
3053
+ {
3054
+ "options": 10,
3055
+ "choice": "click element 5",
3056
+ "p": 0.9630162879372378
3057
+ },
3058
+ {
3059
+ "options": 15,
3060
+ "choice": "click element 10",
3061
+ "p": 0.7675994858173997
3062
+ },
3063
+ {
3064
+ "options": 15,
3065
+ "choice": "click element 14",
3066
+ "p": 0.995774746171946
3067
+ },
3068
+ {
3069
+ "options": 15,
3070
+ "choice": "click element 4",
3071
+ "p": 0.42907766071206693
3072
+ },
3073
+ {
3074
+ "options": 6,
3075
+ "choice": "the task is complete",
3076
+ "p": 0.5277446542096556
3077
+ }
3078
+ ]
3079
+ },
3080
+ {
3081
+ "app": "settings",
3082
+ "seed": 1,
3083
+ "variant": "marks",
3084
+ "task": "Turn on \"Push notifications\" in the Notifications settings and save the change.",
3085
+ "success": true,
3086
+ "goal_met": true,
3087
+ "stopped": true,
3088
+ "steps": 4,
3089
+ "trace": [
3090
+ {
3091
+ "options": 7,
3092
+ "choice": "click element 3",
3093
+ "p": 0.9935602079112924
3094
+ },
3095
+ {
3096
+ "options": 11,
3097
+ "choice": "click element 8",
3098
+ "p": 0.9998454086687328
3099
+ },
3100
+ {
3101
+ "options": 11,
3102
+ "choice": "click element 9",
3103
+ "p": 0.9991662344761395
3104
+ },
3105
+ {
3106
+ "options": 11,
3107
+ "choice": "the task is complete",
3108
+ "p": 0.9738732252466481
3109
+ }
3110
+ ]
3111
+ },
3112
+ {
3113
+ "app": "settings",
3114
+ "seed": 7,
3115
+ "variant": "marks",
3116
+ "task": "Turn on \"Login alerts\" in the Security settings and save the change.",
3117
+ "success": true,
3118
+ "goal_met": true,
3119
+ "stopped": true,
3120
+ "steps": 4,
3121
+ "trace": [
3122
+ {
3123
+ "options": 7,
3124
+ "choice": "click element 3",
3125
+ "p": 0.9968102730943665
3126
+ },
3127
+ {
3128
+ "options": 11,
3129
+ "choice": "click element 8",
3130
+ "p": 0.999931918684158
3131
+ },
3132
+ {
3133
+ "options": 11,
3134
+ "choice": "click element 9",
3135
+ "p": 0.9998791925209146
3136
+ },
3137
+ {
3138
+ "options": 11,
3139
+ "choice": "the task is complete",
3140
+ "p": 0.9713008439318978
3141
+ }
3142
+ ]
3143
+ },
3144
+ {
3145
+ "app": "settings",
3146
+ "seed": 13,
3147
+ "variant": "marks",
3148
+ "task": "Turn on \"Login alerts\" in the Security settings and save the change.",
3149
+ "success": true,
3150
+ "goal_met": true,
3151
+ "stopped": true,
3152
+ "steps": 4,
3153
+ "trace": [
3154
+ {
3155
+ "options": 7,
3156
+ "choice": "click element 5",
3157
+ "p": 0.7522469616611945
3158
+ },
3159
+ {
3160
+ "options": 11,
3161
+ "choice": "click element 8",
3162
+ "p": 0.9997734280204899
3163
+ },
3164
+ {
3165
+ "options": 11,
3166
+ "choice": "click element 9",
3167
+ "p": 0.9997519934686436
3168
+ },
3169
+ {
3170
+ "options": 11,
3171
+ "choice": "the task is complete",
3172
+ "p": 0.9792125486750463
3173
+ }
3174
+ ]
3175
+ },
3176
+ {
3177
+ "app": "settings",
3178
+ "seed": 19,
3179
+ "variant": "marks",
3180
+ "task": "Turn on \"Two-factor authentication\" in the Security settings and save the change.",
3181
+ "success": true,
3182
+ "goal_met": true,
3183
+ "stopped": true,
3184
+ "steps": 4,
3185
+ "trace": [
3186
+ {
3187
+ "options": 7,
3188
+ "choice": "click element 5",
3189
+ "p": 0.944877843469037
3190
+ },
3191
+ {
3192
+ "options": 11,
3193
+ "choice": "click element 7",
3194
+ "p": 0.9991690520603501
3195
+ },
3196
+ {
3197
+ "options": 11,
3198
+ "choice": "click element 9",
3199
+ "p": 0.9966135123525435
3200
+ },
3201
+ {
3202
+ "options": 11,
3203
+ "choice": "the task is complete",
3204
+ "p": 0.9734410757252002
3205
+ }
3206
+ ]
3207
+ },
3208
+ {
3209
+ "app": "mail",
3210
+ "seed": 5,
3211
+ "variant": "marks+text",
3212
+ "task": "Archive the email from Grace Lee about the design review.",
3213
+ "success": true,
3214
+ "goal_met": true,
3215
+ "stopped": true,
3216
+ "steps": 3,
3217
+ "trace": [
3218
+ {
3219
+ "options": 10,
3220
+ "choice": "click element 8: Grace Lee design review (list row)",
3221
+ "p": 0.9999892675561525
3222
+ },
3223
+ {
3224
+ "options": 9,
3225
+ "choice": "click element 6: Archive (button)",
3226
+ "p": 0.9970689596587538
3227
+ },
3228
+ {
3229
+ "options": 9,
3230
+ "choice": "the task is complete",
3231
+ "p": 0.9726428273594905
3232
+ }
3233
+ ]
3234
+ },
3235
+ {
3236
+ "app": "mail",
3237
+ "seed": 11,
3238
+ "variant": "marks+text",
3239
+ "task": "Archive the email from Alice Chen about the quarterly report.",
3240
+ "success": true,
3241
+ "goal_met": true,
3242
+ "stopped": true,
3243
+ "steps": 3,
3244
+ "trace": [
3245
+ {
3246
+ "options": 10,
3247
+ "choice": "click element 4: Alice Chen quarterly report (list row)",
3248
+ "p": 0.9954630350124003
3249
+ },
3250
+ {
3251
+ "options": 9,
3252
+ "choice": "click element 6: Archive (button)",
3253
+ "p": 0.998339405107799
3254
+ },
3255
+ {
3256
+ "options": 9,
3257
+ "choice": "the task is complete",
3258
+ "p": 0.9735429892314412
3259
+ }
3260
+ ]
3261
+ },
3262
+ {
3263
+ "app": "mail",
3264
+ "seed": 17,
3265
+ "variant": "marks+text",
3266
+ "task": "Archive the email from Emma Rossi about the contract renewal.",
3267
+ "success": true,
3268
+ "goal_met": true,
3269
+ "stopped": true,
3270
+ "steps": 3,
3271
+ "trace": [
3272
+ {
3273
+ "options": 10,
3274
+ "choice": "click element 4: Emma Rossi contract renewal (list row)",
3275
+ "p": 0.9936438299265863
3276
+ },
3277
+ {
3278
+ "options": 9,
3279
+ "choice": "click element 6: Archive (button)",
3280
+ "p": 0.9989676992732754
3281
+ },
3282
+ {
3283
+ "options": 9,
3284
+ "choice": "the task is complete",
3285
+ "p": 0.977145677973397
3286
+ }
3287
+ ]
3288
+ },
3289
+ {
3290
+ "app": "shop",
3291
+ "seed": 3,
3292
+ "variant": "marks+text",
3293
+ "task": "Buy a red T-shirt in size XL: add it to the cart and check out.",
3294
+ "success": true,
3295
+ "goal_met": true,
3296
+ "stopped": true,
3297
+ "steps": 7,
3298
+ "trace": [
3299
+ {
3300
+ "options": 10,
3301
+ "choice": "click element 4: T-shirt $19 (product card)",
3302
+ "p": 0.9724649897236056
3303
+ },
3304
+ {
3305
+ "options": 15,
3306
+ "choice": "click element 5: colour swatch",
3307
+ "p": 0.8201496116233312
3308
+ },
3309
+ {
3310
+ "options": 15,
3311
+ "choice": "click element 13: XL (button)",
3312
+ "p": 0.9945416373501986
3313
+ },
3314
+ {
3315
+ "options": 15,
3316
+ "choice": "click element 14: Add to cart (button)",
3317
+ "p": 0.9990969758466067
3318
+ },
3319
+ {
3320
+ "options": 15,
3321
+ "choice": "click element 4: Cart (1) (button)",
3322
+ "p": 0.8309152366189686
3323
+ },
3324
+ {
3325
+ "options": 6,
3326
+ "choice": "click element 4: Checkout (button)",
3327
+ "p": 0.9487986537578865
3328
+ },
3329
+ {
3330
+ "options": 6,
3331
+ "choice": "the task is complete",
3332
+ "p": 0.9818605565008113
3333
+ }
3334
+ ]
3335
+ },
3336
+ {
3337
+ "app": "shop",
3338
+ "seed": 9,
3339
+ "variant": "marks+text",
3340
+ "task": "Buy a black Hoodie in size L: add it to the cart and check out.",
3341
+ "success": true,
3342
+ "goal_met": true,
3343
+ "stopped": true,
3344
+ "steps": 7,
3345
+ "trace": [
3346
+ {
3347
+ "options": 10,
3348
+ "choice": "click element 9: Hoodie $54 (product card)",
3349
+ "p": 0.9960404028285044
3350
+ },
3351
+ {
3352
+ "options": 15,
3353
+ "choice": "click element 8: colour swatch",
3354
+ "p": 0.9805375054980668
3355
+ },
3356
+ {
3357
+ "options": 15,
3358
+ "choice": "click element 12: L (button)",
3359
+ "p": 0.9837963454968081
3360
+ },
3361
+ {
3362
+ "options": 15,
3363
+ "choice": "click element 14: Add to cart (button)",
3364
+ "p": 0.9976957181828331
3365
+ },
3366
+ {
3367
+ "options": 15,
3368
+ "choice": "click element 4: Cart (1) (button)",
3369
+ "p": 0.9195499572559499
3370
+ },
3371
+ {
3372
+ "options": 6,
3373
+ "choice": "click element 4: Checkout (button)",
3374
+ "p": 0.9129357820426369
3375
+ },
3376
+ {
3377
+ "options": 6,
3378
+ "choice": "the task is complete",
3379
+ "p": 0.9870940040069339
3380
+ }
3381
+ ]
3382
+ },
3383
+ {
3384
+ "app": "shop",
3385
+ "seed": 15,
3386
+ "variant": "marks+text",
3387
+ "task": "Buy a red Cap in size S: add it to the cart and check out.",
3388
+ "success": false,
3389
+ "goal_met": false,
3390
+ "stopped": true,
3391
+ "steps": 6,
3392
+ "trace": [
3393
+ {
3394
+ "options": 10,
3395
+ "choice": "click element 5: Cap $26 (product card)",
3396
+ "p": 0.9859670899502154
3397
+ },
3398
+ {
3399
+ "options": 15,
3400
+ "choice": "click element 10: S (button)",
3401
+ "p": 0.678428694655598
3402
+ },
3403
+ {
3404
+ "options": 15,
3405
+ "choice": "click element 14: Add to cart (button)",
3406
+ "p": 0.9989641213642733
3407
+ },
3408
+ {
3409
+ "options": 15,
3410
+ "choice": "click element 4: Cart (0) (button)",
3411
+ "p": 0.9007504654413979
3412
+ },
3413
+ {
3414
+ "options": 6,
3415
+ "choice": "click element 4: Checkout (button)",
3416
+ "p": 0.7522623624428733
3417
+ },
3418
+ {
3419
+ "options": 6,
3420
+ "choice": "the task is complete",
3421
+ "p": 0.6835069943315197
3422
+ }
3423
+ ]
3424
+ },
3425
+ {
3426
+ "app": "settings",
3427
+ "seed": 1,
3428
+ "variant": "marks+text",
3429
+ "task": "Turn on \"Push notifications\" in the Notifications settings and save the change.",
3430
+ "success": true,
3431
+ "goal_met": true,
3432
+ "stopped": true,
3433
+ "steps": 4,
3434
+ "trace": [
3435
+ {
3436
+ "options": 7,
3437
+ "choice": "click element 3: Notifications (button)",
3438
+ "p": 0.9999545186659321
3439
+ },
3440
+ {
3441
+ "options": 11,
3442
+ "choice": "click element 8: switch",
3443
+ "p": 0.9999664376093863
3444
+ },
3445
+ {
3446
+ "options": 11,
3447
+ "choice": "click element 9: Save changes (button)",
3448
+ "p": 0.9996863951619318
3449
+ },
3450
+ {
3451
+ "options": 11,
3452
+ "choice": "the task is complete",
3453
+ "p": 0.9969804824838281
3454
+ }
3455
+ ]
3456
+ },
3457
+ {
3458
+ "app": "settings",
3459
+ "seed": 7,
3460
+ "variant": "marks+text",
3461
+ "task": "Turn on \"Login alerts\" in the Security settings and save the change.",
3462
+ "success": true,
3463
+ "goal_met": true,
3464
+ "stopped": true,
3465
+ "steps": 4,
3466
+ "trace": [
3467
+ {
3468
+ "options": 7,
3469
+ "choice": "click element 3: Security (button)",
3470
+ "p": 0.9929287820354508
3471
+ },
3472
+ {
3473
+ "options": 11,
3474
+ "choice": "click element 8: switch",
3475
+ "p": 0.9999838057845707
3476
+ },
3477
+ {
3478
+ "options": 11,
3479
+ "choice": "click element 9: Save changes (button)",
3480
+ "p": 0.9999077276071512
3481
+ },
3482
+ {
3483
+ "options": 11,
3484
+ "choice": "the task is complete",
3485
+ "p": 0.9956865815692123
3486
+ }
3487
+ ]
3488
+ },
3489
+ {
3490
+ "app": "settings",
3491
+ "seed": 13,
3492
+ "variant": "marks+text",
3493
+ "task": "Turn on \"Login alerts\" in the Security settings and save the change.",
3494
+ "success": true,
3495
+ "goal_met": true,
3496
+ "stopped": true,
3497
+ "steps": 4,
3498
+ "trace": [
3499
+ {
3500
+ "options": 7,
3501
+ "choice": "click element 5: Security (button)",
3502
+ "p": 0.9994821599022279
3503
+ },
3504
+ {
3505
+ "options": 11,
3506
+ "choice": "click element 8: switch",
3507
+ "p": 0.9999815488842919
3508
+ },
3509
+ {
3510
+ "options": 11,
3511
+ "choice": "click element 9: Save changes (button)",
3512
+ "p": 0.999727732069915
3513
+ },
3514
+ {
3515
+ "options": 11,
3516
+ "choice": "the task is complete",
3517
+ "p": 0.9943509873223676
3518
+ }
3519
+ ]
3520
+ },
3521
+ {
3522
+ "app": "settings",
3523
+ "seed": 19,
3524
+ "variant": "marks+text",
3525
+ "task": "Turn on \"Two-factor authentication\" in the Security settings and save the change.",
3526
+ "success": true,
3527
+ "goal_met": true,
3528
+ "stopped": true,
3529
+ "steps": 4,
3530
+ "trace": [
3531
+ {
3532
+ "options": 7,
3533
+ "choice": "click element 5: Security (button)",
3534
+ "p": 0.999270936232728
3535
+ },
3536
+ {
3537
+ "options": 11,
3538
+ "choice": "click element 7: switch",
3539
+ "p": 0.9984709902613281
3540
+ },
3541
+ {
3542
+ "options": 11,
3543
+ "choice": "click element 9: Save changes (button)",
3544
+ "p": 0.9994291102790471
3545
+ },
3546
+ {
3547
+ "options": 11,
3548
+ "choice": "the task is complete",
3549
+ "p": 0.9937489113161283
3550
+ }
3551
+ ]
3552
+ },
3553
+ {
3554
+ "app": "mail",
3555
+ "seed": 6,
3556
+ "variant": "marks",
3557
+ "task": "Archive the email from Hugo Weber about the contract renewal.",
3558
+ "success": false,
3559
+ "goal_met": false,
3560
+ "stopped": true,
3561
+ "steps": 3,
3562
+ "trace": [
3563
+ {
3564
+ "options": 10,
3565
+ "choice": "click element 8",
3566
+ "p": 0.9977978895807249
3567
+ },
3568
+ {
3569
+ "options": 9,
3570
+ "choice": "click element 6",
3571
+ "p": 0.8082693171210587
3572
+ },
3573
+ {
3574
+ "options": 9,
3575
+ "choice": "the task is complete",
3576
+ "p": 0.45693151981236835
3577
+ }
3578
+ ]
3579
+ },
3580
+ {
3581
+ "app": "mail",
3582
+ "seed": 12,
3583
+ "variant": "marks",
3584
+ "task": "Archive the email from David Kim about the lunch on Friday.",
3585
+ "success": true,
3586
+ "goal_met": true,
3587
+ "stopped": true,
3588
+ "steps": 3,
3589
+ "trace": [
3590
+ {
3591
+ "options": 10,
3592
+ "choice": "click element 9",
3593
+ "p": 0.9952809633559083
3594
+ },
3595
+ {
3596
+ "options": 9,
3597
+ "choice": "click element 6",
3598
+ "p": 0.9889362018121068
3599
+ },
3600
+ {
3601
+ "options": 9,
3602
+ "choice": "the task is complete",
3603
+ "p": 0.42661916179808373
3604
+ }
3605
+ ]
3606
+ },
3607
+ {
3608
+ "app": "mail",
3609
+ "seed": 18,
3610
+ "variant": "marks",
3611
+ "task": "Archive the email from Hugo Weber about the team offsite.",
3612
+ "success": false,
3613
+ "goal_met": false,
3614
+ "stopped": true,
3615
+ "steps": 4,
3616
+ "trace": [
3617
+ {
3618
+ "options": 10,
3619
+ "choice": "click element 5",
3620
+ "p": 0.9854551874369589
3621
+ },
3622
+ {
3623
+ "options": 9,
3624
+ "choice": "click element 6",
3625
+ "p": 0.8196608231025655
3626
+ },
3627
+ {
3628
+ "options": 9,
3629
+ "choice": "click element 5",
3630
+ "p": 0.5450367894211691
3631
+ },
3632
+ {
3633
+ "options": 9,
3634
+ "choice": "the task is complete",
3635
+ "p": 0.32470083263231264
3636
+ }
3637
+ ]
3638
+ },
3639
+ {
3640
+ "app": "shop",
3641
+ "seed": 4,
3642
+ "variant": "marks",
3643
+ "task": "Buy a yellow T-shirt in size L: add it to the cart and check out.",
3644
+ "success": false,
3645
+ "goal_met": false,
3646
+ "stopped": true,
3647
+ "steps": 6,
3648
+ "trace": [
3649
+ {
3650
+ "options": 10,
3651
+ "choice": "click element 5",
3652
+ "p": 0.8980278245305588
3653
+ },
3654
+ {
3655
+ "options": 15,
3656
+ "choice": "click element 9",
3657
+ "p": 0.9875382834747788
3658
+ },
3659
+ {
3660
+ "options": 15,
3661
+ "choice": "click element 12",
3662
+ "p": 0.8198989186225369
3663
+ },
3664
+ {
3665
+ "options": 15,
3666
+ "choice": "click element 14",
3667
+ "p": 0.9785015781736833
3668
+ },
3669
+ {
3670
+ "options": 15,
3671
+ "choice": "click element 4",
3672
+ "p": 0.5233256744813881
3673
+ },
3674
+ {
3675
+ "options": 6,
3676
+ "choice": "the task is complete",
3677
+ "p": 0.908995003138921
3678
+ }
3679
+ ]
3680
+ },
3681
+ {
3682
+ "app": "shop",
3683
+ "seed": 10,
3684
+ "variant": "marks",
3685
+ "task": "Buy a green Hoodie in size M: add it to the cart and check out.",
3686
+ "success": false,
3687
+ "goal_met": false,
3688
+ "stopped": true,
3689
+ "steps": 6,
3690
+ "trace": [
3691
+ {
3692
+ "options": 10,
3693
+ "choice": "click element 5",
3694
+ "p": 0.9244280841330427
3695
+ },
3696
+ {
3697
+ "options": 15,
3698
+ "choice": "click element 7",
3699
+ "p": 0.9727558040316957
3700
+ },
3701
+ {
3702
+ "options": 15,
3703
+ "choice": "click element 11",
3704
+ "p": 0.7583926622129231
3705
+ },
3706
+ {
3707
+ "options": 15,
3708
+ "choice": "click element 14",
3709
+ "p": 0.9546084846018491
3710
+ },
3711
+ {
3712
+ "options": 15,
3713
+ "choice": "click element 4",
3714
+ "p": 0.5560089136897925
3715
+ },
3716
+ {
3717
+ "options": 6,
3718
+ "choice": "the task is complete",
3719
+ "p": 0.9117279923105113
3720
+ }
3721
+ ]
3722
+ },
3723
+ {
3724
+ "app": "shop",
3725
+ "seed": 16,
3726
+ "variant": "marks",
3727
+ "task": "Buy a red Sneakers in size S: add it to the cart and check out.",
3728
+ "success": false,
3729
+ "goal_met": false,
3730
+ "stopped": true,
3731
+ "steps": 6,
3732
+ "trace": [
3733
+ {
3734
+ "options": 10,
3735
+ "choice": "click element 8",
3736
+ "p": 0.9890552637127913
3737
+ },
3738
+ {
3739
+ "options": 15,
3740
+ "choice": "click element 5",
3741
+ "p": 0.9037842078472037
3742
+ },
3743
+ {
3744
+ "options": 15,
3745
+ "choice": "click element 10",
3746
+ "p": 0.8556609428632328
3747
+ },
3748
+ {
3749
+ "options": 15,
3750
+ "choice": "click element 14",
3751
+ "p": 0.9877794279325114
3752
+ },
3753
+ {
3754
+ "options": 15,
3755
+ "choice": "click element 4",
3756
+ "p": 0.5052077997295938
3757
+ },
3758
+ {
3759
+ "options": 6,
3760
+ "choice": "the task is complete",
3761
+ "p": 0.9456430542992879
3762
+ }
3763
+ ]
3764
+ },
3765
+ {
3766
+ "app": "settings",
3767
+ "seed": 2,
3768
+ "variant": "marks",
3769
+ "task": "Turn on \"Compact layout\" in the Appearance settings and save the change.",
3770
+ "success": true,
3771
+ "goal_met": true,
3772
+ "stopped": true,
3773
+ "steps": 4,
3774
+ "trace": [
3775
+ {
3776
+ "options": 7,
3777
+ "choice": "click element 6",
3778
+ "p": 0.9958116241350685
3779
+ },
3780
+ {
3781
+ "options": 11,
3782
+ "choice": "click element 8",
3783
+ "p": 0.9998879190405711
3784
+ },
3785
+ {
3786
+ "options": 11,
3787
+ "choice": "click element 9",
3788
+ "p": 0.9999763273129455
3789
+ },
3790
+ {
3791
+ "options": 11,
3792
+ "choice": "the task is complete",
3793
+ "p": 0.9916864915655396
3794
+ }
3795
+ ]
3796
+ },
3797
+ {
3798
+ "app": "settings",
3799
+ "seed": 8,
3800
+ "variant": "marks",
3801
+ "task": "Turn on \"Compact layout\" in the Appearance settings and save the change.",
3802
+ "success": true,
3803
+ "goal_met": true,
3804
+ "stopped": true,
3805
+ "steps": 5,
3806
+ "trace": [
3807
+ {
3808
+ "options": 7,
3809
+ "choice": "click element 5",
3810
+ "p": 0.6064446058191004
3811
+ },
3812
+ {
3813
+ "options": 11,
3814
+ "choice": "click element 4",
3815
+ "p": 0.4973635699651363
3816
+ },
3817
+ {
3818
+ "options": 11,
3819
+ "choice": "click element 8",
3820
+ "p": 0.9994703422234483
3821
+ },
3822
+ {
3823
+ "options": 11,
3824
+ "choice": "click element 9",
3825
+ "p": 0.9988544104560748
3826
+ },
3827
+ {
3828
+ "options": 11,
3829
+ "choice": "the task is complete",
3830
+ "p": 0.9415067293532886
3831
+ }
3832
+ ]
3833
+ },
3834
+ {
3835
+ "app": "settings",
3836
+ "seed": 14,
3837
+ "variant": "marks",
3838
+ "task": "Turn on \"Dark mode\" in the Appearance settings and save the change.",
3839
+ "success": true,
3840
+ "goal_met": true,
3841
+ "stopped": true,
3842
+ "steps": 4,
3843
+ "trace": [
3844
+ {
3845
+ "options": 7,
3846
+ "choice": "click element 4",
3847
+ "p": 0.9856281316967483
3848
+ },
3849
+ {
3850
+ "options": 11,
3851
+ "choice": "click element 7",
3852
+ "p": 0.9994869791292652
3853
+ },
3854
+ {
3855
+ "options": 11,
3856
+ "choice": "click element 9",
3857
+ "p": 0.9998337668981889
3858
+ },
3859
+ {
3860
+ "options": 11,
3861
+ "choice": "the task is complete",
3862
+ "p": 0.9792926559207835
3863
+ }
3864
+ ]
3865
+ },
3866
+ {
3867
+ "app": "settings",
3868
+ "seed": 20,
3869
+ "variant": "marks",
3870
+ "task": "Turn on \"Two-factor authentication\" in the Security settings and save the change.",
3871
+ "success": true,
3872
+ "goal_met": true,
3873
+ "stopped": true,
3874
+ "steps": 4,
3875
+ "trace": [
3876
+ {
3877
+ "options": 7,
3878
+ "choice": "click element 6",
3879
+ "p": 0.9990456531762184
3880
+ },
3881
+ {
3882
+ "options": 11,
3883
+ "choice": "click element 7",
3884
+ "p": 0.9997513970436877
3885
+ },
3886
+ {
3887
+ "options": 11,
3888
+ "choice": "click element 9",
3889
+ "p": 0.996187671253891
3890
+ },
3891
+ {
3892
+ "options": 11,
3893
+ "choice": "the task is complete",
3894
+ "p": 0.9741278562628871
3895
+ }
3896
+ ]
3897
+ },
3898
+ {
3899
+ "app": "mail",
3900
+ "seed": 6,
3901
+ "variant": "marks+text",
3902
+ "task": "Archive the email from Hugo Weber about the contract renewal.",
3903
+ "success": true,
3904
+ "goal_met": true,
3905
+ "stopped": true,
3906
+ "steps": 3,
3907
+ "trace": [
3908
+ {
3909
+ "options": 10,
3910
+ "choice": "click element 7: Hugo Weber contract renewal (list row)",
3911
+ "p": 0.9997271606212903
3912
+ },
3913
+ {
3914
+ "options": 9,
3915
+ "choice": "click element 6: Archive (button)",
3916
+ "p": 0.9907772289056817
3917
+ },
3918
+ {
3919
+ "options": 9,
3920
+ "choice": "the task is complete",
3921
+ "p": 0.9864732674108846
3922
+ }
3923
+ ]
3924
+ },
3925
+ {
3926
+ "app": "mail",
3927
+ "seed": 12,
3928
+ "variant": "marks+text",
3929
+ "task": "Archive the email from David Kim about the lunch on Friday.",
3930
+ "success": true,
3931
+ "goal_met": true,
3932
+ "stopped": true,
3933
+ "steps": 3,
3934
+ "trace": [
3935
+ {
3936
+ "options": 10,
3937
+ "choice": "click element 9: David Kim lunch on Friday (list row)",
3938
+ "p": 0.9999193858698756
3939
+ },
3940
+ {
3941
+ "options": 9,
3942
+ "choice": "click element 6: Archive (button)",
3943
+ "p": 0.9999953339070028
3944
+ },
3945
+ {
3946
+ "options": 9,
3947
+ "choice": "the task is complete",
3948
+ "p": 0.9950677004298637
3949
+ }
3950
+ ]
3951
+ },
3952
+ {
3953
+ "app": "mail",
3954
+ "seed": 18,
3955
+ "variant": "marks+text",
3956
+ "task": "Archive the email from Hugo Weber about the team offsite.",
3957
+ "success": true,
3958
+ "goal_met": true,
3959
+ "stopped": true,
3960
+ "steps": 3,
3961
+ "trace": [
3962
+ {
3963
+ "options": 10,
3964
+ "choice": "click element 4: Hugo Weber team offsite (list row)",
3965
+ "p": 0.994082157941521
3966
+ },
3967
+ {
3968
+ "options": 9,
3969
+ "choice": "click element 6: Archive (button)",
3970
+ "p": 0.9973919332879417
3971
+ },
3972
+ {
3973
+ "options": 9,
3974
+ "choice": "the task is complete",
3975
+ "p": 0.9802493081590956
3976
+ }
3977
+ ]
3978
+ },
3979
+ {
3980
+ "app": "shop",
3981
+ "seed": 4,
3982
+ "variant": "marks+text",
3983
+ "task": "Buy a yellow T-shirt in size L: add it to the cart and check out.",
3984
+ "success": true,
3985
+ "goal_met": true,
3986
+ "stopped": true,
3987
+ "steps": 7,
3988
+ "trace": [
3989
+ {
3990
+ "options": 10,
3991
+ "choice": "click element 5: T-shirt $26 (product card)",
3992
+ "p": 0.9418415900533565
3993
+ },
3994
+ {
3995
+ "options": 15,
3996
+ "choice": "click element 9: colour swatch",
3997
+ "p": 0.9671758878869959
3998
+ },
3999
+ {
4000
+ "options": 15,
4001
+ "choice": "click element 12: L (button)",
4002
+ "p": 0.9775716810435897
4003
+ },
4004
+ {
4005
+ "options": 15,
4006
+ "choice": "click element 14: Add to cart (button)",
4007
+ "p": 0.998624901141156
4008
+ },
4009
+ {
4010
+ "options": 15,
4011
+ "choice": "click element 4: Cart (1) (button)",
4012
+ "p": 0.9090881901569096
4013
+ },
4014
+ {
4015
+ "options": 6,
4016
+ "choice": "click element 4: Checkout (button)",
4017
+ "p": 0.8903531049173349
4018
+ },
4019
+ {
4020
+ "options": 6,
4021
+ "choice": "the task is complete",
4022
+ "p": 0.9829689527867387
4023
+ }
4024
+ ]
4025
+ },
4026
+ {
4027
+ "app": "shop",
4028
+ "seed": 10,
4029
+ "variant": "marks+text",
4030
+ "task": "Buy a green Hoodie in size M: add it to the cart and check out.",
4031
+ "success": true,
4032
+ "goal_met": true,
4033
+ "stopped": true,
4034
+ "steps": 7,
4035
+ "trace": [
4036
+ {
4037
+ "options": 10,
4038
+ "choice": "click element 5: Hoodie $26 (product card)",
4039
+ "p": 0.9642140622495479
4040
+ },
4041
+ {
4042
+ "options": 15,
4043
+ "choice": "click element 7: colour swatch",
4044
+ "p": 0.9063349717362331
4045
+ },
4046
+ {
4047
+ "options": 15,
4048
+ "choice": "click element 11: M (button)",
4049
+ "p": 0.978193066020716
4050
+ },
4051
+ {
4052
+ "options": 15,
4053
+ "choice": "click element 14: Add to cart (button)",
4054
+ "p": 0.9979707448580522
4055
+ },
4056
+ {
4057
+ "options": 15,
4058
+ "choice": "click element 4: Cart (1) (button)",
4059
+ "p": 0.919863573092946
4060
+ },
4061
+ {
4062
+ "options": 6,
4063
+ "choice": "click element 4: Checkout (button)",
4064
+ "p": 0.9456619224631192
4065
+ },
4066
+ {
4067
+ "options": 6,
4068
+ "choice": "the task is complete",
4069
+ "p": 0.986074816399851
4070
+ }
4071
+ ]
4072
+ },
4073
+ {
4074
+ "app": "shop",
4075
+ "seed": 16,
4076
+ "variant": "marks+text",
4077
+ "task": "Buy a red Sneakers in size S: add it to the cart and check out.",
4078
+ "success": true,
4079
+ "goal_met": true,
4080
+ "stopped": true,
4081
+ "steps": 7,
4082
+ "trace": [
4083
+ {
4084
+ "options": 10,
4085
+ "choice": "click element 8: Sneakers $47 (product card)",
4086
+ "p": 0.997407808424899
4087
+ },
4088
+ {
4089
+ "options": 15,
4090
+ "choice": "click element 5: colour swatch",
4091
+ "p": 0.8169221410542152
4092
+ },
4093
+ {
4094
+ "options": 15,
4095
+ "choice": "click element 10: S (button)",
4096
+ "p": 0.9634893782412659
4097
+ },
4098
+ {
4099
+ "options": 15,
4100
+ "choice": "click element 14: Add to cart (button)",
4101
+ "p": 0.996928313018705
4102
+ },
4103
+ {
4104
+ "options": 15,
4105
+ "choice": "click element 4: Cart (1) (button)",
4106
+ "p": 0.9269748557765952
4107
+ },
4108
+ {
4109
+ "options": 6,
4110
+ "choice": "click element 4: Checkout (button)",
4111
+ "p": 0.9738908331240022
4112
+ },
4113
+ {
4114
+ "options": 6,
4115
+ "choice": "the task is complete",
4116
+ "p": 0.9879875474980592
4117
+ }
4118
+ ]
4119
+ },
4120
+ {
4121
+ "app": "settings",
4122
+ "seed": 2,
4123
+ "variant": "marks+text",
4124
+ "task": "Turn on \"Compact layout\" in the Appearance settings and save the change.",
4125
+ "success": true,
4126
+ "goal_met": true,
4127
+ "stopped": true,
4128
+ "steps": 4,
4129
+ "trace": [
4130
+ {
4131
+ "options": 7,
4132
+ "choice": "click element 6: Appearance (button)",
4133
+ "p": 0.9999937106905179
4134
+ },
4135
+ {
4136
+ "options": 11,
4137
+ "choice": "click element 8: switch",
4138
+ "p": 0.999960651534345
4139
+ },
4140
+ {
4141
+ "options": 11,
4142
+ "choice": "click element 9: Save changes (button)",
4143
+ "p": 0.9999929753885327
4144
+ },
4145
+ {
4146
+ "options": 11,
4147
+ "choice": "the task is complete",
4148
+ "p": 0.9978669216983815
4149
+ }
4150
+ ]
4151
+ },
4152
+ {
4153
+ "app": "settings",
4154
+ "seed": 8,
4155
+ "variant": "marks+text",
4156
+ "task": "Turn on \"Compact layout\" in the Appearance settings and save the change.",
4157
+ "success": true,
4158
+ "goal_met": true,
4159
+ "stopped": true,
4160
+ "steps": 4,
4161
+ "trace": [
4162
+ {
4163
+ "options": 7,
4164
+ "choice": "click element 4: Appearance (button)",
4165
+ "p": 0.9999720633741155
4166
+ },
4167
+ {
4168
+ "options": 11,
4169
+ "choice": "click element 8: switch",
4170
+ "p": 0.999977755226722
4171
+ },
4172
+ {
4173
+ "options": 11,
4174
+ "choice": "click element 9: Save changes (button)",
4175
+ "p": 0.9999893245512457
4176
+ },
4177
+ {
4178
+ "options": 11,
4179
+ "choice": "the task is complete",
4180
+ "p": 0.9965392611219008
4181
+ }
4182
+ ]
4183
+ },
4184
+ {
4185
+ "app": "settings",
4186
+ "seed": 14,
4187
+ "variant": "marks+text",
4188
+ "task": "Turn on \"Dark mode\" in the Appearance settings and save the change.",
4189
+ "success": true,
4190
+ "goal_met": true,
4191
+ "stopped": true,
4192
+ "steps": 4,
4193
+ "trace": [
4194
+ {
4195
+ "options": 7,
4196
+ "choice": "click element 4: Appearance (button)",
4197
+ "p": 0.999979186898143
4198
+ },
4199
+ {
4200
+ "options": 11,
4201
+ "choice": "click element 7: switch",
4202
+ "p": 0.9983935140517022
4203
+ },
4204
+ {
4205
+ "options": 11,
4206
+ "choice": "click element 9: Save changes (button)",
4207
+ "p": 0.9999970728455576
4208
+ },
4209
+ {
4210
+ "options": 11,
4211
+ "choice": "the task is complete",
4212
+ "p": 0.9968614361821789
4213
+ }
4214
+ ]
4215
+ },
4216
+ {
4217
+ "app": "settings",
4218
+ "seed": 20,
4219
+ "variant": "marks+text",
4220
+ "task": "Turn on \"Two-factor authentication\" in the Security settings and save the change.",
4221
+ "success": true,
4222
+ "goal_met": true,
4223
+ "stopped": true,
4224
+ "steps": 4,
4225
+ "trace": [
4226
+ {
4227
+ "options": 7,
4228
+ "choice": "click element 6: Security (button)",
4229
+ "p": 0.9999686007458691
4230
+ },
4231
+ {
4232
+ "options": 11,
4233
+ "choice": "click element 7: switch",
4234
+ "p": 0.9989842666774184
4235
+ },
4236
+ {
4237
+ "options": 11,
4238
+ "choice": "click element 9: Save changes (button)",
4239
+ "p": 0.9994442811408799
4240
+ },
4241
+ {
4242
+ "options": 11,
4243
+ "choice": "the task is complete",
4244
+ "p": 0.9951361449249908
4245
+ }
4246
+ ]
4247
+ }
4248
+ ]
4249
+ }
reports/vl/microlens.json ADDED
@@ -0,0 +1,71 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metrics": {
3
+ "random": [
4
+ 0.48894736842105274,
5
+ 0.04,
6
+ 0.205,
7
+ 0.21949352013581572,
8
+ 0.16814529929564767
9
+ ],
10
+ "pop": [
11
+ 0.5047368421052628,
12
+ 0.04,
13
+ 0.23,
14
+ 0.2040320275478807,
15
+ 0.16545971297364492
16
+ ],
17
+ "title_sim": [
18
+ 0.6015789473684214,
19
+ 0.205,
20
+ 0.385,
21
+ 0.3667476255083533,
22
+ 0.32769800815369743
23
+ ],
24
+ "jev_titles": [
25
+ 0.6171052631578946,
26
+ 0.155,
27
+ 0.425,
28
+ 0.36171880061708434,
29
+ 0.3013690903180073
30
+ ],
31
+ "jev_images": [
32
+ 0.7210526315789467,
33
+ 0.255,
34
+ 0.6,
35
+ 0.48104892350240086,
36
+ 0.4064158857783086
37
+ ],
38
+ "jev_both": [
39
+ 0.7149999999999997,
40
+ 0.245,
41
+ 0.54,
42
+ 0.47049149692806513,
43
+ 0.3986715261862323
44
+ ],
45
+ "itemcf": [
46
+ 0.727631578947368,
47
+ 0.355,
48
+ 0.49,
49
+ 0.5031604435662497,
50
+ 0.4600847555222559
51
+ ]
52
+ },
53
+ "bootstrap_auc": {
54
+ "jev_images-jev_titles": [
55
+ 0.10394736842105262,
56
+ 0.05315789473684209,
57
+ 0.15473684210526314
58
+ ],
59
+ "jev_images-title_sim": [
60
+ 0.1194736842105263,
61
+ 0.06789473684210526,
62
+ 0.1694736842105263
63
+ ],
64
+ "jev_images-itemcf": [
65
+ -0.0065789473684210575,
66
+ -0.048947368421052635,
67
+ 0.035789473684210524
68
+ ]
69
+ },
70
+ "n_users": 200
71
+ }
reports/vl/mmrb2.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "t2i": {
3
+ "n": 1000,
4
+ "accuracy": 0.65,
5
+ "order_agreement": 0.637,
6
+ "seconds": 90.77313542366028
7
+ },
8
+ "edit": {
9
+ "n": 1000,
10
+ "accuracy": 0.585,
11
+ "order_agreement": 0.342,
12
+ "seconds": 122.78884029388428
13
+ },
14
+ "interleaved": {
15
+ "n": 1000,
16
+ "accuracy": 0.622,
17
+ "order_agreement": 0.663,
18
+ "seconds": 151.45967888832092
19
+ },
20
+ "reasoning": {
21
+ "n": 1000,
22
+ "accuracy": 0.58,
23
+ "order_agreement": 0.593,
24
+ "seconds": 94.65101504325867
25
+ }
26
+ }
reports/vl/quickdraw.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "n": 960,
3
+ "top1_by_stroke_fraction": {
4
+ "0.3": 0.371875,
5
+ "0.6": 0.525,
6
+ "1.0": 0.7625
7
+ }
8
+ }
reports/vl/robot_arm.json ADDED
@@ -0,0 +1,1464 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "summary": {
3
+ "episodes": 20,
4
+ "success": 0.5,
5
+ "grasped": 0.5,
6
+ "mean_decisions": 61.6,
7
+ "median_ms": 163.21325302124023,
8
+ "seconds": 277.3504137992859
9
+ },
10
+ "episodes": [
11
+ {
12
+ "seed": 1,
13
+ "target": "red",
14
+ "grasped": true,
15
+ "success": true,
16
+ "decisions": 28,
17
+ "latency_s": [
18
+ 0.16429615020751953,
19
+ 0.15776443481445312,
20
+ 0.15830683708190918,
21
+ 0.1535944938659668,
22
+ 0.15360260009765625,
23
+ 0.14891552925109863,
24
+ 0.15513896942138672,
25
+ 0.1583404541015625,
26
+ 0.1505415439605713,
27
+ 0.15151500701904297,
28
+ 0.15295839309692383,
29
+ 0.15436100959777832,
30
+ 0.14976859092712402,
31
+ 0.16703224182128906,
32
+ 0.15848612785339355,
33
+ 0.15624570846557617,
34
+ 0.16089630126953125,
35
+ 0.15336155891418457,
36
+ 0.15105843544006348,
37
+ 0.1510176658630371,
38
+ 0.14844012260437012,
39
+ 0.15520524978637695,
40
+ 0.15081167221069336,
41
+ 0.15312910079956055,
42
+ 0.15756893157958984,
43
+ 0.15378618240356445,
44
+ 0.1525251865386963,
45
+ 0.15658211708068848
46
+ ],
47
+ "err_cube_cm": 1.1,
48
+ "err_tray_cm": 2.1
49
+ },
50
+ {
51
+ "seed": 2,
52
+ "target": "blue",
53
+ "grasped": true,
54
+ "success": true,
55
+ "decisions": 48,
56
+ "latency_s": [
57
+ 0.15256929397583008,
58
+ 0.15643525123596191,
59
+ 0.14794182777404785,
60
+ 0.14764976501464844,
61
+ 0.12120175361633301,
62
+ 0.14845824241638184,
63
+ 0.14960455894470215,
64
+ 0.14653682708740234,
65
+ 0.1530618667602539,
66
+ 0.14752554893493652,
67
+ 0.1508030891418457,
68
+ 0.15003490447998047,
69
+ 0.14949798583984375,
70
+ 0.15344691276550293,
71
+ 0.15499567985534668,
72
+ 0.15093731880187988,
73
+ 0.1544790267944336,
74
+ 0.1501610279083252,
75
+ 0.1559765338897705,
76
+ 0.1554403305053711,
77
+ 0.15172815322875977,
78
+ 0.15061450004577637,
79
+ 0.18222713470458984,
80
+ 0.15479183197021484,
81
+ 0.14963650703430176,
82
+ 0.15522122383117676,
83
+ 0.15098309516906738,
84
+ 0.14656662940979004,
85
+ 0.14910221099853516,
86
+ 0.14737510681152344,
87
+ 0.15102934837341309,
88
+ 0.14995050430297852,
89
+ 0.15009093284606934,
90
+ 0.14732837677001953,
91
+ 0.147308349609375,
92
+ 0.1545259952545166,
93
+ 0.1512002944946289,
94
+ 0.15062594413757324,
95
+ 0.15013861656188965,
96
+ 0.14808416366577148,
97
+ 0.15249896049499512,
98
+ 0.14633679389953613,
99
+ 0.14722561836242676,
100
+ 0.1504042148590088,
101
+ 0.14682531356811523,
102
+ 0.1454761028289795,
103
+ 0.15203118324279785,
104
+ 0.14587736129760742
105
+ ],
106
+ "err_cube_cm": 1.5,
107
+ "err_tray_cm": 1.0
108
+ },
109
+ {
110
+ "seed": 3,
111
+ "target": "green",
112
+ "grasped": false,
113
+ "success": false,
114
+ "decisions": 90,
115
+ "latency_s": [
116
+ 0.15128469467163086,
117
+ 0.14608550071716309,
118
+ 0.15372586250305176,
119
+ 0.14850115776062012,
120
+ 0.10045218467712402,
121
+ 0.1545865535736084,
122
+ 0.14556145668029785,
123
+ 0.14880585670471191,
124
+ 0.15238070487976074,
125
+ 0.15112638473510742,
126
+ 0.15643954277038574,
127
+ 0.15695500373840332,
128
+ 0.15022683143615723,
129
+ 0.153731107711792,
130
+ 0.1563715934753418,
131
+ 0.1520071029663086,
132
+ 0.14841222763061523,
133
+ 0.1529843807220459,
134
+ 0.1584625244140625,
135
+ 0.15793251991271973,
136
+ 0.15379834175109863,
137
+ 0.15744495391845703,
138
+ 0.152817964553833,
139
+ 0.15059947967529297,
140
+ 0.1567084789276123,
141
+ 0.15385031700134277,
142
+ 0.15833306312561035,
143
+ 0.14472222328186035,
144
+ 0.15470075607299805,
145
+ 0.15184664726257324,
146
+ 0.15483355522155762,
147
+ 0.15488672256469727,
148
+ 0.15174198150634766,
149
+ 0.155836820602417,
150
+ 0.148482084274292,
151
+ 0.13758039474487305,
152
+ 0.15195798873901367,
153
+ 0.14637088775634766,
154
+ 0.1517791748046875,
155
+ 0.1466226577758789,
156
+ 0.1544477939605713,
157
+ 0.15534186363220215,
158
+ 0.15623259544372559,
159
+ 0.15033459663391113,
160
+ 0.14940810203552246,
161
+ 0.1492152214050293,
162
+ 0.15506768226623535,
163
+ 0.15378928184509277,
164
+ 0.15277600288391113,
165
+ 0.15210700035095215,
166
+ 0.15005922317504883,
167
+ 0.15384626388549805,
168
+ 0.15366148948669434,
169
+ 0.15520858764648438,
170
+ 0.15662932395935059,
171
+ 0.15624070167541504,
172
+ 0.1539297103881836,
173
+ 0.1547682285308838,
174
+ 0.1517496109008789,
175
+ 0.15997743606567383,
176
+ 0.14963340759277344,
177
+ 0.15323901176452637,
178
+ 0.15114259719848633,
179
+ 0.1550431251525879,
180
+ 0.1550900936126709,
181
+ 0.1580510139465332,
182
+ 0.13998174667358398,
183
+ 0.1541891098022461,
184
+ 0.147566556930542,
185
+ 0.14937567710876465,
186
+ 0.1489124298095703,
187
+ 0.15399789810180664,
188
+ 0.1941089630126953,
189
+ 0.1538066864013672,
190
+ 0.1693429946899414,
191
+ 0.16510796546936035,
192
+ 0.1680469512939453,
193
+ 0.17217755317687988,
194
+ 0.16513323783874512,
195
+ 0.16628789901733398,
196
+ 0.1513369083404541,
197
+ 0.16958403587341309,
198
+ 0.1649014949798584,
199
+ 0.1623539924621582,
200
+ 0.1649625301361084,
201
+ 0.17579078674316406,
202
+ 0.1707143783569336,
203
+ 0.16825175285339355,
204
+ 0.18377900123596191,
205
+ 0.1658163070678711
206
+ ],
207
+ "err_cube_cm": 3.4,
208
+ "err_tray_cm": 67.8
209
+ },
210
+ {
211
+ "seed": 4,
212
+ "target": "blue",
213
+ "grasped": false,
214
+ "success": false,
215
+ "decisions": 90,
216
+ "latency_s": [
217
+ 0.16331028938293457,
218
+ 0.1719965934753418,
219
+ 0.1665046215057373,
220
+ 0.15205836296081543,
221
+ 0.1648242473602295,
222
+ 0.16338253021240234,
223
+ 0.16593265533447266,
224
+ 0.1666417121887207,
225
+ 0.16753530502319336,
226
+ 0.16296601295471191,
227
+ 0.16569137573242188,
228
+ 0.16828584671020508,
229
+ 0.16405916213989258,
230
+ 0.16759133338928223,
231
+ 0.16232013702392578,
232
+ 0.16239094734191895,
233
+ 0.1758885383605957,
234
+ 0.16239213943481445,
235
+ 0.1656661033630371,
236
+ 0.16174674034118652,
237
+ 0.16253280639648438,
238
+ 0.16225767135620117,
239
+ 0.16350889205932617,
240
+ 0.16369247436523438,
241
+ 0.1609792709350586,
242
+ 0.16810178756713867,
243
+ 0.16269779205322266,
244
+ 0.16515851020812988,
245
+ 0.16607451438903809,
246
+ 0.16668057441711426,
247
+ 0.1679079532623291,
248
+ 0.1631636619567871,
249
+ 0.16274571418762207,
250
+ 0.1639406681060791,
251
+ 0.17934131622314453,
252
+ 0.16142821311950684,
253
+ 0.1486978530883789,
254
+ 0.16370725631713867,
255
+ 0.19603681564331055,
256
+ 0.16637873649597168,
257
+ 0.16416549682617188,
258
+ 0.1754295825958252,
259
+ 0.17461347579956055,
260
+ 0.19592571258544922,
261
+ 0.1870255470275879,
262
+ 0.17638206481933594,
263
+ 0.1683955192565918,
264
+ 0.16839146614074707,
265
+ 0.16920948028564453,
266
+ 0.17516541481018066,
267
+ 0.16480636596679688,
268
+ 0.1444244384765625,
269
+ 0.16182327270507812,
270
+ 0.15734124183654785,
271
+ 0.16174745559692383,
272
+ 0.16596579551696777,
273
+ 0.17611002922058105,
274
+ 0.16789841651916504,
275
+ 0.17365479469299316,
276
+ 0.16453862190246582,
277
+ 0.1724545955657959,
278
+ 0.17128849029541016,
279
+ 0.1731107234954834,
280
+ 0.1629636287689209,
281
+ 0.17299103736877441,
282
+ 0.16163325309753418,
283
+ 0.18559670448303223,
284
+ 0.1737668514251709,
285
+ 0.17210674285888672,
286
+ 0.1630702018737793,
287
+ 0.17554283142089844,
288
+ 0.16904067993164062,
289
+ 0.17999529838562012,
290
+ 0.1708543300628662,
291
+ 0.1746959686279297,
292
+ 0.17174530029296875,
293
+ 0.1687009334564209,
294
+ 0.16135740280151367,
295
+ 0.16906332969665527,
296
+ 0.16861319541931152,
297
+ 0.1733541488647461,
298
+ 0.1628274917602539,
299
+ 0.16569757461547852,
300
+ 0.18170785903930664,
301
+ 0.170884370803833,
302
+ 0.20191383361816406,
303
+ 0.16751527786254883,
304
+ 0.16791796684265137,
305
+ 0.16507863998413086,
306
+ 0.16106033325195312
307
+ ],
308
+ "err_cube_cm": 3.3,
309
+ "err_tray_cm": 25.5
310
+ },
311
+ {
312
+ "seed": 5,
313
+ "target": "red",
314
+ "grasped": true,
315
+ "success": true,
316
+ "decisions": 32,
317
+ "latency_s": [
318
+ 0.17061877250671387,
319
+ 0.16766047477722168,
320
+ 0.1648540496826172,
321
+ 0.18130874633789062,
322
+ 0.16256999969482422,
323
+ 0.17158842086791992,
324
+ 0.17049741744995117,
325
+ 0.16570425033569336,
326
+ 0.17221426963806152,
327
+ 0.1765904426574707,
328
+ 0.17269062995910645,
329
+ 0.18073415756225586,
330
+ 0.16491293907165527,
331
+ 0.16219663619995117,
332
+ 0.17185115814208984,
333
+ 0.16907238960266113,
334
+ 0.17139053344726562,
335
+ 0.1810009479522705,
336
+ 0.16678571701049805,
337
+ 0.16317987442016602,
338
+ 0.134141206741333,
339
+ 0.1635448932647705,
340
+ 0.17546653747558594,
341
+ 0.16579508781433105,
342
+ 0.1645979881286621,
343
+ 0.16788482666015625,
344
+ 0.1677229404449463,
345
+ 0.16513609886169434,
346
+ 0.1661679744720459,
347
+ 0.16600275039672852,
348
+ 0.16371417045593262,
349
+ 0.16642045974731445
350
+ ],
351
+ "err_cube_cm": 2.1,
352
+ "err_tray_cm": 4.9
353
+ },
354
+ {
355
+ "seed": 6,
356
+ "target": "green",
357
+ "grasped": true,
358
+ "success": true,
359
+ "decisions": 26,
360
+ "latency_s": [
361
+ 0.1640763282775879,
362
+ 0.17063593864440918,
363
+ 0.1661543846130371,
364
+ 0.16583776473999023,
365
+ 0.16586637496948242,
366
+ 0.190385103225708,
367
+ 0.15907859802246094,
368
+ 0.1711275577545166,
369
+ 0.17822980880737305,
370
+ 0.16608762741088867,
371
+ 0.17181921005249023,
372
+ 0.16666603088378906,
373
+ 0.1689445972442627,
374
+ 0.16193127632141113,
375
+ 0.17075872421264648,
376
+ 0.16643023490905762,
377
+ 0.17090964317321777,
378
+ 0.16591906547546387,
379
+ 0.16864633560180664,
380
+ 0.1692194938659668,
381
+ 0.16112899780273438,
382
+ 0.1623060703277588,
383
+ 0.16805386543273926,
384
+ 0.16295886039733887,
385
+ 0.16415691375732422,
386
+ 0.16364073753356934
387
+ ],
388
+ "err_cube_cm": 1.1,
389
+ "err_tray_cm": 2.0
390
+ },
391
+ {
392
+ "seed": 7,
393
+ "target": "red",
394
+ "grasped": true,
395
+ "success": true,
396
+ "decisions": 30,
397
+ "latency_s": [
398
+ 0.20673155784606934,
399
+ 0.1564633846282959,
400
+ 0.20264840126037598,
401
+ 0.16524863243103027,
402
+ 0.20116233825683594,
403
+ 0.16195130348205566,
404
+ 0.2088174819946289,
405
+ 0.16545438766479492,
406
+ 0.20531058311462402,
407
+ 0.15031743049621582,
408
+ 0.20110559463500977,
409
+ 0.16726350784301758,
410
+ 0.21893930435180664,
411
+ 0.16807150840759277,
412
+ 0.203688383102417,
413
+ 0.16510319709777832,
414
+ 0.20451068878173828,
415
+ 0.16624140739440918,
416
+ 0.20621180534362793,
417
+ 0.16712069511413574,
418
+ 0.205963134765625,
419
+ 0.17266273498535156,
420
+ 0.23237943649291992,
421
+ 0.16929173469543457,
422
+ 0.20206809043884277,
423
+ 0.16595673561096191,
424
+ 0.2043459415435791,
425
+ 0.17035222053527832,
426
+ 0.21439814567565918,
427
+ 0.1685488224029541
428
+ ],
429
+ "err_cube_cm": 2.4,
430
+ "err_tray_cm": 4.8
431
+ },
432
+ {
433
+ "seed": 8,
434
+ "target": "blue",
435
+ "grasped": false,
436
+ "success": false,
437
+ "decisions": 82,
438
+ "latency_s": [
439
+ 0.19339776039123535,
440
+ 0.1569969654083252,
441
+ 0.2024688720703125,
442
+ 0.16430354118347168,
443
+ 0.19573736190795898,
444
+ 0.15096282958984375,
445
+ 0.21553683280944824,
446
+ 0.1773369312286377,
447
+ 0.2087562084197998,
448
+ 0.15973472595214844,
449
+ 0.1607527732849121,
450
+ 0.1675410270690918,
451
+ 0.2064979076385498,
452
+ 0.16005754470825195,
453
+ 0.19417786598205566,
454
+ 0.16245245933532715,
455
+ 0.20153260231018066,
456
+ 0.16419696807861328,
457
+ 0.19811701774597168,
458
+ 0.16049909591674805,
459
+ 0.20149493217468262,
460
+ 0.16025876998901367,
461
+ 0.1987006664276123,
462
+ 0.1595156192779541,
463
+ 0.16425085067749023,
464
+ 0.1614375114440918,
465
+ 0.19330739974975586,
466
+ 0.16610455513000488,
467
+ 0.20542025566101074,
468
+ 0.16302061080932617,
469
+ 0.16231656074523926,
470
+ 0.17142820358276367,
471
+ 0.16280269622802734,
472
+ 0.16056585311889648,
473
+ 0.16420698165893555,
474
+ 0.16082239151000977,
475
+ 0.16008305549621582,
476
+ 0.15918278694152832,
477
+ 0.16196179389953613,
478
+ 0.1614227294921875,
479
+ 0.17899203300476074,
480
+ 0.1736924648284912,
481
+ 0.15526628494262695,
482
+ 0.15932178497314453,
483
+ 0.1582167148590088,
484
+ 0.1595761775970459,
485
+ 0.16014838218688965,
486
+ 0.16128087043762207,
487
+ 0.15979480743408203,
488
+ 0.16332149505615234,
489
+ 0.16652798652648926,
490
+ 0.1645801067352295,
491
+ 0.16188311576843262,
492
+ 0.16529011726379395,
493
+ 0.16726446151733398,
494
+ 0.16331839561462402,
495
+ 0.16661477088928223,
496
+ 0.16471290588378906,
497
+ 0.15854263305664062,
498
+ 0.1667335033416748,
499
+ 0.16406965255737305,
500
+ 0.16534137725830078,
501
+ 0.12666773796081543,
502
+ 0.1603085994720459,
503
+ 0.16961407661437988,
504
+ 0.1626873016357422,
505
+ 0.16137385368347168,
506
+ 0.1643974781036377,
507
+ 0.16205692291259766,
508
+ 0.16374564170837402,
509
+ 0.16326546669006348,
510
+ 0.16357660293579102,
511
+ 0.16809558868408203,
512
+ 0.16595888137817383,
513
+ 0.1626734733581543,
514
+ 0.16634774208068848,
515
+ 0.16391253471374512,
516
+ 0.16840028762817383,
517
+ 0.16501998901367188,
518
+ 0.1619739532470703,
519
+ 0.1612253189086914,
520
+ 0.16166090965270996
521
+ ],
522
+ "err_cube_cm": 3.2,
523
+ "err_tray_cm": 19.3
524
+ },
525
+ {
526
+ "seed": 9,
527
+ "target": "blue",
528
+ "grasped": false,
529
+ "success": false,
530
+ "decisions": 96,
531
+ "latency_s": [
532
+ 0.20352482795715332,
533
+ 0.15891504287719727,
534
+ 0.20598554611206055,
535
+ 0.16164159774780273,
536
+ 0.1978316307067871,
537
+ 0.16097354888916016,
538
+ 0.19813299179077148,
539
+ 0.15992426872253418,
540
+ 0.19629311561584473,
541
+ 0.16271734237670898,
542
+ 0.20453929901123047,
543
+ 0.168745756149292,
544
+ 0.20212674140930176,
545
+ 0.1644301414489746,
546
+ 0.20218706130981445,
547
+ 0.16606974601745605,
548
+ 0.21196532249450684,
549
+ 0.17016220092773438,
550
+ 0.21654105186462402,
551
+ 0.16472482681274414,
552
+ 0.21234130859375,
553
+ 0.16014409065246582,
554
+ 0.21528267860412598,
555
+ 0.16640281677246094,
556
+ 0.16515588760375977,
557
+ 0.16235709190368652,
558
+ 0.21180319786071777,
559
+ 0.1678633689880371,
560
+ 0.20135235786437988,
561
+ 0.18933749198913574,
562
+ 0.20406746864318848,
563
+ 0.16589903831481934,
564
+ 0.20027589797973633,
565
+ 0.16285991668701172,
566
+ 0.2020554542541504,
567
+ 0.1627969741821289,
568
+ 0.1991586685180664,
569
+ 0.15867209434509277,
570
+ 0.2000436782836914,
571
+ 0.16484737396240234,
572
+ 0.20023536682128906,
573
+ 0.16389060020446777,
574
+ 0.20389413833618164,
575
+ 0.16626453399658203,
576
+ 0.16895651817321777,
577
+ 0.15232396125793457,
578
+ 0.16523027420043945,
579
+ 0.1673595905303955,
580
+ 0.1624760627746582,
581
+ 0.18807291984558105,
582
+ 0.16384482383728027,
583
+ 0.1913154125213623,
584
+ 0.16869187355041504,
585
+ 0.16475343704223633,
586
+ 0.16739821434020996,
587
+ 0.16143369674682617,
588
+ 0.16172218322753906,
589
+ 0.16272282600402832,
590
+ 0.1643662452697754,
591
+ 0.18226981163024902,
592
+ 0.17708897590637207,
593
+ 0.17068266868591309,
594
+ 0.16834306716918945,
595
+ 0.16597628593444824,
596
+ 0.18541598320007324,
597
+ 0.16615867614746094,
598
+ 0.17827820777893066,
599
+ 0.15961718559265137,
600
+ 0.14308762550354004,
601
+ 0.17467641830444336,
602
+ 0.16643261909484863,
603
+ 0.16379046440124512,
604
+ 0.15670323371887207,
605
+ 0.17038297653198242,
606
+ 0.1670687198638916,
607
+ 0.16424846649169922,
608
+ 0.17002391815185547,
609
+ 0.16855573654174805,
610
+ 0.12075543403625488,
611
+ 0.17441058158874512,
612
+ 0.18262815475463867,
613
+ 0.1669609546661377,
614
+ 0.16337013244628906,
615
+ 0.16478872299194336,
616
+ 0.16783714294433594,
617
+ 0.1670534610748291,
618
+ 0.1699380874633789,
619
+ 0.16397762298583984,
620
+ 0.1703777313232422,
621
+ 0.16193437576293945,
622
+ 0.16991376876831055,
623
+ 0.16217470169067383,
624
+ 0.16189932823181152,
625
+ 0.17664766311645508,
626
+ 0.16300320625305176,
627
+ 0.16489315032958984
628
+ ],
629
+ "err_cube_cm": 4.6,
630
+ "err_tray_cm": 17.9
631
+ },
632
+ {
633
+ "seed": 10,
634
+ "target": "green",
635
+ "grasped": false,
636
+ "success": false,
637
+ "decisions": 82,
638
+ "latency_s": [
639
+ 0.2093064785003662,
640
+ 0.17333698272705078,
641
+ 0.2068624496459961,
642
+ 0.16109156608581543,
643
+ 0.19836783409118652,
644
+ 0.15723943710327148,
645
+ 0.23041701316833496,
646
+ 0.16355466842651367,
647
+ 0.2147824764251709,
648
+ 0.16574716567993164,
649
+ 0.16030669212341309,
650
+ 0.16135144233703613,
651
+ 0.19646430015563965,
652
+ 0.15660572052001953,
653
+ 0.1918315887451172,
654
+ 0.15804052352905273,
655
+ 0.22192764282226562,
656
+ 0.1637883186340332,
657
+ 0.20128393173217773,
658
+ 0.1352522373199463,
659
+ 0.2088463306427002,
660
+ 0.16486644744873047,
661
+ 0.19813060760498047,
662
+ 0.17567682266235352,
663
+ 0.20305347442626953,
664
+ 0.16396760940551758,
665
+ 0.2142963409423828,
666
+ 0.16744136810302734,
667
+ 0.1691584587097168,
668
+ 0.1834094524383545,
669
+ 0.16382336616516113,
670
+ 0.15951037406921387,
671
+ 0.1638176441192627,
672
+ 0.1723639965057373,
673
+ 0.16925978660583496,
674
+ 0.16902637481689453,
675
+ 0.2306532859802246,
676
+ 0.1661090850830078,
677
+ 0.16634488105773926,
678
+ 0.16460251808166504,
679
+ 0.1669023036956787,
680
+ 0.1646256446838379,
681
+ 0.1673583984375,
682
+ 0.17415475845336914,
683
+ 0.16911911964416504,
684
+ 0.16770100593566895,
685
+ 0.16658449172973633,
686
+ 0.1627058982849121,
687
+ 0.16500186920166016,
688
+ 0.19654583930969238,
689
+ 0.16625189781188965,
690
+ 0.17008614540100098,
691
+ 0.1638941764831543,
692
+ 0.16731023788452148,
693
+ 0.1579911708831787,
694
+ 0.16180157661437988,
695
+ 0.1655125617980957,
696
+ 0.17016148567199707,
697
+ 0.16460537910461426,
698
+ 0.1617908477783203,
699
+ 0.1627798080444336,
700
+ 0.1598196029663086,
701
+ 0.16285467147827148,
702
+ 0.16386103630065918,
703
+ 0.19768166542053223,
704
+ 0.16223406791687012,
705
+ 0.1666584014892578,
706
+ 0.18295693397521973,
707
+ 0.1596236228942871,
708
+ 0.16376686096191406,
709
+ 0.16198110580444336,
710
+ 0.19023442268371582,
711
+ 0.16594243049621582,
712
+ 0.1618199348449707,
713
+ 0.16651463508605957,
714
+ 0.16783881187438965,
715
+ 0.16245222091674805,
716
+ 0.1649329662322998,
717
+ 0.16631221771240234,
718
+ 0.16660594940185547,
719
+ 0.1741631031036377,
720
+ 0.1656017303466797
721
+ ],
722
+ "err_cube_cm": 4.8,
723
+ "err_tray_cm": 40.7
724
+ },
725
+ {
726
+ "seed": 11,
727
+ "target": "green",
728
+ "grasped": true,
729
+ "success": true,
730
+ "decisions": 48,
731
+ "latency_s": [
732
+ 0.2099461555480957,
733
+ 0.16610193252563477,
734
+ 0.20555925369262695,
735
+ 0.17211651802062988,
736
+ 0.20979571342468262,
737
+ 0.1739034652709961,
738
+ 0.21056532859802246,
739
+ 0.16616058349609375,
740
+ 0.21015548706054688,
741
+ 0.17554807662963867,
742
+ 0.20904779434204102,
743
+ 0.17281842231750488,
744
+ 0.2286539077758789,
745
+ 0.1633753776550293,
746
+ 0.212935209274292,
747
+ 0.15981602668762207,
748
+ 0.1979985237121582,
749
+ 0.17574715614318848,
750
+ 0.20976543426513672,
751
+ 0.1637122631072998,
752
+ 0.16968560218811035,
753
+ 0.1721057891845703,
754
+ 0.1926555633544922,
755
+ 0.15809392929077148,
756
+ 0.20854568481445312,
757
+ 0.1690981388092041,
758
+ 0.21499085426330566,
759
+ 0.1654520034790039,
760
+ 0.20512986183166504,
761
+ 0.1705944538116455,
762
+ 0.21517562866210938,
763
+ 0.16565704345703125,
764
+ 0.20313739776611328,
765
+ 0.16817927360534668,
766
+ 0.20700621604919434,
767
+ 0.17198610305786133,
768
+ 0.20272541046142578,
769
+ 0.16412973403930664,
770
+ 0.1991407871246338,
771
+ 0.16657757759094238,
772
+ 0.21828222274780273,
773
+ 0.15997028350830078,
774
+ 0.21883916854858398,
775
+ 0.16474127769470215,
776
+ 0.20826053619384766,
777
+ 0.16187214851379395,
778
+ 0.16382312774658203,
779
+ 0.16101884841918945
780
+ ],
781
+ "err_cube_cm": 0.6,
782
+ "err_tray_cm": 3.9
783
+ },
784
+ {
785
+ "seed": 12,
786
+ "target": "blue",
787
+ "grasped": false,
788
+ "success": false,
789
+ "decisions": 96,
790
+ "latency_s": [
791
+ 0.20508813858032227,
792
+ 0.16765975952148438,
793
+ 0.20066523551940918,
794
+ 0.16635632514953613,
795
+ 0.2124922275543213,
796
+ 0.16355204582214355,
797
+ 0.19300413131713867,
798
+ 0.16161298751831055,
799
+ 0.2024393081665039,
800
+ 0.16226935386657715,
801
+ 0.19719457626342773,
802
+ 0.1598668098449707,
803
+ 0.20614075660705566,
804
+ 0.16141557693481445,
805
+ 0.1970667839050293,
806
+ 0.16299176216125488,
807
+ 0.21443486213684082,
808
+ 0.16666150093078613,
809
+ 0.20787334442138672,
810
+ 0.16379880905151367,
811
+ 0.19938182830810547,
812
+ 0.1652083396911621,
813
+ 0.16272187232971191,
814
+ 0.16046977043151855,
815
+ 0.1608293056488037,
816
+ 0.1602458953857422,
817
+ 0.19964218139648438,
818
+ 0.15793347358703613,
819
+ 0.16341829299926758,
820
+ 0.1585865020751953,
821
+ 0.1629936695098877,
822
+ 0.16678357124328613,
823
+ 0.20026707649230957,
824
+ 0.1667344570159912,
825
+ 0.1977238655090332,
826
+ 0.1635890007019043,
827
+ 0.20461106300354004,
828
+ 0.14213180541992188,
829
+ 0.19145941734313965,
830
+ 0.16380834579467773,
831
+ 0.20421814918518066,
832
+ 0.17346692085266113,
833
+ 0.20344996452331543,
834
+ 0.16104435920715332,
835
+ 0.16668200492858887,
836
+ 0.16087579727172852,
837
+ 0.16176438331604004,
838
+ 0.1772608757019043,
839
+ 0.11407470703125,
840
+ 0.16593170166015625,
841
+ 0.16763758659362793,
842
+ 0.16327881813049316,
843
+ 0.1678760051727295,
844
+ 0.1662130355834961,
845
+ 0.15969038009643555,
846
+ 0.16609430313110352,
847
+ 0.1652989387512207,
848
+ 0.17200803756713867,
849
+ 0.17134666442871094,
850
+ 0.17074322700500488,
851
+ 0.17874908447265625,
852
+ 0.17688727378845215,
853
+ 0.15912795066833496,
854
+ 0.1687479019165039,
855
+ 0.17106413841247559,
856
+ 0.17171669006347656,
857
+ 0.17596101760864258,
858
+ 0.1757974624633789,
859
+ 0.14510655403137207,
860
+ 0.16336774826049805,
861
+ 0.1688995361328125,
862
+ 0.16318249702453613,
863
+ 0.16742849349975586,
864
+ 0.1629958152770996,
865
+ 0.1641983985900879,
866
+ 0.1644456386566162,
867
+ 0.16176891326904297,
868
+ 0.16602206230163574,
869
+ 0.17296576499938965,
870
+ 0.16426610946655273,
871
+ 0.16906332969665527,
872
+ 0.16421079635620117,
873
+ 0.16587615013122559,
874
+ 0.1594860553741455,
875
+ 0.1622481346130371,
876
+ 0.14533352851867676,
877
+ 0.17366552352905273,
878
+ 0.16648125648498535,
879
+ 0.17390751838684082,
880
+ 0.162398099899292,
881
+ 0.16595983505249023,
882
+ 0.16602444648742676,
883
+ 0.16598224639892578,
884
+ 0.1660773754119873,
885
+ 0.1659703254699707,
886
+ 0.16227459907531738
887
+ ],
888
+ "err_cube_cm": 4.9,
889
+ "err_tray_cm": 55.5
890
+ },
891
+ {
892
+ "seed": 13,
893
+ "target": "red",
894
+ "grasped": true,
895
+ "success": true,
896
+ "decisions": 42,
897
+ "latency_s": [
898
+ 0.2104494571685791,
899
+ 0.1671748161315918,
900
+ 0.19681191444396973,
901
+ 0.1678330898284912,
902
+ 0.20037317276000977,
903
+ 0.16840815544128418,
904
+ 0.2084059715270996,
905
+ 0.16996169090270996,
906
+ 0.20596766471862793,
907
+ 0.1646730899810791,
908
+ 0.2149975299835205,
909
+ 0.17001914978027344,
910
+ 0.21709752082824707,
911
+ 0.1546487808227539,
912
+ 0.2042093276977539,
913
+ 0.1749873161315918,
914
+ 0.21361947059631348,
915
+ 0.17026305198669434,
916
+ 0.2116410732269287,
917
+ 0.1642756462097168,
918
+ 0.21039986610412598,
919
+ 0.15838265419006348,
920
+ 0.2049713134765625,
921
+ 0.17393827438354492,
922
+ 0.15663981437683105,
923
+ 0.16469311714172363,
924
+ 0.2125532627105713,
925
+ 0.16513681411743164,
926
+ 0.20154476165771484,
927
+ 0.16494965553283691,
928
+ 0.21805644035339355,
929
+ 0.17029213905334473,
930
+ 0.2231004238128662,
931
+ 0.16659092903137207,
932
+ 0.21116995811462402,
933
+ 0.17401981353759766,
934
+ 0.21162748336791992,
935
+ 0.15955114364624023,
936
+ 0.21157598495483398,
937
+ 0.16707611083984375,
938
+ 0.16859745979309082,
939
+ 0.16827106475830078
940
+ ],
941
+ "err_cube_cm": 2.5,
942
+ "err_tray_cm": 1.6
943
+ },
944
+ {
945
+ "seed": 14,
946
+ "target": "green",
947
+ "grasped": true,
948
+ "success": true,
949
+ "decisions": 26,
950
+ "latency_s": [
951
+ 0.20794367790222168,
952
+ 0.16303467750549316,
953
+ 0.20073843002319336,
954
+ 0.16667604446411133,
955
+ 0.20035004615783691,
956
+ 0.16644573211669922,
957
+ 0.2053675651550293,
958
+ 0.16655683517456055,
959
+ 0.20473337173461914,
960
+ 0.15445828437805176,
961
+ 0.20827102661132812,
962
+ 0.17091012001037598,
963
+ 0.21140575408935547,
964
+ 0.16683554649353027,
965
+ 0.2151799201965332,
966
+ 0.17155909538269043,
967
+ 0.199265718460083,
968
+ 0.1639242172241211,
969
+ 0.20447635650634766,
970
+ 0.1589372158050537,
971
+ 0.21308422088623047,
972
+ 0.16586780548095703,
973
+ 0.2147533893585205,
974
+ 0.16468167304992676,
975
+ 0.20565152168273926,
976
+ 0.16919589042663574
977
+ ],
978
+ "err_cube_cm": 2.8,
979
+ "err_tray_cm": 1.0
980
+ },
981
+ {
982
+ "seed": 15,
983
+ "target": "green",
984
+ "grasped": false,
985
+ "success": false,
986
+ "decisions": 88,
987
+ "latency_s": [
988
+ 0.2010512351989746,
989
+ 0.16399931907653809,
990
+ 0.2018425464630127,
991
+ 0.16584348678588867,
992
+ 0.20552778244018555,
993
+ 0.15807271003723145,
994
+ 0.1779947280883789,
995
+ 0.14650988578796387,
996
+ 0.19093608856201172,
997
+ 0.15465950965881348,
998
+ 0.1909487247467041,
999
+ 0.15062499046325684,
1000
+ 0.19574952125549316,
1001
+ 0.14584970474243164,
1002
+ 0.18716859817504883,
1003
+ 0.1480884552001953,
1004
+ 0.1545860767364502,
1005
+ 0.146575927734375,
1006
+ 0.18424582481384277,
1007
+ 0.14893555641174316,
1008
+ 0.1815495491027832,
1009
+ 0.14976811408996582,
1010
+ 0.19052720069885254,
1011
+ 0.1500382423400879,
1012
+ 0.1880505084991455,
1013
+ 0.15225529670715332,
1014
+ 0.18429803848266602,
1015
+ 0.15305304527282715,
1016
+ 0.18686652183532715,
1017
+ 0.1560533046722412,
1018
+ 0.19207000732421875,
1019
+ 0.15450000762939453,
1020
+ 0.1884307861328125,
1021
+ 0.15884947776794434,
1022
+ 0.15046191215515137,
1023
+ 0.1552431583404541,
1024
+ 0.15410327911376953,
1025
+ 0.15042686462402344,
1026
+ 0.15559768676757812,
1027
+ 0.15528535842895508,
1028
+ 0.15277552604675293,
1029
+ 0.14755606651306152,
1030
+ 0.1487257480621338,
1031
+ 0.1535649299621582,
1032
+ 0.15439081192016602,
1033
+ 0.15460419654846191,
1034
+ 0.15598034858703613,
1035
+ 0.1492600440979004,
1036
+ 0.15436291694641113,
1037
+ 0.15207481384277344,
1038
+ 0.15363597869873047,
1039
+ 0.14864754676818848,
1040
+ 0.14864349365234375,
1041
+ 0.11920166015625,
1042
+ 0.1504807472229004,
1043
+ 0.1514298915863037,
1044
+ 0.1483321189880371,
1045
+ 0.14338469505310059,
1046
+ 0.14738011360168457,
1047
+ 0.1526341438293457,
1048
+ 0.15166473388671875,
1049
+ 0.1522212028503418,
1050
+ 0.151991605758667,
1051
+ 0.15129995346069336,
1052
+ 0.15061402320861816,
1053
+ 0.1481921672821045,
1054
+ 0.14974737167358398,
1055
+ 0.1532595157623291,
1056
+ 0.15449237823486328,
1057
+ 0.1485745906829834,
1058
+ 0.1556706428527832,
1059
+ 0.1509861946105957,
1060
+ 0.15256714820861816,
1061
+ 0.14603543281555176,
1062
+ 0.14660239219665527,
1063
+ 0.15736770629882812,
1064
+ 0.15657496452331543,
1065
+ 0.15641474723815918,
1066
+ 0.1621689796447754,
1067
+ 0.15315985679626465,
1068
+ 0.15720558166503906,
1069
+ 0.16175222396850586,
1070
+ 0.15321946144104004,
1071
+ 0.14884638786315918,
1072
+ 0.15335488319396973,
1073
+ 0.14185357093811035,
1074
+ 0.147216796875,
1075
+ 0.15457749366760254
1076
+ ],
1077
+ "err_cube_cm": 3.1,
1078
+ "err_tray_cm": 44.0
1079
+ },
1080
+ {
1081
+ "seed": 16,
1082
+ "target": "green",
1083
+ "grasped": false,
1084
+ "success": false,
1085
+ "decisions": 90,
1086
+ "latency_s": [
1087
+ 0.1863095760345459,
1088
+ 0.14640498161315918,
1089
+ 0.17949247360229492,
1090
+ 0.1438908576965332,
1091
+ 0.1792771816253662,
1092
+ 0.14409160614013672,
1093
+ 0.19663715362548828,
1094
+ 0.15041017532348633,
1095
+ 0.18999028205871582,
1096
+ 0.15653491020202637,
1097
+ 0.18713021278381348,
1098
+ 0.14917564392089844,
1099
+ 0.21341919898986816,
1100
+ 0.14922475814819336,
1101
+ 0.1923975944519043,
1102
+ 0.14612627029418945,
1103
+ 0.18579936027526855,
1104
+ 0.14771580696105957,
1105
+ 0.15149474143981934,
1106
+ 0.14855337142944336,
1107
+ 0.18038225173950195,
1108
+ 0.1516737937927246,
1109
+ 0.1938624382019043,
1110
+ 0.15357017517089844,
1111
+ 0.18480610847473145,
1112
+ 0.15396714210510254,
1113
+ 0.21619415283203125,
1114
+ 0.15787291526794434,
1115
+ 0.18575048446655273,
1116
+ 0.15303325653076172,
1117
+ 0.18363094329833984,
1118
+ 0.1519005298614502,
1119
+ 0.19411683082580566,
1120
+ 0.1539757251739502,
1121
+ 0.1917247772216797,
1122
+ 0.1547250747680664,
1123
+ 0.15434670448303223,
1124
+ 0.15434813499450684,
1125
+ 0.1531219482421875,
1126
+ 0.14895057678222656,
1127
+ 0.15167450904846191,
1128
+ 0.1536424160003662,
1129
+ 0.15346145629882812,
1130
+ 0.14870905876159668,
1131
+ 0.16071534156799316,
1132
+ 0.16001200675964355,
1133
+ 0.1505286693572998,
1134
+ 0.20056962966918945,
1135
+ 0.15694141387939453,
1136
+ 0.15254688262939453,
1137
+ 0.15268254280090332,
1138
+ 0.15708112716674805,
1139
+ 0.15485906600952148,
1140
+ 0.15320825576782227,
1141
+ 0.15541291236877441,
1142
+ 0.156541109085083,
1143
+ 0.1590423583984375,
1144
+ 0.15610742568969727,
1145
+ 0.1533973217010498,
1146
+ 0.15067648887634277,
1147
+ 0.15171146392822266,
1148
+ 0.15763616561889648,
1149
+ 0.14809536933898926,
1150
+ 0.15169453620910645,
1151
+ 0.17575883865356445,
1152
+ 0.15857648849487305,
1153
+ 0.1371607780456543,
1154
+ 0.15185308456420898,
1155
+ 0.14739131927490234,
1156
+ 0.14653444290161133,
1157
+ 0.15288066864013672,
1158
+ 0.15011239051818848,
1159
+ 0.15007925033569336,
1160
+ 0.153181791305542,
1161
+ 0.15373015403747559,
1162
+ 0.15018320083618164,
1163
+ 0.14987468719482422,
1164
+ 0.14783024787902832,
1165
+ 0.14866900444030762,
1166
+ 0.15441298484802246,
1167
+ 0.15438246726989746,
1168
+ 0.15246367454528809,
1169
+ 0.15833711624145508,
1170
+ 0.15277695655822754,
1171
+ 0.15595507621765137,
1172
+ 0.15456056594848633,
1173
+ 0.15111780166625977,
1174
+ 0.15342354774475098,
1175
+ 0.15571236610412598,
1176
+ 0.15742826461791992
1177
+ ],
1178
+ "err_cube_cm": 4.4,
1179
+ "err_tray_cm": 24.5
1180
+ },
1181
+ {
1182
+ "seed": 17,
1183
+ "target": "red",
1184
+ "grasped": true,
1185
+ "success": true,
1186
+ "decisions": 36,
1187
+ "latency_s": [
1188
+ 0.19892191886901855,
1189
+ 0.15857934951782227,
1190
+ 0.18412327766418457,
1191
+ 0.1894071102142334,
1192
+ 0.23799467086791992,
1193
+ 0.20935297012329102,
1194
+ 0.19408392906188965,
1195
+ 0.153184175491333,
1196
+ 0.19673657417297363,
1197
+ 0.15222549438476562,
1198
+ 0.19276642799377441,
1199
+ 0.153184175491333,
1200
+ 0.1903059482574463,
1201
+ 0.1530749797821045,
1202
+ 0.1880481243133545,
1203
+ 0.1557021141052246,
1204
+ 0.15387177467346191,
1205
+ 0.15622568130493164,
1206
+ 0.18669867515563965,
1207
+ 0.15277981758117676,
1208
+ 0.18590641021728516,
1209
+ 0.15250301361083984,
1210
+ 0.18456363677978516,
1211
+ 0.13977646827697754,
1212
+ 0.1888272762298584,
1213
+ 0.15041112899780273,
1214
+ 0.18471717834472656,
1215
+ 0.1508166790008545,
1216
+ 0.18912029266357422,
1217
+ 0.15329980850219727,
1218
+ 0.19160914421081543,
1219
+ 0.15145301818847656,
1220
+ 0.22835493087768555,
1221
+ 0.1547379493713379,
1222
+ 0.19521784782409668,
1223
+ 0.1500089168548584
1224
+ ],
1225
+ "err_cube_cm": 1.8,
1226
+ "err_tray_cm": 3.7
1227
+ },
1228
+ {
1229
+ "seed": 18,
1230
+ "target": "red",
1231
+ "grasped": false,
1232
+ "success": false,
1233
+ "decisions": 84,
1234
+ "latency_s": [
1235
+ 0.14917302131652832,
1236
+ 0.14902591705322266,
1237
+ 0.18802285194396973,
1238
+ 0.1488494873046875,
1239
+ 0.19779396057128906,
1240
+ 0.1502213478088379,
1241
+ 0.18252110481262207,
1242
+ 0.1793663501739502,
1243
+ 0.14252161979675293,
1244
+ 0.1527729034423828,
1245
+ 0.1607067584991455,
1246
+ 0.1475067138671875,
1247
+ 0.15296697616577148,
1248
+ 0.15538716316223145,
1249
+ 0.1878504753112793,
1250
+ 0.15251851081848145,
1251
+ 0.192216157913208,
1252
+ 0.1532421112060547,
1253
+ 0.1836543083190918,
1254
+ 0.15541458129882812,
1255
+ 0.18592190742492676,
1256
+ 0.15483307838439941,
1257
+ 0.1885521411895752,
1258
+ 0.1530923843383789,
1259
+ 0.18607759475708008,
1260
+ 0.15130090713500977,
1261
+ 0.21628093719482422,
1262
+ 0.17477178573608398,
1263
+ 0.15333986282348633,
1264
+ 0.15810012817382812,
1265
+ 0.15193629264831543,
1266
+ 0.15450263023376465,
1267
+ 0.15192556381225586,
1268
+ 0.15848541259765625,
1269
+ 0.15929532051086426,
1270
+ 0.15741205215454102,
1271
+ 0.1589069366455078,
1272
+ 0.15372443199157715,
1273
+ 0.15258526802062988,
1274
+ 0.14943385124206543,
1275
+ 0.1532139778137207,
1276
+ 0.1499948501586914,
1277
+ 0.15138840675354004,
1278
+ 0.15170717239379883,
1279
+ 0.15247416496276855,
1280
+ 0.1512160301208496,
1281
+ 0.15250301361083984,
1282
+ 0.15441393852233887,
1283
+ 0.1526334285736084,
1284
+ 0.14800572395324707,
1285
+ 0.14567255973815918,
1286
+ 0.15387845039367676,
1287
+ 0.15760350227355957,
1288
+ 0.16466164588928223,
1289
+ 0.14678955078125,
1290
+ 0.14803600311279297,
1291
+ 0.1524035930633545,
1292
+ 0.1500399112701416,
1293
+ 0.1523733139038086,
1294
+ 0.15362954139709473,
1295
+ 0.15180087089538574,
1296
+ 0.14690303802490234,
1297
+ 0.1516871452331543,
1298
+ 0.14657998085021973,
1299
+ 0.16429424285888672,
1300
+ 0.16204333305358887,
1301
+ 0.14835166931152344,
1302
+ 0.15311050415039062,
1303
+ 0.14937949180603027,
1304
+ 0.1531980037689209,
1305
+ 0.15014195442199707,
1306
+ 0.15306806564331055,
1307
+ 0.15035271644592285,
1308
+ 0.15212750434875488,
1309
+ 0.14694666862487793,
1310
+ 0.14936232566833496,
1311
+ 0.1474590301513672,
1312
+ 0.14731240272521973,
1313
+ 0.14923858642578125,
1314
+ 0.15301203727722168,
1315
+ 0.1545107364654541,
1316
+ 0.15243792533874512,
1317
+ 0.15109896659851074,
1318
+ 0.1507411003112793
1319
+ ],
1320
+ "err_cube_cm": 4.7,
1321
+ "err_tray_cm": 21.1
1322
+ },
1323
+ {
1324
+ "seed": 19,
1325
+ "target": "green",
1326
+ "grasped": false,
1327
+ "success": false,
1328
+ "decisions": 88,
1329
+ "latency_s": [
1330
+ 0.18272829055786133,
1331
+ 0.14740395545959473,
1332
+ 0.1850569248199463,
1333
+ 0.15241193771362305,
1334
+ 0.1895771026611328,
1335
+ 0.15398049354553223,
1336
+ 0.18838000297546387,
1337
+ 0.1485741138458252,
1338
+ 0.197432279586792,
1339
+ 0.15067601203918457,
1340
+ 0.18927884101867676,
1341
+ 0.14853906631469727,
1342
+ 0.19145846366882324,
1343
+ 0.14862394332885742,
1344
+ 0.19844865798950195,
1345
+ 0.14926791191101074,
1346
+ 0.1520991325378418,
1347
+ 0.14821600914001465,
1348
+ 0.17774200439453125,
1349
+ 0.14293479919433594,
1350
+ 0.17765212059020996,
1351
+ 0.1472179889678955,
1352
+ 0.18136882781982422,
1353
+ 0.1493363380432129,
1354
+ 0.18008112907409668,
1355
+ 0.15492677688598633,
1356
+ 0.1895129680633545,
1357
+ 0.15097284317016602,
1358
+ 0.1771717071533203,
1359
+ 0.1497499942779541,
1360
+ 0.15463900566101074,
1361
+ 0.1505417823791504,
1362
+ 0.15276432037353516,
1363
+ 0.1515202522277832,
1364
+ 0.150726318359375,
1365
+ 0.14743852615356445,
1366
+ 0.15345549583435059,
1367
+ 0.14914584159851074,
1368
+ 0.15303301811218262,
1369
+ 0.15663790702819824,
1370
+ 0.14861035346984863,
1371
+ 0.15157651901245117,
1372
+ 0.15334343910217285,
1373
+ 0.15677762031555176,
1374
+ 0.14587879180908203,
1375
+ 0.14952564239501953,
1376
+ 0.1508779525756836,
1377
+ 0.15147018432617188,
1378
+ 0.14437389373779297,
1379
+ 0.14249277114868164,
1380
+ 0.14483952522277832,
1381
+ 0.14285492897033691,
1382
+ 0.1421341896057129,
1383
+ 0.14131975173950195,
1384
+ 0.14621233940124512,
1385
+ 0.14841580390930176,
1386
+ 0.14836525917053223,
1387
+ 0.15229058265686035,
1388
+ 0.1924731731414795,
1389
+ 0.16162538528442383,
1390
+ 0.1532454490661621,
1391
+ 0.15455269813537598,
1392
+ 0.1509103775024414,
1393
+ 0.15767478942871094,
1394
+ 0.14765501022338867,
1395
+ 0.15124893188476562,
1396
+ 0.1492767333984375,
1397
+ 0.1549053192138672,
1398
+ 0.1488041877746582,
1399
+ 0.18890643119812012,
1400
+ 0.14744877815246582,
1401
+ 0.16169333457946777,
1402
+ 0.1540541648864746,
1403
+ 0.15714478492736816,
1404
+ 0.15188384056091309,
1405
+ 0.15732979774475098,
1406
+ 0.15911126136779785,
1407
+ 0.15279817581176758,
1408
+ 0.15752315521240234,
1409
+ 0.15200042724609375,
1410
+ 0.16004657745361328,
1411
+ 0.14958810806274414,
1412
+ 0.15110135078430176,
1413
+ 0.14805388450622559,
1414
+ 0.1421337127685547,
1415
+ 0.1553494930267334,
1416
+ 0.1535491943359375,
1417
+ 0.15477442741394043
1418
+ ],
1419
+ "err_cube_cm": 3.0,
1420
+ "err_tray_cm": 29.2
1421
+ },
1422
+ {
1423
+ "seed": 20,
1424
+ "target": "red",
1425
+ "grasped": true,
1426
+ "success": true,
1427
+ "decisions": 30,
1428
+ "latency_s": [
1429
+ 0.18891525268554688,
1430
+ 0.1517958641052246,
1431
+ 0.1882014274597168,
1432
+ 0.15535545349121094,
1433
+ 0.19060683250427246,
1434
+ 0.15251970291137695,
1435
+ 0.18739748001098633,
1436
+ 0.15070867538452148,
1437
+ 0.19600963592529297,
1438
+ 0.1452345848083496,
1439
+ 0.19422125816345215,
1440
+ 0.15745139122009277,
1441
+ 0.18992924690246582,
1442
+ 0.1534719467163086,
1443
+ 0.18849945068359375,
1444
+ 0.16321325302124023,
1445
+ 0.19374942779541016,
1446
+ 0.14859318733215332,
1447
+ 0.184706449508667,
1448
+ 0.15281152725219727,
1449
+ 0.1841297149658203,
1450
+ 0.16469335556030273,
1451
+ 0.20492768287658691,
1452
+ 0.1560044288635254,
1453
+ 0.2009575366973877,
1454
+ 0.1532597541809082,
1455
+ 0.19934630393981934,
1456
+ 0.173966646194458,
1457
+ 0.2388148307800293,
1458
+ 0.15384340286254883
1459
+ ],
1460
+ "err_cube_cm": 3.0,
1461
+ "err_tray_cm": 3.5
1462
+ }
1463
+ ]
1464
+ }
reports/vl/synthetic_images.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "colour (8 options)": 1.0,
3
+ "shape (4 options)": 1.0,
4
+ "digits (8 options)": 0.7,
5
+ "red object? (yes/no)": 1.0
6
+ }
reports/vl/text_parity.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "n": 300,
3
+ "mean_abs_dp_max": 0.002224256388519142,
4
+ "max_abs_dp": 0.010940762902599044,
5
+ "share_over_0.05": 0.0,
6
+ "same_top_option": 0.99,
7
+ "kinds": {
8
+ "noul": 140,
9
+ "choice": 83,
10
+ "score": 77
11
+ }
12
+ }
reports/vl/vl_rewardbench.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "n": 1247,
3
+ "seconds": 135.443541765213,
4
+ "overall": 74.33841218925421,
5
+ "general": 48.61878453038674,
6
+ "hallucination": 81.84245660881176,
7
+ "reasoning": 71.29337539432177,
8
+ "macro": 67.25153884450675,
9
+ "ci95": [
10
+ 71.85244587008822,
11
+ 76.74418604651163
12
+ ],
13
+ "order_agree": 84.04170008019246
14
+ }
videos/computer_use_mail.mp4 ADDED
Binary file (75.1 kB). View file
 
videos/computer_use_settings.mp4 ADDED
Binary file (64.3 kB). View file
 
videos/computer_use_shop.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8b03d24c54f20c7432301cb7ec9b3fae461c1a2a8cdb145002a7f3899532b8a2
3
+ size 103216
videos/robot_arm_pick_place.mp4 ADDED
Binary file (74.9 kB). View file
 
vl/adapter_vllm/adapter_config.json ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alora_invocation_tokens": null,
3
+ "alpha_pattern": {},
4
+ "arrow_config": null,
5
+ "auto_mapping": {
6
+ "base_model_class": "Qwen3_5ForCausalLM",
7
+ "parent_library": "transformers.models.qwen3_5.modeling_qwen3_5"
8
+ },
9
+ "base_model_name_or_path": null,
10
+ "bias": "none",
11
+ "corda_config": null,
12
+ "ensure_weight_tying": false,
13
+ "eva_config": null,
14
+ "exclude_modules": null,
15
+ "fan_in_fan_out": false,
16
+ "inference_mode": true,
17
+ "init_lora_weights": true,
18
+ "kasa_config": null,
19
+ "layer_replication": null,
20
+ "layers_pattern": null,
21
+ "layers_to_transform": null,
22
+ "loftq_config": {},
23
+ "lora_alpha": 64.0,
24
+ "lora_bias": false,
25
+ "lora_dropout": 0.0,
26
+ "lora_ga_config": null,
27
+ "megatron_config": null,
28
+ "megatron_core": "megatron.core",
29
+ "modules_to_save": null,
30
+ "monteclora_config": null,
31
+ "peft_type": "LORA",
32
+ "peft_version": "0.21.0",
33
+ "qalora_group_size": 16,
34
+ "r": 32,
35
+ "rank_pattern": {},
36
+ "revision": null,
37
+ "target_modules": [
38
+ "down_proj",
39
+ "gate_proj",
40
+ "in_proj_qkv",
41
+ "in_proj_z",
42
+ "k_proj",
43
+ "lm_head",
44
+ "o_proj",
45
+ "out_proj",
46
+ "q_proj",
47
+ "up_proj",
48
+ "v_proj"
49
+ ],
50
+ "target_parameters": null,
51
+ "task_type": null,
52
+ "trainable_token_indices": null,
53
+ "use_bdlora": null,
54
+ "use_dora": false,
55
+ "use_qalora": false,
56
+ "use_rslora": false,
57
+ "velora_config": null
58
+ }
vl/adapter_vllm/adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:39d508b2692c85849ac252aa3f84726ec50e8c750b5bfb4f2b55ba4a7b5e1787
3
+ size 176647800
vl/adapter_vllm/decision_head.json ADDED
@@ -0,0 +1,99 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bias": [
3
+ 0.00044773018453270197,
4
+ -0.0004479771596379578,
5
+ -0.0025878134183585644,
6
+ -0.0012615364976227283,
7
+ -0.00025007399381138384,
8
+ 0.0019558551721274853,
9
+ 0.0013559526996687055,
10
+ -0.0023348911199718714,
11
+ -0.0004082925443071872,
12
+ -0.0002078723773593083,
13
+ -1.2957353646925185e-05,
14
+ 0.0001676026004133746,
15
+ 0.0005569425993598998,
16
+ -0.001412136247381568,
17
+ -0.0028663526754826307,
18
+ -0.005061559844762087,
19
+ -0.00593328895047307,
20
+ -0.004319756291806698,
21
+ -0.004922541789710522,
22
+ -0.0049067651852965355,
23
+ -0.003943523392081261,
24
+ -0.003319827374070883,
25
+ -0.005581587553024292,
26
+ -0.0034480984322726727
27
+ ],
28
+ "verbalizer_ids": [
29
+ 3721,
30
+ 1802,
31
+ 15,
32
+ 16,
33
+ 17,
34
+ 18,
35
+ 19,
36
+ 20,
37
+ 32,
38
+ 33,
39
+ 34,
40
+ 35,
41
+ 36,
42
+ 37,
43
+ 38,
44
+ 39,
45
+ 40,
46
+ 41,
47
+ 42,
48
+ 43,
49
+ 44,
50
+ 45,
51
+ 46,
52
+ 47
53
+ ],
54
+ "slots": {
55
+ "num_slots": 24,
56
+ "ranges": {
57
+ "noul": [
58
+ 0,
59
+ 2
60
+ ],
61
+ "score": [
62
+ 2,
63
+ 8
64
+ ],
65
+ "choice": [
66
+ 8,
67
+ 24
68
+ ]
69
+ },
70
+ "verbalizers": [
71
+ "false",
72
+ "true",
73
+ "0",
74
+ "1",
75
+ "2",
76
+ "3",
77
+ "4",
78
+ "5",
79
+ "A",
80
+ "B",
81
+ "C",
82
+ "D",
83
+ "E",
84
+ "F",
85
+ "G",
86
+ "H",
87
+ "I",
88
+ "J",
89
+ "K",
90
+ "L",
91
+ "M",
92
+ "N",
93
+ "O",
94
+ "P"
95
+ ],
96
+ "template_version": "bare-v1"
97
+ },
98
+ "note": "decision logits = lm_head-LoRA logprobs of verbalizer ids + bias, then per-kind temperature"
99
+ }
vl/calibration.json ADDED
@@ -0,0 +1,44 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "version": 1,
3
+ "per_kind": {
4
+ "noul": 1.0022178236691426,
5
+ "choice": 0.9840233476014968,
6
+ "score": 1.012184473350984
7
+ },
8
+ "per_kind_family": {},
9
+ "fit": {
10
+ "noul": {
11
+ "n": 4363,
12
+ "T": 1.0022178236691426,
13
+ "kl_before": 0.004595278762280941,
14
+ "kl_after": 0.0045948573388159275
15
+ },
16
+ "choice": {
17
+ "n": 3418,
18
+ "T": 0.9840233476014968,
19
+ "kl_before": 0.04468083754181862,
20
+ "kl_after": 0.04462483152747154
21
+ },
22
+ "score": {
23
+ "n": 3173,
24
+ "T": 1.012184473350984,
25
+ "kl_before": 0.022512147203087807,
26
+ "kl_after": 0.022472143173217773
27
+ }
28
+ },
29
+ "diagnostic_calibration_split": {
30
+ "raw": {
31
+ "kl": 0.02229316346347332,
32
+ "ece": 0.0014972193181917403,
33
+ "mce": 0.0034518837928771973
34
+ },
35
+ "calibrated": {
36
+ "kl": 0.02226393297314644,
37
+ "ece": 0.0013502779446837457,
38
+ "mce": 0.005257606506347656
39
+ }
40
+ },
41
+ "source": "checkpoints/s2_9b_epoch1/best",
42
+ "fit_rows": 10954,
43
+ "d1_excluded": true
44
+ }
vl/demos/computer_use.py ADDED
@@ -0,0 +1,178 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Computer-use click loop with JEV-9B-VL System 1 (POST /v1/decide), in a real headless Chromium (Playwright).
2
+
3
+ Every step: screenshot -> every interactive element gets a numbered red box (Set-of-Mark) -> one System 1 `choice`
4
+ over "click element k" for each mark plus "the task is complete" -> the browser really clicks the chosen element.
5
+ Variants: "marks" (the options are only the mark numbers: the model must read the screenshot to know what each mark
6
+ is) and "marks+text" (each option also carries the element's visible text, like an accessibility tree).
7
+ Tasks are randomised per seed (which email / product / colour / size / setting; element order and contents shuffled).
8
+
9
+ python3 computer_use.py run # 3 apps x N seeds x 2 variants -> cu_results.json
10
+ python3 computer_use.py record APP SEED VARIANT OUT.mp4
11
+ """
12
+ import base64, io, json, os, sys, time, subprocess, tempfile
13
+ import requests
14
+ from PIL import Image, ImageDraw, ImageFont
15
+
16
+ HERE = os.path.dirname(os.path.abspath(__file__))
17
+ URL = "file://" + os.path.join(HERE, "webapp.html")
18
+ API = os.environ.get("JEV_URL", "http://localhost:8000") + "/v1/decide"
19
+ VW, VH = 1100, 640
20
+ FONT = "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf"
21
+ FONTB = "/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf"
22
+ MAX_STEPS = 10
23
+
24
+
25
+ def task_for(app, world, seed):
26
+ import random
27
+ r = random.Random(seed * 7919 + len(app))
28
+ if app == "mail":
29
+ m = r.choice(world["MAILS"])
30
+ return f"Archive the email from {m['from']} about the {m['subject']}.", {"mail": m["id"]}
31
+ if app == "shop":
32
+ p, c, s = r.choice(world["PRODUCTS"]), r.choice(world["COLORS"]), r.choice(world["SIZES"])
33
+ return f"Buy a {c} {p['name']} in size {s}: add it to the cart and check out.", {"name": p["name"], "color": c, "size": s}
34
+ sec = r.choice(list(world["SECTIONS"])); t = r.choice(world["SECTIONS"][sec])
35
+ return f"Turn on \"{t}\" in the {sec} settings and save the change.", {"toggle": t}
36
+
37
+
38
+ def goal_met(app, S, g):
39
+ if app == "mail":
40
+ return S["archived"] == [g["mail"]] and not S["deleted"]
41
+ if app == "shop":
42
+ return bool(S["ordered"]) and S["cart"] == [{"name": g["name"], "color": g["color"], "size": g["size"]}]
43
+ on = [t for t, v in (S["saved"] or {}).items() if v]
44
+ return on == [g["toggle"]]
45
+
46
+
47
+ ELEMS_JS = """() => [...document.querySelectorAll('[data-click]')].map(e => { const r = e.getBoundingClientRect();
48
+ return {x: r.x, y: r.y, w: r.width, h: r.height, text: (e.innerText || '').trim().replace(/\\s+/g, ' ').slice(0, 60),
49
+ kind: e.classList.contains('swatch') ? 'colour swatch' : e.classList.contains('sw') ? 'switch' : e.classList.contains('row') ? 'list row' : e.classList.contains('card') ? 'product card' : 'button'}; })
50
+ .filter(e => e.w > 0 && e.h > 0 && e.y < innerHeight)"""
51
+
52
+
53
+ def mark(png, elems):
54
+ im = Image.open(io.BytesIO(png)).convert("RGB"); d = ImageDraw.Draw(im); f = ImageFont.truetype(FONTB, 15)
55
+ for i, e in enumerate(elems):
56
+ d.rectangle((e["x"] - 2, e["y"] - 2, e["x"] + e["w"] + 2, e["y"] + e["h"] + 2), outline=(230, 0, 0), width=2)
57
+ lab = str(i + 1); tw = d.textlength(lab, font=f)
58
+ ly = e["y"] - 20 if e["y"] >= 22 else e["y"] + e["h"] + 2 # label above the box, or below it at the top edge
59
+ d.rectangle((e["x"] - 2, ly, e["x"] + tw + 6, ly + 18), fill=(230, 0, 0))
60
+ d.text((e["x"] + 2, ly + 1), lab, fill="white", font=f)
61
+ return im
62
+
63
+
64
+ def to_url(im):
65
+ b = io.BytesIO(); im.save(b, "PNG"); return "data:image/png;base64," + base64.b64encode(b.getvalue()).decode()
66
+
67
+
68
+ def episode(page, app, seed, variant, frames=None):
69
+ page.goto(f"{URL}?app={app}&seed={seed}"); page.wait_for_timeout(50)
70
+ world = page.evaluate("() => window.WORLD")
71
+ task, goal = task_for(app, world, seed)
72
+ hist, steps, stopped, lat = [], [], False, []
73
+ for step in range(MAX_STEPS):
74
+ elems = page.evaluate(ELEMS_JS)
75
+ im = mark(page.screenshot(), elems)
76
+ if variant == "marks":
77
+ opts = [f"click element {i + 1}" for i in range(len(elems))]
78
+ else:
79
+ opts = [f"click element {i + 1}: {e['text'] or e['kind']}" + (f" ({e['kind']})" if e["text"] else "") for i, e in enumerate(elems)]
80
+ opts.append("the task is complete")
81
+ state = ["Browser screenshot. Every clickable element has a numbered red box.\n", {"image": to_url(im)},
82
+ f"\nTask: {task}\nActions taken so far: {'; '.join(hist) if hist else 'none'}"]
83
+ t0 = time.time()
84
+ r = requests.post(API, json={"kind": "choice", "state": state, "question": "What should be done next to complete the task?",
85
+ "options": opts, "thinking": "off"}, timeout=120).json()
86
+ lat.append(time.time() - t0)
87
+ probs = r["probabilities"]; k = max(range(len(probs)), key=probs.__getitem__)
88
+ if frames is not None:
89
+ frames.append({"img": im, "task": task, "step": step + 1, "opts": opts, "probs": probs, "k": k, "ms": 1000 * lat[-1], "variant": variant})
90
+ steps.append({"options": len(opts), "choice": opts[k], "p": probs[k]})
91
+ if k == len(elems):
92
+ stopped = True
93
+ break
94
+ e = elems[k]
95
+ hist.append(f"clicked element {k + 1}" + (f" ({e['text'] or e['kind']})" if variant != "marks" else ""))
96
+ page.mouse.click(e["x"] + e["w"] / 2, e["y"] + e["h"] / 2); page.wait_for_timeout(30)
97
+ S = page.evaluate("() => window.S")
98
+ ok = goal_met(app, S, goal)
99
+ return {"app": app, "seed": seed, "variant": variant, "task": task, "success": bool(ok and stopped), "goal_met": bool(ok),
100
+ "stopped": stopped, "steps": len(steps), "trace": steps, "latency_s": lat}
101
+
102
+
103
+ def worker(jobs):
104
+ from playwright.sync_api import sync_playwright
105
+ out = []
106
+ with sync_playwright() as p:
107
+ b = p.chromium.launch(); page = b.new_page(viewport={"width": VW, "height": VH})
108
+ for app, seed, variant in jobs:
109
+ out.append(episode(page, app, seed, variant))
110
+ b.close()
111
+ return out
112
+
113
+
114
+ def run(n=10, procs=6):
115
+ from multiprocessing import Pool
116
+ jobs = [(a, s, v) for v in ("marks", "marks+text") for a in ("mail", "shop", "settings") for s in range(1, n + 1)]
117
+ chunks = [jobs[i::procs] for i in range(procs)]
118
+ t0 = time.time()
119
+ with Pool(procs) as pool:
120
+ res = [r for part in pool.map(worker, chunks) for r in part]
121
+ summ = {}
122
+ for v in ("marks", "marks+text"):
123
+ for a in ("mail", "shop", "settings", "all"):
124
+ rs = [r for r in res if r["variant"] == v and (a == "all" or r["app"] == a)]
125
+ lat = sorted(x for r in rs for x in r["latency_s"])
126
+ summ[f"{v}/{a}"] = {"episodes": len(rs), "success": sum(r["success"] for r in rs) / len(rs),
127
+ "mean_steps": sum(r["steps"] for r in rs) / len(rs), "median_ms": 1000 * lat[len(lat) // 2]}
128
+ json.dump({"summary": summ, "seconds": time.time() - t0, "episodes": res}, open(os.path.join(HERE, "cu_results.json"), "w"), indent=1)
129
+ for k, v in summ.items():
130
+ print(f"{k:22s} success {v['success']:.0%} steps {v['mean_steps']:.1f} median {v['median_ms']:.0f} ms/decision (n={v['episodes']})")
131
+
132
+
133
+ def panel(fr, W=1280, H=720):
134
+ """Video frame: marked screenshot on the left, task + option probabilities on the right."""
135
+ can = Image.new("RGB", (W, H), (250, 250, 252)); d = ImageDraw.Draw(can)
136
+ f, fb, fs = ImageFont.truetype(FONT, 17), ImageFont.truetype(FONTB, 19), ImageFont.truetype(FONT, 15)
137
+ shot = fr["img"].resize((880, int(880 * VH / VW))); can.paste(shot, (10, 80))
138
+ d.text((14, 12), f"{os.environ.get('JEV_LABEL', 'JEV-9B')} System 1 · computer use: screenshot → which element to click", fill=(20, 30, 50), font=fb)
139
+ d.text((14, 44), f"Task: {fr['task']}", fill=(40, 40, 40), font=f)
140
+ x0 = 905; d.text((x0, 86), f"Step {fr['step']} · one forward pass · {fr['ms']:.0f} ms", fill=(20, 30, 50), font=fb)
141
+ order = sorted(range(len(fr["probs"])), key=lambda i: -fr["probs"][i])[:6]
142
+ y = 126
143
+ for i in order:
144
+ p = fr["probs"][i]; lab = fr["opts"][i].replace("click element", "click").replace("the task is complete", "task complete")
145
+ lab = lab if len(lab) <= 34 else lab[:33] + "…"
146
+ d.text((x0, y), lab, fill=(30, 30, 30) if i != fr["k"] else (200, 0, 0), font=fs)
147
+ d.rectangle((x0, y + 21, x0 + 340, y + 33), fill=(225, 228, 235))
148
+ d.rectangle((x0, y + 21, x0 + int(340 * p), y + 33), fill=(200, 0, 0) if i == fr["k"] else (120, 140, 170))
149
+ d.text((x0 + 300, y), f"{p:.2f}", fill=(30, 30, 30), font=fs)
150
+ y += 50
151
+ note = ("options = numbered marks only\n(the model must read the screenshot)" if fr.get("variant") == "marks" else
152
+ "options = mark number + visible text;\ncolour swatches and switches have no\ntext: the screenshot decides")
153
+ d.text((x0, H - 80), note, fill=(90, 90, 90), font=fs)
154
+ return can
155
+
156
+
157
+ def record(app, seed, variant, out):
158
+ from playwright.sync_api import sync_playwright
159
+ frames = []
160
+ with sync_playwright() as p:
161
+ b = p.chromium.launch(); page = b.new_page(viewport={"width": VW, "height": VH})
162
+ res = episode(page, app, seed, variant, frames); b.close()
163
+ tmp = tempfile.mkdtemp()
164
+ n = 0
165
+ for fr in frames:
166
+ im = panel(fr)
167
+ for _ in range(3): # 1.5 s per decision at 2 fps
168
+ im.save(f"{tmp}/{n:04d}.png"); n += 1
169
+ subprocess.run(["ffmpeg", "-y", "-loglevel", "error", "-framerate", "2", "-i", f"{tmp}/%04d.png", "-c:v", "libx264", "-pix_fmt", "yuv420p",
170
+ "-vf", "fps=10", out], check=True)
171
+ print(out, "success" if res["success"] else "FAILED", res["steps"], "steps", [round(1000 * x) for x in res["latency_s"]], "ms")
172
+
173
+
174
+ if __name__ == "__main__":
175
+ if sys.argv[1] == "run":
176
+ run(int(sys.argv[2]) if len(sys.argv) > 2 else 10)
177
+ else:
178
+ record(sys.argv[2], int(sys.argv[3]), sys.argv[4], sys.argv[5])
vl/demos/robot_arm.py ADDED
@@ -0,0 +1,323 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Robot-arm pick-and-place with JEV-9B-VL System 1 (POST /v1/decide), simulated in MuJoCo.
2
+
3
+ A gantry arm with a two-finger gripper above a table with three coloured cubes and a yellow tray. Every step the model
4
+ sees two camera images (front and top; the top view shows a black dot under the gripper) and the gripper state, and
5
+ picks one of 8 actions in one forward pass. Grasping is simplified: closing the gripper low over a cube attaches the
6
+ cube; opening releases it and it falls under MuJoCo physics.
7
+
8
+ MUJOCO_GL=glfw DISPLAY=:99 python3 robot_arm.py run 20
9
+ MUJOCO_GL=glfw DISPLAY=:99 python3 robot_arm.py record SEED OUT.mp4
10
+ """
11
+ import base64, io, json, os, random, subprocess, sys, tempfile, time
12
+ os.environ.setdefault("MUJOCO_GL", "glfw"); os.environ.setdefault("DISPLAY", ":99")
13
+ import numpy as np, mujoco, requests
14
+ from PIL import Image, ImageDraw, ImageFont
15
+
16
+ API = os.environ.get("JEV_URL", "http://localhost:8000") + "/v1/decide"
17
+ HERE = os.path.dirname(os.path.abspath(__file__))
18
+ FONT, FONTB = "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf", "/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf"
19
+ ACTIONS = ["move left", "move right", "move forward (away from the front camera)", "move backward (toward the front camera)",
20
+ "lower the gripper", "raise the gripper", "close the gripper (grasp)", "open the gripper (release)"]
21
+ COLORS = {"red": "0.85 0.15 0.15 1", "green": "0.15 0.65 0.25 1", "blue": "0.15 0.3 0.85 1"}
22
+ STEP, ZSTEP, ZTOP, ZLOW, HALF = 0.04, 0.05, 0.25, 0.05, 0.02
23
+ MAX_STEPS = 45
24
+
25
+
26
+ def xml(cubes, tray):
27
+ cube_xml = "".join(f'<body name="{n}" pos="{x} {y} {HALF + 0.001}"><freejoint name="j_{n}"/>'
28
+ f'<geom type="box" size="{HALF} {HALF} {HALF}" rgba="{COLORS[n]}" mass="0.05" friction="1 0.01 0.001"/></body>'
29
+ for n, (x, y) in cubes.items())
30
+ tx, ty = tray
31
+ return f"""<mujoco><option timestep="0.002"/>
32
+ <visual><headlight ambient=".3 .3 .3" diffuse=".3 .3 .3"/><global offwidth="640" offheight="480"/><quality shadowsize="2048"/></visual>
33
+ <asset><texture name="g" type="2d" builtin="checker" rgb1=".72 .70 .66" rgb2=".62 .60 .56" width="256" height="256"/>
34
+ <material name="g" texture="g" texrepeat="6 6"/></asset>
35
+ <worldbody>
36
+ <light pos="0 0 1.6" dir="0 0 -1" diffuse=".45 .45 .45" castshadow="false"/>
37
+ <geom type="plane" size=".45 .45 .01" material="g"/>
38
+ <geom type="box" size=".012 .012 .22" pos="-.42 .42 .22" rgba=".35 .35 .4 1" group="2"/><geom type="box" size=".012 .012 .22" pos=".42 .42 .22" rgba=".35 .35 .4 1" group="2"/>
39
+ <geom type="box" size=".43 .012 .012" pos="0 .42 .44" rgba=".35 .35 .4 1" group="2"/>
40
+ <body name="tray" pos="{tx} {ty} 0">
41
+ <geom type="box" size=".065 .065 .003" pos="0 0 .003" rgba=".95 .8 .1 1"/>
42
+ <geom type="box" size=".065 .004 .02" pos="0 .061 .02" rgba=".95 .8 .1 1"/><geom type="box" size=".065 .004 .02" pos="0 -.061 .02" rgba=".95 .8 .1 1"/>
43
+ <geom type="box" size=".004 .065 .02" pos=".061 0 .02" rgba=".95 .8 .1 1"/><geom type="box" size=".004 .065 .02" pos="-.061 0 .02" rgba=".95 .8 .1 1"/>
44
+ </body>
45
+ {cube_xml}
46
+ <body name="dot" mocap="true" pos="0 0 .0015"><geom type="cylinder" size=".012 .001" rgba="0 0 0 0.85" contype="0" conaffinity="0"/></body>
47
+ <body name="arm" mocap="true" pos="0 0 {ZTOP}">
48
+ <geom type="cylinder" size=".012 .2" pos="0 0 .23" rgba=".55 .55 .6 1" contype="0" conaffinity="0" group="2"/>
49
+ <geom type="box" size=".035 .02 .015" rgba=".25 .25 .3 1" contype="0" conaffinity="0" group="2"/>
50
+ <camera name="wrist" pos="0 0 -0.02" xyaxes="1 0 0 0 1 0" fovy="70"/>
51
+ </body>
52
+ <body name="fl" mocap="true" pos="-.03 0 {ZTOP - .035}"><geom type="box" size=".005 .015 .03" rgba=".15 .15 .15 1" contype="0" conaffinity="0" group="2"/></body>
53
+ <body name="fr" mocap="true" pos=".03 0 {ZTOP - .035}"><geom type="box" size=".005 .015 .03" rgba=".15 .15 .15 1" contype="0" conaffinity="0" group="2"/></body>
54
+ <camera name="front" pos="0 -0.78 0.62" xyaxes="1 0 0 0 0.62 0.78"/>
55
+ <camera name="top" pos="0 0 1.05" xyaxes="1 0 0 0 1 0"/>
56
+ </worldbody></mujoco>"""
57
+
58
+
59
+ class Env:
60
+ def __init__(self, seed):
61
+ r = random.Random(seed)
62
+ while True: # cubes and tray apart from each other, inside reach
63
+ pts = [(round(r.uniform(-.3, .3), 3), round(r.uniform(-.25, .25), 3)) for _ in range(4)]
64
+ if all(np.hypot(a[0] - b[0], a[1] - b[1]) > .12 for i, a in enumerate(pts) for b in pts[i + 1:]):
65
+ break
66
+ self.cubes = dict(zip(COLORS, pts[:3])); self.tray = pts[3]
67
+ self.target = r.choice(list(COLORS))
68
+ self.m = mujoco.MjModel.from_xml_string(xml(self.cubes, self.tray)); self.d = mujoco.MjData(self.m)
69
+ self.g = np.array([round(r.uniform(-.2, .2), 2), round(r.uniform(-.2, .2), 2), ZTOP]); self.closed = False; self.held = None
70
+ self.ren = mujoco.Renderer(self.m, 300, 400); self.ren_top = mujoco.Renderer(self.m, 300, 300); self.ren_w = mujoco.Renderer(self.m, 300, 300)
71
+ self._pose(); self.settle(200)
72
+
73
+ def mocap(self, name):
74
+ return self.m.body_mocapid[mujoco.mj_name2id(self.m, mujoco.mjtObj.mjOBJ_BODY, name)]
75
+
76
+ def cube_q(self, n):
77
+ return self.m.jnt_qposadr[mujoco.mj_name2id(self.m, mujoco.mjtObj.mjOBJ_JOINT, f"j_{n}")]
78
+
79
+ def cube_pos(self, n):
80
+ a = self.cube_q(n); return self.d.qpos[a:a + 3].copy()
81
+
82
+ def _pose(self):
83
+ x, y, z = self.g; w = 0.022 if self.closed else 0.03
84
+ self.d.mocap_pos[self.mocap("arm")] = [x, y, z]; self.d.mocap_pos[self.mocap("dot")] = [x, y, .0015]
85
+ self.d.mocap_pos[self.mocap("fl")] = [x - w, y, z - .035]; self.d.mocap_pos[self.mocap("fr")] = [x + w, y, z - .035]
86
+ if self.held:
87
+ a = self.cube_q(self.held); self.d.qpos[a:a + 3] = [x, y, z - .045]; self.d.qpos[a + 3:a + 7] = [1, 0, 0, 0]
88
+ va = self.m.jnt_dofadr[mujoco.mj_name2id(self.m, mujoco.mjtObj.mjOBJ_JOINT, f"j_{self.held}")]; self.d.qvel[va:va + 6] = 0
89
+
90
+ def settle(self, n=300):
91
+ for _ in range(n):
92
+ self._pose(); mujoco.mj_step(self.m, self.d)
93
+
94
+ def step(self, a):
95
+ x, y, z = self.g
96
+ if a == 0: x -= STEP
97
+ elif a == 1: x += STEP
98
+ elif a == 2: y += STEP
99
+ elif a == 3: y -= STEP
100
+ elif a == 4: z = max(ZLOW, z - ZSTEP)
101
+ elif a == 5: z = min(ZTOP, z + ZSTEP)
102
+ elif a == 6:
103
+ self.closed = True
104
+ if self.held is None and z <= ZLOW + 1e-6:
105
+ near = [n for n in COLORS if np.hypot(*(self.cube_pos(n)[:2] - [x, y])) < 0.03 and self.cube_pos(n)[2] < .06]
106
+ self.held = near[0] if near else None
107
+ elif a == 7:
108
+ self.closed = False; self.held = None
109
+ self.g = np.clip([x, y, z], [-.38, -.32, ZLOW], [.38, .36, ZTOP]); self.settle()
110
+
111
+ def done(self):
112
+ p = self.cube_pos(self.target)
113
+ return self.held is None and abs(p[0] - self.tray[0]) < .06 and abs(p[1] - self.tray[1]) < .06 and p[2] < .06
114
+
115
+ def wrist(self):
116
+ opt = mujoco.MjvOption(); opt.geomgroup[2] = 0
117
+ self.ren_w.update_scene(self.d, camera="wrist", scene_option=opt); im = Image.fromarray(self.ren_w.render())
118
+ d = ImageDraw.Draw(im); c = im.width // 2
119
+ d.line((c - 18, c, c + 18, c), fill=(255, 255, 255), width=2); d.line((c, c - 18, c, c + 18), fill=(255, 255, 255), width=2)
120
+ return im
121
+
122
+ def images(self):
123
+ self.ren.update_scene(self.d, camera="front"); f = Image.fromarray(self.ren.render())
124
+ opt = mujoco.MjvOption(); opt.geomgroup[2] = 0 # hide gantry and arm column from above
125
+ self.ren_top.update_scene(self.d, camera="top", scene_option=opt); t = Image.fromarray(self.ren_top.render())
126
+ return f, t
127
+
128
+
129
+ def url(im):
130
+ b = io.BytesIO(); im.save(b, "PNG"); return "data:image/png;base64," + base64.b64encode(b.getvalue()).decode()
131
+
132
+
133
+ def episode(seed, frames=None):
134
+ env = Env(seed); hist, lat, ok = [], [], False
135
+ for step in range(MAX_STEPS):
136
+ f, t = env.images()
137
+ grip = ("closed, holding the " + env.held + " cube") if env.held else ("closed, holding nothing" if env.closed else "open")
138
+ state = ["Front camera:", {"image": url(f)}, "\nTop camera (the black dot is directly under the gripper; up in this image is forward):",
139
+ {"image": url(t)}, f"\nTask: pick up the {env.target} cube and put it in the yellow tray.\nGripper: {grip}.\n"
140
+ f"Gripper height: {'raised' if env.g[2] >= ZTOP - 1e-6 else ('lowest (at cube height)' if env.g[2] <= ZLOW + 1e-6 else 'in between')}.\n"
141
+ f"Last actions: {'; '.join(hist[-3:]) if hist else 'none'}"]
142
+ t0 = time.time()
143
+ r = requests.post(API, json={"kind": "choice", "state": state, "question": "Which action should the robot take next?",
144
+ "options": ACTIONS, "thinking": "off"}, timeout=120).json()
145
+ lat.append(time.time() - t0)
146
+ p = r["probabilities"]; a = int(np.argmax(p))
147
+ if frames is not None:
148
+ frames.append({"front": f, "top": t, "probs": p, "a": a, "step": step + 1, "ms": 1000 * lat[-1], "task": env.target, "grip": grip})
149
+ env.step(a); hist.append(ACTIONS[a].split(" (")[0])
150
+ if env.done():
151
+ ok = True
152
+ if frames is not None:
153
+ f, t = env.images(); frames.append({"front": f, "top": t, "probs": None, "a": None, "step": step + 1, "ms": 0, "task": env.target, "grip": "open"})
154
+ break
155
+ held_ever = any(h == "close the gripper" for h in hist)
156
+ return {"seed": seed, "target": env.target, "success": ok, "steps": len(hist), "actions": hist, "latency_s": lat}
157
+
158
+
159
+ def run(n=20):
160
+ res = [episode(s) for s in range(1, n + 1)]
161
+ lat = sorted(x for r in res for x in r["latency_s"])
162
+ summ = {"episodes": n, "success": sum(r["success"] for r in res) / n, "mean_steps_success": float(np.mean([r["steps"] for r in res if r["success"]] or [0])),
163
+ "median_ms": 1000 * lat[len(lat) // 2]}
164
+ json.dump({"summary": summ, "episodes": res}, open(os.path.join(HERE, "arm_results.json"), "w"), indent=1)
165
+ print(json.dumps(summ)); [print(r["seed"], r["target"], r["success"], r["steps"], " ".join(a.split()[0][:2] + a.split()[-1][:2] for a in r["actions"])) for r in res]
166
+
167
+
168
+ def frame(fr, W=1280, H=720):
169
+ can = Image.new("RGB", (W, H), (248, 248, 250)); d = ImageDraw.Draw(can)
170
+ f, fb, fs = ImageFont.truetype(FONT, 17), ImageFont.truetype(FONTB, 19), ImageFont.truetype(FONT, 15)
171
+ d.text((14, 12), f"{os.environ.get('JEV_LABEL', 'JEV-9B')} System 1 · robot arm: camera images → action probabilities", fill=(20, 30, 50), font=fb)
172
+ d.text((14, 44), f"Task: pick up the {fr['task']} cube and put it in the yellow tray · gripper: {fr['grip']}", fill=(40, 40, 40), font=f)
173
+ can.paste(fr["front"].resize((560, 420)), (14, 84)); can.paste(fr["top"].resize((420, 420)), (584, 84))
174
+ d.text((14, 510), "front camera", fill=(90, 90, 90), font=fs); d.text((584, 510), "top camera", fill=(90, 90, 90), font=fs)
175
+ x0 = 1018
176
+ if fr["probs"] is None:
177
+ d.text((x0, 100), "Task complete", fill=(0, 140, 60), font=fb)
178
+ else:
179
+ d.text((x0, 90), f"Step {fr['step']} · {fr['ms']:.0f} ms", fill=(20, 30, 50), font=fb)
180
+ y = 130
181
+ for i, a in enumerate(ACTIONS):
182
+ p = fr["probs"][i]; c = (200, 0, 0) if i == fr["a"] else (120, 140, 170)
183
+ d.text((x0, y), a.split(" (")[0], fill=(200, 0, 0) if i == fr["a"] else (30, 30, 30), font=fs)
184
+ d.rectangle((x0, y + 21, x0 + 240, y + 31), fill=(225, 228, 235)); d.rectangle((x0, y + 21, x0 + int(240 * p), y + 31), fill=c)
185
+ d.text((x0 + 200, y), f"{p:.2f}", fill=(30, 30, 30), font=fs); y += 46
186
+ return can
187
+
188
+
189
+ def record(seed, out):
190
+ frames = []; res = episode(seed, frames); tmp = tempfile.mkdtemp(); n = 0
191
+ for fr in frames:
192
+ im = frame(fr)
193
+ for _ in range(2 if fr["probs"] is not None else 6):
194
+ im.save(f"{tmp}/{n:04d}.png"); n += 1
195
+ subprocess.run(["ffmpeg", "-y", "-loglevel", "error", "-framerate", "4", "-i", f"{tmp}/%04d.png", "-c:v", "libx264", "-pix_fmt", "yuv420p",
196
+ "-vf", "fps=12", out], check=True)
197
+ print(out, "success" if res["success"] else "FAILED", res["steps"], "steps")
198
+
199
+
200
+ if __name__ == "__main__" and sys.argv[1] in ("run", "view", "record"):
201
+ if sys.argv[1] == "run":
202
+ run(int(sys.argv[2]) if len(sys.argv) > 2 else 20)
203
+ elif sys.argv[1] == "view":
204
+ e = Env(int(sys.argv[2])); f, t = e.images(); f.save("/tmp/opencode/arm_front.png"); t.save("/tmp/opencode/arm_top.png"); print(e.target, e.cubes, e.tray, e.g)
205
+ else:
206
+ record(int(sys.argv[2]), sys.argv[3])
207
+
208
+
209
+ # ---------------------------------------------------------------- v2: System 1 visual servoing (binary decisions)
210
+ ZCARRY = 0.15
211
+
212
+
213
+ def ask(imgs, q, opts):
214
+ state = []
215
+ for title, im in imgs:
216
+ state += [title, {"image": url(im)}, "\n"]
217
+ t0 = time.time()
218
+ p = requests.post(API, json={"kind": "choice", "state": state, "question": q, "options": opts, "thinking": "off"},
219
+ timeout=120).json()["probabilities"]
220
+ return p, time.time() - t0
221
+
222
+
223
+ def zoom(env, t, half=60):
224
+ """Crop of the top image centred on the black dot (projected position), upscaled 2.5x."""
225
+ f = (t.height / 2) / np.tan(np.radians(45 / 2)); u = t.width / 2 + f * env.g[0] / 1.05; v = t.height / 2 - f * env.g[1] / 1.05
226
+ u, v = int(np.clip(u, half, t.width - half)), int(np.clip(v, half, t.height - half))
227
+ return t.crop((u - half, v - half, u + half, v + half)).resize((300, 300), Image.BICUBIC)
228
+
229
+
230
+ def servo(env, what, ref, log, frames, phase, max_iter=36):
231
+ """Move the gripper over `what` by an adaptive bisection on each axis, driven only by two System 1 decisions per
232
+ step: is `what` left or right of `ref`, and above or below it, in the top camera image (plus a zoomed view of the
233
+ area around the gripper once the steps are small). A flipped answer halves that axis' step; three equal answers in a
234
+ row double it."""
235
+ step, hist = {"x": 0.08, "y": 0.08}, {"x": [], "y": []}
236
+ for it in range(max_iter):
237
+ f, t = env.images()
238
+ imgs = [("Top camera image:", t)]
239
+ if max(step.values()) <= 0.04:
240
+ imgs.append(("Zoomed top view around the black dot (2.5x):", zoom(env, t)))
241
+ plr, l1 = ask(imgs, f"In the top camera view, is the {what} to the left or to the right of the {ref}?", ["left", "right"])
242
+ pud, l2 = ask(imgs, f"In the top camera view, is the {what} above or below the {ref}?", ["above", "below"])
243
+ sx = -1 if plr[0] > plr[1] else 1; sy = 1 if pud[0] > pud[1] else -1
244
+ for ax, sgn in (("x", sx), ("y", sy)):
245
+ h = hist[ax]
246
+ if h and sgn != h[-1]:
247
+ step[ax] /= 2
248
+ elif len(h) >= 2 and h[-1] == h[-2] == sgn:
249
+ step[ax] = min(0.08, step[ax] * 2)
250
+ h.append(sgn)
251
+ log["decisions"] += 2; log["lat"] += [l1, l2]
252
+ if frames is not None:
253
+ frames.append({"front": f, "top": t, "phase": phase, "plr": plr, "pud": pud, "ms": 1000 * (l1 + l2), "task": env.target,
254
+ "grip": "holding the " + env.held + " cube" if env.held else "open"})
255
+ if step["x"] < 0.012 and step["y"] < 0.012:
256
+ return
257
+ env.g = np.clip(env.g + [sx * step["x"], sy * step["y"], 0], [-.38, -.32, ZLOW], [.38, .36, ZTOP]); env.settle(150)
258
+
259
+
260
+ def episode2(seed, frames=None):
261
+ env = Env(seed); log = {"decisions": 0, "lat": []}
262
+ servo(env, f"{env.target} cube", "black dot", log, frames, "1 · move over the cube")
263
+ c = env.cube_pos(env.target); log["err_cube_cm"] = round(100 * float(np.hypot(c[0] - env.g[0], c[1] - env.g[1])), 1)
264
+ env.g[2] = ZLOW; env.settle(); env.step(6) # lower and grasp
265
+ grasped = env.held == env.target
266
+ env.g[2] = ZCARRY; env.settle()
267
+ servo(env, "yellow tray", f"{env.target} cube held by the gripper", log, frames, "2 · carry it over the tray")
268
+ log["err_tray_cm"] = round(100 * float(np.hypot(env.tray[0] - env.g[0], env.tray[1] - env.g[1])), 1)
269
+ env.step(7); env.settle(600) # release
270
+ ok = env.done()
271
+ if frames is not None:
272
+ f, t = env.images(); frames.append({"front": f, "top": t, "phase": "done" if ok else "missed", "plr": None, "pud": None, "ms": 0,
273
+ "task": env.target, "grip": "open"})
274
+ return {"seed": seed, "target": env.target, "grasped": bool(grasped), "success": bool(ok), "decisions": log["decisions"], "latency_s": log["lat"],
275
+ "err_cube_cm": log.get("err_cube_cm"), "err_tray_cm": log.get("err_tray_cm")}
276
+
277
+
278
+ def run2(n=20):
279
+ t0 = time.time(); res = [episode2(s) for s in range(1, n + 1)]
280
+ lat = sorted(x for r in res for x in r["latency_s"])
281
+ summ = {"episodes": n, "success": sum(r["success"] for r in res) / n, "grasped": sum(r["grasped"] for r in res) / n,
282
+ "mean_decisions": float(np.mean([r["decisions"] for r in res])), "median_ms": 1000 * lat[len(lat) // 2], "seconds": time.time() - t0}
283
+ json.dump({"summary": summ, "episodes": res}, open(os.path.join(HERE, "arm2_results.json"), "w"), indent=1)
284
+ print(json.dumps(summ)); [print(r["seed"], r["target"], "grasp", r["grasped"], "success", r["success"], r["decisions"], "err cube/tray cm", r["err_cube_cm"], r["err_tray_cm"]) for r in res]
285
+
286
+
287
+ def frame2(fr, W=1280, H=720):
288
+ can = Image.new("RGB", (W, H), (248, 248, 250)); d = ImageDraw.Draw(can)
289
+ f, fb, fs = ImageFont.truetype(FONT, 17), ImageFont.truetype(FONTB, 19), ImageFont.truetype(FONT, 16)
290
+ d.text((14, 12), f"{os.environ.get('JEV_LABEL', 'JEV-9B')} System 1 · robot arm: camera image → two yes/no-style decisions per step", fill=(20, 30, 50), font=fb)
291
+ d.text((14, 44), f"Task: pick up the {fr['task']} cube and put it in the yellow tray · gripper: {fr['grip']}", fill=(40, 40, 40), font=f)
292
+ can.paste(fr["front"].resize((560, 420)), (14, 84)); can.paste(fr["top"].resize((420, 420)), (584, 84))
293
+ d.text((14, 510), "front camera", fill=(90, 90, 90), font=fs); d.text((584, 510), "top camera (what System 1 reads)", fill=(90, 90, 90), font=fs)
294
+ x0 = 1016; d.text((x0, 90), fr["phase"].replace(" · ", ". "), fill=(20, 30, 50), font=ImageFont.truetype(FONTB, 16))
295
+ if fr["plr"] is None:
296
+ d.text((x0, 130), "Task complete" if fr["phase"] == "done" else "Missed", fill=(0, 140, 60) if fr["phase"] == "done" else (200, 0, 0), font=fb)
297
+ else:
298
+ d.text((x0, 125), f"{fr['ms']:.0f} ms for 2 decisions", fill=(60, 60, 60), font=fs)
299
+ y = 165
300
+ for title, labs, p in (("target left or right?", ["left", "right"], fr["plr"]), ("target above or below?", ["above", "below"], fr["pud"])):
301
+ d.text((x0, y), title, fill=(30, 30, 30), font=fs); y += 26
302
+ for lab, v in zip(labs, p):
303
+ c = (200, 0, 0) if v == max(p) else (120, 140, 170)
304
+ d.text((x0, y), lab, fill=(30, 30, 30), font=fs); d.rectangle((x0 + 62, y + 4, x0 + 62 + int(130 * v), y + 16), fill=c)
305
+ d.text((x0 + 200, y), f"{v:.2f}", fill=(30, 30, 30), font=fs); y += 26
306
+ y += 18
307
+ d.text((x0, y + 10), "step size halves each time\na decision flips (bisection)", fill=(90, 90, 90), font=fs)
308
+ return can
309
+
310
+
311
+ def record2(seed, out):
312
+ frames = []; res = episode2(seed, frames); tmp = tempfile.mkdtemp(); n = 0
313
+ for fr in frames:
314
+ im = frame2(fr)
315
+ for _ in range(2 if fr["plr"] is not None else 8):
316
+ im.save(f"{tmp}/{n:04d}.png"); n += 1
317
+ subprocess.run(["ffmpeg", "-y", "-loglevel", "error", "-framerate", "4", "-i", f"{tmp}/%04d.png", "-c:v", "libx264", "-pix_fmt", "yuv420p",
318
+ "-vf", "fps=12", out], check=True)
319
+ print(out, "success" if res["success"] else "FAILED", res["decisions"], "decisions")
320
+
321
+
322
+ if __name__ == "__main__" and sys.argv[1] in ("run2", "record2"):
323
+ run2(int(sys.argv[2])) if sys.argv[1] == "run2" else record2(int(sys.argv[2]), sys.argv[3])
vl/demos/webapp.html ADDED
@@ -0,0 +1,119 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <!doctype html>
2
+ <html><head><meta charset="utf-8"><title>Acme Workspace</title>
3
+ <style>
4
+ body { font-family: "DejaVu Sans", Arial, sans-serif; margin: 0; background: #f4f6f9; color: #1d2733; }
5
+ .top { background: #24324a; color: #fff; padding: 12px 20px; display: flex; align-items: center; gap: 18px; font-size: 18px; }
6
+ .top .brand { font-weight: bold; margin-right: auto; }
7
+ .main { padding: 18px 24px; }
8
+ .btn { display: inline-block; padding: 9px 16px; margin: 4px 6px 4px 0; border-radius: 6px; border: 1px solid #c5ccd6; background: #fff; cursor: pointer; font-size: 15px; }
9
+ .top .btn { color: #1d2733; }
10
+ .btn.primary { background: #2f6fde; color: #fff; border-color: #2f6fde; }
11
+ .btn.danger { color: #b3261e; }
12
+ .row { background: #fff; border: 1px solid #dde2e8; border-radius: 6px; padding: 10px 14px; margin: 6px 0; cursor: pointer; display: flex; gap: 16px; font-size: 15px; }
13
+ .row b { width: 170px; }
14
+ .grid { display: grid; grid-template-columns: repeat(3, 230px); gap: 14px; }
15
+ .card { background: #fff; border: 1px solid #dde2e8; border-radius: 8px; padding: 12px; cursor: pointer; font-size: 15px; }
16
+ .card .img { height: 70px; border-radius: 6px; margin-bottom: 8px; }
17
+ .swatch { display: inline-block; width: 34px; height: 34px; border-radius: 50%; margin: 6px 10px 6px 0; border: 3px solid #fff; box-shadow: 0 0 0 1px #aaa; cursor: pointer; }
18
+ .swatch.sel { box-shadow: 0 0 0 3px #1d2733; }
19
+ .size { display: inline-block; padding: 8px 14px; border: 1px solid #aab; margin-right: 8px; border-radius: 4px; cursor: pointer; background: #fff; }
20
+ .size.sel { background: #1d2733; color: #fff; }
21
+ .side { width: 190px; float: left; }
22
+ .side .item { padding: 10px 12px; margin: 4px 0; border-radius: 6px; cursor: pointer; background: #fff; border: 1px solid #dde2e8; }
23
+ .side .item.sel { background: #dfe9fb; border-color: #2f6fde; }
24
+ .panel { margin-left: 215px; background: #fff; border: 1px solid #dde2e8; border-radius: 8px; padding: 16px 20px; min-height: 300px; }
25
+ .toggle { display: flex; align-items: center; justify-content: space-between; width: 470px; padding: 10px 0; border-bottom: 1px solid #eee; font-size: 15px; }
26
+ .sw { width: 52px; height: 28px; border-radius: 14px; background: #c9ced6; position: relative; cursor: pointer; }
27
+ .sw::after { content: ""; position: absolute; width: 22px; height: 22px; border-radius: 50%; background: #fff; top: 3px; left: 3px; }
28
+ .sw.on { background: #2e9d57; } .sw.on::after { left: 27px; }
29
+ .toast { margin-top: 14px; color: #2e9d57; font-weight: bold; }
30
+ h2 { margin: 6px 0 12px; font-size: 21px; }
31
+ </style></head>
32
+ <body><div id="app"></div>
33
+ <script>
34
+ // A tiny three-app workspace (mail, shop, settings) driven by window.S; every interactive element carries data-click.
35
+ const P = new URLSearchParams(location.search);
36
+ let seed = +(P.get("seed") || 1);
37
+ function rnd() { seed = (seed * 1103515245 + 12345) % 2147483648; return seed / 2147483648; }
38
+ function shuffle(a) { for (let i = a.length - 1; i > 0; i--) { const j = Math.floor(rnd() * (i + 1)); [a[i], a[j]] = [a[j], a[i]]; } return a; }
39
+ const app = P.get("app");
40
+ const S = window.S = { app, view: "home", archived: [], deleted: [], cart: [], ordered: false, sel: {}, section: null, saved: {}, toggles: {} };
41
+ const NAMES = shuffle(["Alice Chen", "Bob Martin", "Carla Diaz", "David Kim", "Emma Rossi", "Farid Khan", "Grace Lee", "Hugo Weber"]).slice(0, 6);
42
+ const TOPICS = shuffle(["invoice for March", "team offsite", "contract renewal", "server outage", "lunch on Friday", "design review", "travel booking", "quarterly report"]).slice(0, 6);
43
+ const MAILS = NAMES.map((n, i) => ({ id: i, from: n, subject: TOPICS[i] }));
44
+ const PRODUCTS = shuffle(["T-shirt", "Hoodie", "Sneakers", "Backpack", "Cap", "Jacket"]).map((n, i) => ({ id: i, name: n, price: 19 + 7 * i }));
45
+ const COLORS = { red: "#d93a3a", blue: "#2f6fde", green: "#2e9d57", black: "#222222", yellow: "#e8c21b" };
46
+ const SIZES = ["S", "M", "L", "XL"];
47
+ const SECTIONS = { Profile: ["Show online status", "Public profile"], Security: ["Two-factor authentication", "Login alerts"],
48
+ Notifications: ["Email digests", "Push notifications"], Appearance: ["Dark mode", "Compact layout"] };
49
+ const secOrder = shuffle(Object.keys(SECTIONS));
50
+ for (const s in SECTIONS) for (const t of SECTIONS[s]) S.toggles[t] = false;
51
+ window.WORLD = { MAILS, PRODUCTS, COLORS: Object.keys(COLORS), SIZES, SECTIONS };
52
+ function el(html) { document.getElementById("app").innerHTML = html; }
53
+ function click(k) { return `data-click="${k}"`; }
54
+ function header(title) { return `<div class="top"><span class="brand">${title}</span><span class="btn" ${click("nav:help")}>Help</span><span class="btn" ${click("nav:account")}>Account</span></div>`; }
55
+ function render() {
56
+ if (app === "mail") {
57
+ if (S.view === "home") {
58
+ const rows = MAILS.filter(m => !S.archived.includes(m.id) && !S.deleted.includes(m.id))
59
+ .map(m => `<div class="row" ${click("open:" + m.id)}><b>${m.from}</b><span>${m.subject}</span></div>`).join("");
60
+ el(header("Acme Mail") + `<div class="main"><h2>Inbox</h2><span class="btn primary" ${click("compose")}>Compose</span>${rows}</div>`);
61
+ } else {
62
+ const m = MAILS[S.open];
63
+ el(header("Acme Mail") + `<div class="main"><span class="btn" ${click("back")}>&larr; Back to inbox</span><h2>${m.subject}</h2>
64
+ <p>From: <b>${m.from}</b></p><p style="width:640px">Hi, please see the details about the ${m.subject} below. Let me know if you have questions.</p>
65
+ <span class="btn" ${click("reply")}>Reply</span><span class="btn" ${click("forward")}>Forward</span>
66
+ <span class="btn" ${click("archive")}>Archive</span><span class="btn danger" ${click("delete")}>Delete</span><span class="btn" ${click("unread")}>Mark unread</span></div>`);
67
+ }
68
+ } else if (app === "shop") {
69
+ const cartBtn = `<span class="btn" ${click("cart")}>Cart (${S.cart.length})</span>`;
70
+ if (S.view === "home") {
71
+ const cards = PRODUCTS.map(p => `<div class="card" ${click("product:" + p.id)}><div class="img" style="background:#e3e8ef"></div><b>${p.name}</b><br>$${p.price}</div>`).join("");
72
+ el(header("Acme Shop") + `<div class="main">${cartBtn}<h2>New arrivals</h2><div class="grid">${cards}</div></div>`);
73
+ } else if (S.view === "product") {
74
+ const p = PRODUCTS[S.product];
75
+ const sw = Object.entries(COLORS).map(([n, c]) => `<span class="swatch ${S.sel.color === n ? "sel" : ""}" style="background:${c}" ${click("color:" + n)}></span>`).join("");
76
+ const sz = SIZES.map(s => `<span class="size ${S.sel.size === s ? "sel" : ""}" ${click("size:" + s)}>${s}</span>`).join("");
77
+ el(header("Acme Shop") + `<div class="main"><span class="btn" ${click("back")}>&larr; All products</span>${cartBtn}<h2>${p.name} &mdash; $${p.price}</h2>
78
+ <p>Colour:</p><div>${sw}</div><p>Size:</p><div>${sz}</div><br><span class="btn primary" ${click("add")}>Add to cart</span>
79
+ ${S.added ? '<div class="toast">Added to cart</div>' : ""}</div>`);
80
+ } else if (S.view === "cart") {
81
+ const items = S.cart.map(c => `<div class="row"><b>${c.name}</b><span>${c.color}, size ${c.size}</span></div>`).join("") || "<p>Your cart is empty.</p>";
82
+ el(header("Acme Shop") + `<div class="main"><span class="btn" ${click("back")}>&larr; Continue shopping</span><h2>Your cart</h2>${items}
83
+ <span class="btn primary" ${click("checkout")}>Checkout</span><span class="btn danger" ${click("empty")}>Empty cart</span>
84
+ ${S.ordered ? '<div class="toast">Order placed. Thank you!</div>' : ""}</div>`);
85
+ }
86
+ } else if (app === "settings") {
87
+ const side = secOrder.map(s => `<div class="item ${S.section === s ? "sel" : ""}" ${click("section:" + s)}>${s}</div>`).join("");
88
+ let panel = "<p>Select a section on the left.</p>";
89
+ if (S.section) {
90
+ panel = `<h2>${S.section}</h2>` + SECTIONS[S.section].map(t => `<div class="toggle"><span>${t}</span><span class="sw ${S.toggles[t] ? "on" : ""}" ${click("toggle:" + t)}></span></div>`).join("")
91
+ + `<br><span class="btn primary" ${click("save")}>Save changes</span><span class="btn" ${click("cancel")}>Cancel</span>`
92
+ + (S.justSaved ? '<div class="toast">Settings saved</div>' : "");
93
+ }
94
+ el(header("Acme Settings") + `<div class="main"><div class="side">${side}</div><div class="panel">${panel}</div></div>`);
95
+ }
96
+ }
97
+ function act(k) {
98
+ const [a, v] = k.split(/:(.*)/s);
99
+ S.justSaved = false; S.added = false;
100
+ if (a === "open") { S.view = "mail"; S.open = +v; }
101
+ else if (a === "back") { S.view = "home"; }
102
+ else if (a === "archive") { S.archived.push(S.open); S.view = "home"; }
103
+ else if (a === "delete") { S.deleted.push(S.open); S.view = "home"; }
104
+ else if (a === "product") { S.view = "product"; S.product = +v; S.sel = {}; }
105
+ else if (a === "color") { S.sel.color = v; }
106
+ else if (a === "size") { S.sel.size = v; }
107
+ else if (a === "add") { if (S.sel.color && S.sel.size) { S.cart.push({ name: PRODUCTS[S.product].name, color: S.sel.color, size: S.sel.size }); S.added = true; } }
108
+ else if (a === "cart") { S.view = "cart"; }
109
+ else if (a === "checkout") { if (S.cart.length) S.ordered = true; }
110
+ else if (a === "empty") { S.cart = []; S.ordered = false; }
111
+ else if (a === "section") { S.section = v; }
112
+ else if (a === "toggle") { S.toggles[v] = !S.toggles[v]; }
113
+ else if (a === "save") { S.saved = Object.assign({}, S.toggles); S.justSaved = true; }
114
+ else if (a === "cancel") { for (const t in S.toggles) S.toggles[t] = !!S.saved[t]; }
115
+ render();
116
+ }
117
+ document.addEventListener("click", e => { const t = e.target.closest("[data-click]"); if (t) act(t.getAttribute("data-click")); });
118
+ render();
119
+ </script></body></html>
vl/serve.sh ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ # JEV-9B with vision: the multimodal Qwen/Qwen3.5-9B (unchanged, including its vision encoder) + JEV-9B's System 1
3
+ # adapter and decision head (vl/adapter_vllm: the same weights as adapter_vllm/, with the layer names moved under
4
+ # model.language_model). JEV-9B's language weights are bit-identical to Qwen3.5-9B's, so text decisions are unchanged.
5
+ # System 1: POST /v1/decide {kind, state, question, options} -> calibrated probabilities (text and images)
6
+ # or model "jev-decision" through /v1/completions and /v1/chat/completions
7
+ # System 2: model "autotrust/JEV-9B" through /v1/chat/completions (text and images)
8
+ # --max-num-seqs 8 is required: with more than 8 sequences in one batch, vLLM's LoRA path for this multimodal model class
9
+ # returns wrong System 1 probabilities. --trust-request-chat-template lets image decisions use the raw decision template.
10
+ set -e
11
+ JEV_DIR=${JEV_DIR:-JEV-9B}
12
+ BASE_DIR=${BASE_DIR:-Qwen3.5-9B}
13
+ [ -f "$JEV_DIR/vl/serve_decide.py" ] || hf download autotrust/JEV-9B --include "vl/*" --local-dir "$JEV_DIR"
14
+ [ -f "$BASE_DIR/config.json" ] || hf download Qwen/Qwen3.5-9B --revision c202236235762e1c871ad0ccb60c8ee5ba337b9a --local-dir "$BASE_DIR"
15
+ exec python3 "$JEV_DIR/vl/serve_decide.py" --model "$BASE_DIR" --served-model-name autotrust/JEV-9B \
16
+ --enable-lora --max-lora-rank 32 --lora-modules jev-decision="$JEV_DIR/vl/adapter_vllm" \
17
+ --logprobs-mode processed_logprobs --max-model-len ${MAX_MODEL_LEN:-32768} --enable-prefix-caching --mamba-cache-mode align \
18
+ --limit-mm-per-prompt '{"image": 8}' --max-num-seqs 8 --max-logprobs 256 --trust-request-chat-template \
19
+ --port ${PORT:-8000} ${EXTRA_ARGS}
vl/serve_decide.py ADDED
@@ -0,0 +1,444 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """vLLM OpenAI-compatible server with a built-in System 1 endpoint (POST /v1/decide) and adaptive thinking.
3
+
4
+ One engine serves both systems, for JEV-27B / JEV-27B-VL (Qwen3.8) and GEV-26B-A4B (Gemma-4):
5
+ System 2 the unmodified base model, through the usual OpenAI endpoints (served model name)
6
+ System 1 the LoRA module "jev-decision" (backbone LoRA + decision head as an lm_head LoRA), through /v1/decide
7
+
8
+ POST /v1/decide
9
+ kind "noul" (yes/no), "score" (0-5) or "choice" (2-256 options)
10
+ state string, JSON object, or a list mixing text and images: ["text", {"image": "https://... | data:..."}]
11
+ question string
12
+ options list of strings (choice only)
13
+ strategy choices with more options than the head's trained choice slots (16 for JEV-27B / GEV; 256 for wide heads,
14
+ which always read up to 256 options in one pass): "single" (one pass, labels A-P then Q-Z, AA, ...),
15
+ "tournament" (groups of <=16 + a final of 16), "permute" (single pass over 4 option orders, averaged);
16
+ default per model
17
+ thinking "off" (default: System 1 only), "auto" (think only when the leading option is below `threshold`),
18
+ "on" (always think)
19
+ threshold System 1 confidence below which "auto" switches thinking on (default per model)
20
+ reasoning controls, as for the base model's own chat API:
21
+ chat_template_kwargs passed to the base model's chat template (e.g. Qwen3.8: {"reasoning_effort": "low"})
22
+ reasoning_effort shorthand for chat_template_kwargs.reasoning_effort, if the base model has it
23
+ (Qwen3.8 / JEV-27B: xhigh (default), medium, low; Gemma-4 / GEV: not available)
24
+ think_budget maximum thinking tokens; default: no limit beyond the context window
25
+ return_reasoning include System 2's reasoning text
26
+ debug include the System 1 and System 2 distributions
27
+
28
+ Adaptive thinking: System 1 gives p1 in one pass. If thinking is switched on, the base model reasons in its thinking
29
+ mode over the same state, question and options; when the thinking channel closes, the answer-letter distribution p2 is
30
+ read in one step, and the result is p = (1 - w) * p1 + w * p2 (default w = 0.5: the fast decision and the reasoning get
31
+ equal weight, which keeps the calibration of System 1; p2 alone is close to one-hot and over-confident).
32
+
33
+ Response (same shape as the hosted JEV API, plus "thinking" when requested):
34
+ {"kind", "effective_kind", "options", "probabilities", "choice_index", "choice", "adaptation", "protocol", "model",
35
+ "usage", "elapsed_seconds", "num_model_requests", "thinking": {...}}
36
+
37
+ Run it like `python -m vllm.entrypoints.openai.api_server` (same flags), with --lora-modules jev-decision=<adapter dir>
38
+ and --trust-request-chat-template. decision_head.json is read from the adapter directory, calibration.json from its
39
+ parent (override with JEV_DECIDE_CALIBRATION).
40
+ """
41
+ import asyncio
42
+ import json
43
+ import math
44
+ import os
45
+ import string
46
+ import time
47
+ import zlib
48
+ from typing import Literal
49
+
50
+ from fastapi import APIRouter, Request
51
+ from fastapi.responses import JSONResponse
52
+ from pydantic import BaseModel
53
+
54
+ import vllm.entrypoints.launchers.api_server.entry as entry
55
+ from vllm.entrypoints.openai.chat_completion.protocol import ChatCompletionRequest
56
+ from vllm.entrypoints.openai.completion.protocol import CompletionRequest
57
+
58
+ LORA = os.environ.get("JEV_DECIDE_LORA", "jev-decision")
59
+ # JEV_DECIDE_SYSTEM2=0: the served model has no usable thinking mode (e.g. a pretrained base model without instruction
60
+ # tuning); thinking "auto"/"on" and system2_only are refused with 400.
61
+ SYSTEM2 = os.environ.get("JEV_DECIDE_SYSTEM2", "1") != "0"
62
+ MAX_OPTIONS = 256
63
+ CHUNK = 128 # vLLM caps logprob_token_ids at 128 per request
64
+ PROTOCOL = "jev27-bare-v1"
65
+ PLACEHOLDER = "XQXCONTENTXQX"
66
+
67
+ # Per-model settings. threshold / mix: adaptive-thinking defaults. GEV: fitted on 1,754 questions outside the Decision
68
+ # Index (CommonsenseQA, OpenBookQA, AQuA-RAT, MedMCQA, LogiQA, StrategyQA). JEV-27B: same defaults, not yet validated.
69
+ PROFILES = {
70
+ "qwen": {"prefix": "", "image": "<|vision_start|><|image_pad|><|vision_end|>", "end_think": "</think>",
71
+ "after_think": "\n\n", "strategy": "single", "threshold": 0.8, "mix": 0.5, "thinking": "off"},
72
+ "gemma": {"prefix": "<bos>", "image": "<|image|>", "end_think": "<channel|>", "after_think": "",
73
+ "strategy": "tournament", "threshold": 0.8, "mix": 0.5, "thinking": "off"},
74
+ }
75
+ # Read the full distribution: override generation_config defaults (top_k/top_p) that would truncate processed logprobs.
76
+ READ = dict(max_tokens=1, temperature=1.0, top_p=1.0, top_k=0, min_p=0.0, repetition_penalty=1.0,
77
+ add_special_tokens=False, return_tokens_as_token_ids=True)
78
+ S = {}
79
+
80
+
81
+ class DecideRequest(BaseModel):
82
+ kind: Literal["noul", "score", "choice"]
83
+ state: str | dict | list = ""
84
+ question: str
85
+ options: list[str] | None = None
86
+ strategy: Literal["auto", "single", "tournament", "permute"] = "auto"
87
+ thinking: Literal["default", "off", "auto", "on"] = "default"
88
+ threshold: float | None = None
89
+ think_budget: int | None = None
90
+ reasoning_effort: str | None = None
91
+ chat_template_kwargs: dict | None = None
92
+ return_reasoning: bool = False
93
+ debug: bool = False
94
+ system2_only: bool = False # skip System 1: think and return System 2's answer distribution (for callers that already have p1)
95
+
96
+
97
+ # ------------------------------------------------------------------ setup
98
+ def setup(args):
99
+ from transformers import AutoConfig, AutoTokenizer
100
+
101
+ path = next((m.path for m in (args.lora_modules or []) if m.name == LORA), None)
102
+ if path is None:
103
+ raise SystemExit(f"serve_decide: start vLLM with --lora-modules {LORA}=<adapter dir>")
104
+ head = json.load(open(os.path.join(path, "decision_head.json")))
105
+ calib = os.environ.get("JEV_DECIDE_CALIBRATION") or os.path.join(os.path.dirname(os.path.abspath(path)), "calibration.json")
106
+ temps = json.load(open(calib))["per_kind"] if os.path.exists(calib) else {"noul": 1.0, "score": 1.0, "choice": 1.0}
107
+ temps.setdefault("score", 1.0)
108
+ src = args.tokenizer or args.model
109
+ tok = AutoTokenizer.from_pretrained(src, trust_remote_code=args.trust_remote_code)
110
+ mtype = AutoConfig.from_pretrained(args.model, trust_remote_code=args.trust_remote_code).model_type
111
+ prof = dict(PROFILES["gemma" if "gemma" in mtype else "qwen"])
112
+ for k in ("strategy", "threshold", "mix", "thinking"):
113
+ if os.environ.get(f"JEV_DECIDE_{k.upper()}"):
114
+ v = os.environ[f"JEV_DECIDE_{k.upper()}"]
115
+ prof[k] = v if k in ("strategy", "thinking") else float(v)
116
+
117
+ def single_token_labels(context):
118
+ out = []
119
+ for lab in list(string.ascii_uppercase) + [a + b for a in string.ascii_uppercase for b in string.ascii_uppercase]:
120
+ t = tok.encode(lab, add_special_tokens=False)
121
+ if len(t) == 1 and t[0] in tok.encode(context.format(lab), add_special_tokens=False):
122
+ out.append((lab, t[0]))
123
+ if len(out) == MAX_OPTIONS:
124
+ break
125
+ return out
126
+
127
+ dec = single_token_labels("x\n{}) y") # System 1 option-line labels
128
+ lo, hi = head["slots"]["ranges"]["choice"]
129
+ assert [t for _, t in dec[: hi - lo]] == head["verbalizer_ids"][lo:hi], "first labels must be the trained A-P head"
130
+ base = tok.encode("Answer: (", add_special_tokens=False) # System 2 answer labels
131
+ ans = []
132
+ for lab in list(string.ascii_uppercase) + [a + b for a in string.ascii_uppercase for b in string.ascii_uppercase]:
133
+ ids = tok.encode(f"Answer: ({lab})", add_special_tokens=False)
134
+ if ids[: len(base)] == base and len(ids) == len(base) + 2:
135
+ ans.append((lab, ids[len(base)]))
136
+ if len(ans) == MAX_OPTIONS:
137
+ break
138
+ chat = tok.apply_chat_template([{"role": "user", "content": PLACEHOLDER}], tokenize=False, add_generation_prompt=True,
139
+ enable_thinking=True)
140
+ pre, post = chat.split(PLACEHOLDER)
141
+ raw = ("{%- for m in messages -%}{%- if m['content'] is string -%}{{ m['content'] }}{%- else -%}"
142
+ "{%- for c in m['content'] -%}{%- if c['type'] == 'text' -%}{{ c['text'] }}{%- else -%}" + prof["image"] +
143
+ "{%- endif -%}{%- endfor -%}{%- endif -%}{%- endfor -%}")
144
+ names = args.served_model_name
145
+ S.update(tok=tok, has_effort="reasoning_effort" in (tok.chat_template or ""), think_cache={},
146
+ max_logprobs=int(getattr(args, "max_logprobs", 20) or 20),
147
+ head=head, temps=temps, labels=[l for l, _ in dec], label_ids=[t for _, t in dec], ans_labels=ans, prof=prof,
148
+ think_pre=pre, think_post=post, raw=raw, end_think_id=tok.convert_tokens_to_ids(prof["end_think"]),
149
+ model=(names[0] if isinstance(names, list) else names) or args.model, model_type=mtype)
150
+
151
+
152
+ # ------------------------------------------------------------------ helpers
153
+ def _parts(state):
154
+ if isinstance(state, str):
155
+ return [{"type": "text", "text": state}], False
156
+ if isinstance(state, dict):
157
+ return [{"type": "text", "text": json.dumps(state, ensure_ascii=False)}], False
158
+ out, img = [], False
159
+ for p in state:
160
+ if isinstance(p, str):
161
+ out.append({"type": "text", "text": p})
162
+ elif isinstance(p, dict) and "image" in p:
163
+ out.append({"type": "image_url", "image_url": {"url": p["image"]}}); img = True
164
+ elif isinstance(p, dict) and p.get("type") == "image_url":
165
+ out.append(p); img = True
166
+ elif isinstance(p, dict) and p.get("type") == "text":
167
+ out.append(p)
168
+ else:
169
+ out.append({"type": "text", "text": json.dumps(p, ensure_ascii=False)})
170
+ return out, img
171
+
172
+
173
+ def _softmax(z):
174
+ m = max(z); e = [math.exp(x - m) for x in z]; s = sum(e)
175
+ return [x / s for x in e]
176
+
177
+
178
+ def _err(msg, code=400):
179
+ return JSONResponse({"error": {"message": msg, "type": "BadRequestError", "code": code}}, status_code=code)
180
+
181
+
182
+ class Upstream(Exception):
183
+ def __init__(self, resp):
184
+ self.resp = resp
185
+
186
+
187
+ def _check(out):
188
+ if hasattr(out, "error"):
189
+ raise Upstream(out)
190
+ return out
191
+
192
+
193
+ class Ctx:
194
+ def __init__(self, raw):
195
+ self.raw, self.requests, self.prompt_tokens, self.completion_tokens = raw, 0, 0, 0
196
+
197
+ def count(self, out):
198
+ self.requests += 1
199
+ if getattr(out, "usage", None):
200
+ self.prompt_tokens += out.usage.prompt_tokens or 0
201
+ self.completion_tokens += out.usage.completion_tokens or 0
202
+
203
+
204
+ def _readout_allowed(ids):
205
+ """Read-out path: restrict the next token to `ids` and return all of them as top logprobs when the server allows that
206
+ many (--max-logprobs); otherwise ask for `ids` explicitly with logprob_token_ids (chunks of 128). The first path also
207
+ works with speculative decoding; logprob_token_ids does not in this vLLM build. Both give the same distribution."""
208
+ return ids if len(ids) <= S["max_logprobs"] else None
209
+
210
+
211
+ async def _logprobs(ctx, content, has_img, ids, model, allowed=None):
212
+ """One-token read-out with a fallback to the other read-out path if the first one fails."""
213
+ try:
214
+ return await _logprobs_once(ctx, content, has_img, ids, model, allowed)
215
+ except Upstream:
216
+ raise
217
+ except Exception:
218
+ other = None if allowed is not None else (ids if len(ids) <= S["max_logprobs"] else None)
219
+ if other is None and allowed is None:
220
+ raise
221
+ return await _logprobs_once(ctx, content, has_img, ids, model, other)
222
+
223
+
224
+ async def _logprobs_once(ctx, content, has_img, ids, model, allowed=None):
225
+ """One-token read-out: {token_id: logprob} for `ids` after the content (text or chat parts)."""
226
+ lp = {}
227
+ step = CHUNK if allowed is None else len(ids)
228
+ for i in range(0, len(ids), step):
229
+ chunk = ids[i: i + step]
230
+ kw = dict(logprob_token_ids=chunk) if allowed is None else dict(allowed_token_ids=allowed)
231
+ if has_img:
232
+ r = ChatCompletionRequest(model=model, messages=[{"role": "user", "content": content}], chat_template=S["raw"],
233
+ add_generation_prompt=False, logprobs=True, top_logprobs=len(chunk) if allowed else 1, **kw, **READ)
234
+ out = _check(await ctx.raw.app.state.openai_serving_chat.create_chat_completion(r, ctx.raw)); ctx.count(out)
235
+ lp.update({int(t.token.split(":")[1]): t.logprob for t in out.choices[0].logprobs.content[0].top_logprobs})
236
+ else:
237
+ text = "".join(c["text"] for c in content)
238
+ r = CompletionRequest(model=model, prompt=text, logprobs=len(chunk) if allowed else 1, **kw, **READ)
239
+ out = _check(await ctx.raw.app.state.openai_serving_completion.create_completion(r, ctx.raw)); ctx.count(out)
240
+ lp.update({int(k.split(":")[1]): v for k, v in out.choices[0].logprobs.top_logprobs[0].items()})
241
+ if allowed is not None:
242
+ break
243
+ return lp
244
+
245
+
246
+ # ------------------------------------------------------------------ System 1
247
+ async def s1_pass(ctx, kind, parts, has_img, question, opts):
248
+ """One System 1 pass over `opts` (<=256). Returns probabilities aligned with opts."""
249
+ head, temps, prof = S["head"], S["temps"], S["prof"]
250
+ lo, hi = head["slots"]["ranges"][kind]
251
+ if kind == "choice":
252
+ n = len(opts); ids = S["label_ids"][:n]
253
+ bias = [head["bias"][lo + i] if lo + i < hi else 0.0 for i in range(n)]
254
+ lines = [f"{S['labels'][i]}) {o}" for i, o in enumerate(opts)]
255
+ else:
256
+ ids = head["verbalizer_ids"][lo:hi]; bias = head["bias"][lo:hi]; lines = opts
257
+ content = ([{"type": "text", "text": f"{prof['prefix']}[kind] {kind}\n[state] "}] + parts +
258
+ [{"type": "text", "text": f"\n[question] {question}\n[options]\n" + "\n".join(lines) + "\n[decision]:"}])
259
+ lp = await _logprobs(ctx, content, has_img, ids, LORA, allowed=_readout_allowed(ids))
260
+ return _softmax([(max(lp.get(t, -1e9), -1e9) + b) / temps[kind] for t, b in zip(ids, bias)])
261
+
262
+
263
+ def _groups(n, k=16):
264
+ g = math.ceil(n / k); base, extra = divmod(n, g); out, i = [], 0
265
+ for j in range(g):
266
+ size = base + (1 if j < extra else 0); out.append(list(range(i, i + size))); i += size
267
+ return out
268
+
269
+
270
+ def _native():
271
+ lo, hi = S["head"]["slots"]["ranges"]["choice"]
272
+ return hi - lo # trained choice slots: 16 (JEV-27B, GEV) or 256 (wide heads)
273
+
274
+
275
+ async def s1_dist(ctx, kind, parts, has_img, question, opts, strategy):
276
+ if kind != "choice" or len(opts) <= _native() or strategy == "single":
277
+ return await s1_pass(ctx, kind, parts, has_img, question, opts)
278
+ n = len(opts)
279
+ if strategy == "permute":
280
+ import random
281
+ rng = random.Random(zlib.crc32(question.encode()))
282
+ orders = [list(range(n))] + [rng.sample(range(n), n) for _ in range(3)]
283
+ res = await asyncio.gather(*(s1_pass(ctx, kind, parts, has_img, question, [opts[i] for i in o]) for o in orders))
284
+ p = [0.0] * n
285
+ for o, r in zip(orders, res):
286
+ for i, v in zip(o, r):
287
+ p[i] += v / len(orders)
288
+ return p
289
+ groups = _groups(n) # tournament: groups of <=16 in the given order (in parallel), then a final of 16
290
+ parts_g = await asyncio.gather(*(s1_pass(ctx, kind, parts, has_img, question, [opts[i] for i in g]) for g in groups))
291
+ in_group = {o: p for g, ps in zip(groups, parts_g) for o, p in zip(g, ps)}
292
+ chosen = [max(g, key=lambda o: (in_group[o], -o)) for g in groups]
293
+ rest = sorted((o for g in groups for o in g if o not in set(chosen)), key=lambda o: (-in_group[o], o))
294
+ fin = sorted(chosen + rest[: max(0, 16 - len(chosen))])
295
+ final = dict(zip(fin, await s1_pass(ctx, kind, parts, has_img, question, [opts[i] for i in fin])))
296
+ group_of = {o: gi for gi, g in enumerate(groups) for o in g}
297
+ share = [0.0] * len(groups); cap = [0.0] * len(groups)
298
+ for f in fin:
299
+ share[group_of[f]] += final[f]; cap[group_of[f]] += in_group[f]
300
+ among = sum(a * b for a, b in zip(share, cap))
301
+ p = [final[o] * among if o in final else share[group_of[o]] * in_group[o] for o in range(n)]
302
+ s = sum(p)
303
+ return [x / s for x in p]
304
+
305
+
306
+ # ------------------------------------------------------------------ System 2 (thinking)
307
+ def _think_frame(kwargs):
308
+ """Chat-template prefix / suffix around the user turn, thinking on, with the caller's template kwargs (cached)."""
309
+ key = json.dumps(kwargs or {}, sort_keys=True)
310
+ if key not in S["think_cache"]:
311
+ kw = {k: v for k, v in (kwargs or {}).items() if k != "enable_thinking"}
312
+ chat = S["tok"].apply_chat_template([{"role": "user", "content": PLACEHOLDER}], tokenize=False, add_generation_prompt=True,
313
+ enable_thinking=True, **kw)
314
+ S["think_cache"][key] = tuple(chat.split(PLACEHOLDER))
315
+ return S["think_cache"][key]
316
+
317
+
318
+ async def s2_dist(ctx, kind, parts, has_img, question, opts, budget, want_text, tmpl_kwargs=None):
319
+ """The base model thinks over the same input; the answer-letter distribution is read after the thinking channel."""
320
+ shown = ["Yes (true)", "No (false)"] if kind == "noul" else opts
321
+ labs = S["ans_labels"][: len(shown)]
322
+ body = "\n".join(f"({l}) {o}" for (l, _), o in zip(labs, shown))
323
+ tail = (f"\n\nQuestion: {question}\n\nOptions:\n{body}\n\n"
324
+ "Think it through carefully, then give your final answer on the last line in the form: Answer: (X)")
325
+ pre, post = _think_frame(tmpl_kwargs)
326
+ user = [{"type": "text", "text": pre}] + parts + [{"type": "text", "text": tail + post}]
327
+ seed = zlib.crc32((question + body).encode()) & 0x7FFFFFFF
328
+ t0 = time.time()
329
+ gen = ChatCompletionRequest(model=S["model"], messages=[{"role": "user", "content": user}], chat_template=S["raw"],
330
+ add_generation_prompt=False, add_special_tokens=False, max_tokens=budget, seed=seed, # None: up to the context window
331
+ stop_token_ids=[S["end_think_id"]], skip_special_tokens=False)
332
+ out = _check(await ctx.raw.app.state.openai_serving_chat.create_chat_completion(gen, ctx.raw)); ctx.count(out)
333
+ thought = out.choices[0].message.content or ""
334
+ finished = out.choices[0].finish_reason == "stop"
335
+ ntok = out.usage.completion_tokens if out.usage else None
336
+ read = user + [{"type": "text", "text": thought + S["prof"]["end_think"] + S["prof"]["after_think"] + "Answer: ("}]
337
+ ids = [t for _, t in labs]
338
+ lp = await _logprobs(ctx, read, True, ids, S["model"], allowed=_readout_allowed(ids))
339
+ p = _softmax([max(lp.get(t, -1e9), -1e9) for t in ids])
340
+ if kind == "noul":
341
+ p = [p[1], p[0]] # (A) yes / (B) no -> [P(false), P(true)]
342
+ info = {"think_tokens": ntok, "think_seconds": round(time.time() - t0, 3), "finished_within_budget": finished}
343
+ if want_text:
344
+ info["reasoning"] = thought.replace("<|channel>thought\n", "").strip()
345
+ return p, info
346
+
347
+
348
+ def _fold(p1, p2, w):
349
+ return [(1 - w) * x + w * y for x, y in zip(p1, p2)]
350
+
351
+
352
+ # ------------------------------------------------------------------ routes
353
+ router = APIRouter()
354
+
355
+
356
+ @router.get("/v1/decide/info")
357
+ async def info():
358
+ p = S["prof"]
359
+ return {"protocol": PROTOCOL, "model": S["model"], "model_type": S["model_type"], "max_options": len(S["labels"]),
360
+ "native_choice_options": len(S["head"]["slots"]["verbalizers"]) - 8, "temperatures": S["temps"],
361
+ "defaults": {"strategy": p["strategy"], "thinking": p["thinking"], "threshold": p["threshold"], "mix": p["mix"],
362
+ "think_budget": None},
363
+ "system2_available": SYSTEM2,
364
+ "reasoning_controls": {"enable_thinking": True, "reasoning_effort": ["xhigh", "medium", "low"] if S["has_effort"] else None,
365
+ "think_budget": "max thinking tokens (default: up to the context window)"}}
366
+
367
+
368
+ @router.post("/v1/decide")
369
+ async def decide(req: DecideRequest, raw: Request):
370
+ t0 = time.time()
371
+ if req.kind == "choice":
372
+ opts = req.options or []
373
+ if not 2 <= len(opts) <= len(S["labels"]):
374
+ return _err(f"choice needs 2-{len(S['labels'])} options, got {len(opts)}")
375
+ else:
376
+ opts = ["false", "true"] if req.kind == "noul" else [str(i) for i in range(6)]
377
+ prof = S["prof"]
378
+ thinking = prof["thinking"] if req.thinking == "default" else req.thinking
379
+ if req.kind == "score":
380
+ if req.thinking in ("auto", "on"):
381
+ return _err("thinking is supported for noul and choice")
382
+ thinking = "off"
383
+ tmpl = dict(req.chat_template_kwargs or {})
384
+ if req.reasoning_effort is not None:
385
+ if not S["has_effort"]:
386
+ return _err("this base model has no reasoning_effort setting; control the reasoning length with think_budget")
387
+ tmpl["reasoning_effort"] = req.reasoning_effort
388
+ strategy = prof["strategy"] if req.strategy == "auto" else req.strategy
389
+ tau = prof["threshold"] if req.threshold is None else req.threshold
390
+ parts, has_img = _parts(req.state)
391
+ ctx = Ctx(raw)
392
+ if not SYSTEM2 and (thinking in ("auto", "on") or req.system2_only):
393
+ return _err("thinking (System 2) is not available for this model: it is a pretrained base model without "
394
+ "instruction tuning; use thinking \"off\"")
395
+ try:
396
+ if req.system2_only:
397
+ if req.kind == "score":
398
+ return _err("system2_only is supported for noul and choice")
399
+ p2, think = await s2_dist(ctx, req.kind, parts, has_img, req.question, opts, req.think_budget, req.return_reasoning, tmpl)
400
+ k = max(range(len(p2)), key=p2.__getitem__)
401
+ return {"kind": req.kind, "options": opts, "probabilities": p2, "choice_index": k, "choice": opts[k], "system": 2,
402
+ "model": S["model"], "thinking": {"used": True, "budget": req.think_budget, **think},
403
+ "usage": {"prompt_tokens": ctx.prompt_tokens, "completion_tokens": ctx.completion_tokens},
404
+ "elapsed_seconds": time.time() - t0, "num_model_requests": ctx.requests}
405
+ p1 = await s1_dist(ctx, req.kind, parts, has_img, req.question, opts, strategy)
406
+ probs, think = p1, None
407
+ if thinking == "on" or (thinking == "auto" and max(p1) < tau):
408
+ p2, think = await s2_dist(ctx, req.kind, parts, has_img, req.question, opts, req.think_budget, req.return_reasoning, tmpl)
409
+ probs = _fold(p1, p2, prof["mix"])
410
+ think = {"used": True, **think}
411
+ if req.debug:
412
+ think.update(system1=p1, system2=p2)
413
+ elif thinking != "off":
414
+ think = {"used": False}
415
+ if req.debug:
416
+ think["system1"] = p1
417
+ except Upstream as e:
418
+ return JSONResponse(e.resp.model_dump(), status_code=e.resp.error.code)
419
+ k = max(range(len(probs)), key=probs.__getitem__)
420
+ native = req.kind != "choice" or len(opts) <= _native()
421
+ resp = {"kind": req.kind, "effective_kind": req.kind, "options": opts, "probabilities": probs, "choice_index": k,
422
+ "choice": opts[k], "adaptation": "native" if native else f"{strategy}", "protocol": PROTOCOL, "model": S["model"],
423
+ "usage": {"prompt_tokens": ctx.prompt_tokens, "completion_tokens": ctx.completion_tokens,
424
+ "total_tokens": ctx.prompt_tokens + ctx.completion_tokens},
425
+ "elapsed_seconds": time.time() - t0, "num_model_requests": ctx.requests}
426
+ if think is not None:
427
+ resp["thinking"] = {"mode": thinking, "threshold": tau, "budget": req.think_budget, **({"chat_template_kwargs": tmpl} if tmpl else {}), **think}
428
+ return resp
429
+
430
+
431
+ _build_app = entry.build_app
432
+
433
+
434
+ def build_app(args, *a, **kw):
435
+ app = _build_app(args, *a, **kw)
436
+ setup(args)
437
+ app.include_router(router)
438
+ return app
439
+
440
+
441
+ entry.build_app = build_app
442
+
443
+ if __name__ == "__main__":
444
+ entry.main()