Payload and OS integration is one thing. What I still didn't figure out is how to tell vboot to check if RW partition is valid? Imagine the platform is in recovery mode and I have flashed the RW with correctly signed image. vboot will not attempt to boot from RW, because of the recovery reason being non-zero. Thus the only way I see is to clear the recovery reason. How it is solved on ChromeOS systems after updating with correct RW firmware? Is there any flag (in vboot shared data/workbuf) to tell vboot to attempt RW check despite recovery?
Well, in our flow we always clear the recovery reason from the payload (as part of running through vb2api_kernel_phase2()). So if you aren't using the kernel verification part but otherwise want the equivalent of that, then the solution would be to make vb2_clear_recovery() available as a top-level API function and call it from your payload.
Alternatively, you could also just have your OS run `crossystem recovery_request=0` after it has reinstalled the RW firmware.