Writing this up, maybe it helps someone avoid the same amount of time and frustration. Through trial and error and helpful posts like this one: https://community.netgear.com/t5/Orbi-AX/RBS850-Satellite-will-not-sync-with-RBR850-Router-when-in-AP/td-p/1873979
First, before purchasing this kit at Costco I validated that AP mode was a function supported. But didn’t realize just how broken or strict the process was to get it working. The following instructions will help connect the RBS units via wireless backhaul. Wired backhaul says it is synced but there’s still oddities like opening your entire home network to anyone in range without a key...
To get Satellites connect you essentially need to ignore all warnings, suggestions, etc from both the app and the browser.
Whether you do this at a staging desk or where you actually have devices is up to you. If you set devices where you want them it will save you time powering off and powering back on. Recommend at least having the RBR unit physically where you want it.
Step 0. Reset all devices. Tech support said it needed to be a pin in the hole for 50 seconds - assuming already booted. Cut power for all devices.
Step 1. Plug the RBR750 internet port into your network switch environment, assuming something else is running DHCP for you and you know how you have your scope assigned. Power on RBR750.
Step 2. Using the Orbi app go through setup as a new system. Ignore steps to power on satellites. Make the SSID and password what you want it in the end. This is just to be safe to not have to start the process over again if changing the SSID and password isn’t supported when it’s already configured.
Step 3. Once setup, login to your RBR750 with a desktop web interface (Http://orbilogin.com) while connected to the SSID.
Step 4. Go to advanced tab -> advanced -> router / AP mode. Select AP mode
Step 5. Set a static IP. It says it’s not recommended - but this is the only way it works. Pick a free IP on your network outside of your DHCP scope. Then apply it, and wait for it to reboot.
Step 6. Power up your first RBS unit. Nothing else plugged into it. Once booted up (give it 2 minutes), initialize sync on your RBR, you can do this via the web interface so you aren’t walking up and down stairs. Then press hold the sync button on your RBS. Blue light should turn on shortly after. I wouldn’t recommend powering off the RBS unit until the blue light turns off. During it’s config process if you logon to the web interface of the RBS you can see evidence of the password and other configuration items being set.
7. Repeat step 6 for any additional RBS units.
Best of luck! Hope this short write up helps someone.