I expect the relay switch is a simple on/off.

This won't work with a brushless motor, they NEED an ESC. Brushless motors have basically had their timing ability (done by the brushes and comm) taken off the motor and put into the programming and sensors of the ESC. As such without a brushless ESC a brushless motor like we use cannot turn.\

The brushless ESC would presumably be taking the place of the relay.