Clients can't interact with NetworkRigidbody2D GameObjects while host can

[jovial quartz]

I have an object that a player can interact with and move around.
The player can only control the player itself, but by moving into that object they can move it.
When opening a server with a host and a client, only the host can move the object around.

I have a NetworkIdentity, NetworkTransform and NetworkRigidbody2d for the players and the object.
Note that I want the movement of the object to be physically correct, I do not want to add a controller to it and recreate the physics of its movement.

Is it possible to do this without creating commands and RPCs and recreating the physics based motion?

[ivory hull]

RPC commands are how this is done. Only the host/server should be interactive with networked physics

You'll otherwise have to make some kind of client prediction system. Even that will be sending inputs with RPCs

[jovial quartz]

So how can I send RPCs to interact with the physics system and apply the correct forces and such?
I understand running my own code in RPCs but not imitating the Unity physics system

[ivory hull]

You would send an RPC on collision to add the force of that collision to the object with rigidbody.AddForce()