You need to check the gravity on your character rigidbody while climbing otherwise the gravity will always take your character down.
There's a few ways to accomplish this.
- Destroy the rigidbody when you are on the ladder, and just have a ladder-traversal script that slides you up and down relative to the ladder's position, which would also take care of the ladder moving. Then when you get off the ladder, add the Rigidbody back on, with any custom values it might be using.
- You could use a Fixed Joint to attach to the ladder (ladder would need a rigidbody too obviously, one that is kinematic), and then drive the offset of the Fixed Joint to go up and down the ladder.
- Select the player in Inspector panel Search for Character Controller (OffSet) Increase the "Set Offset".
