There has been an increasing demand for physical human-robot collaboration during the design prototyping phase. For example, users would like to maneuver the end-effector compliantly in free space followed by supplying a contact force to obtain a firm adhesive connection. The technical challenges is the design of the controller, especially during the switching from human-robot interaction (human guides robot) to robot-environment interaction for the robot to continuously maintain the contact force even after the human lets go. Traditional controllers often result in unstable interaction during the switches of the controllers. Therefore, this letter proposes a control scheme that unifies impedance and admittance in the outer loop, and unifies the adaptive position and velocity control in the inner loop to address this issue. The cooperation of the cobot is divided into two modes, namely, an augmentation mode where the human force is the desired input to guide the motion of the cobot, and an autonomous mode where predefined position and force commands are used (e.g., to maintain a desired holding force). With the proposed control scheme, the physical interaction between the robot, human and environment can be smoothly and stably transited from augmentation mode to autonomous mode. Experiments are then conducted to validate the proposed approach.