You can start a schedule of decreased wearing time. Take the next week to 10 days to gradually wean yourself off the collar. If you are wearing it during your waking hours, put it on 15 minutes later in the morning. Take it off 15 minutes earlier at night. Repeat this formula every day. If you feel comfortable, increase the time you delay/stop to 30 minutes each day.
At the same time you need to continue your program of exercise and strengthening exercises. Your rehab program will help you expand your limitations and over come your physical and mental dependence on the collar.
See your surgeon if you have trouble adopting a weaning schedule. An exam may be needed to ensure the stability of your surgical site.