Casts on a foot can be a burden, they’re heavy, they’re hot, they make your foot itch, they limit your ability to do things, and you can’t get them wet (the plaster ones, anyway). However, casts play an important role in healing.
A cast that is properly applied keeps your foot in the prescribed position so the bones can heal in the right shape. A cast also protects your foot from further harm as you can’t dislodge the bone as it is healing and it prevents you from putting undue weight on your foot.
While it’s possible that your foot may have healed without a cast, if your doctor was concerned about the break being displaced, the cast was likely the best bet. You may be very good at using your crutches, but it would only take one loss of balance for you to be forced to put unexpected weight on your foot and cause more damage.