Can I remove past media from the calendar?

No, you can't delete a past media from the post scheduler calendar. You can only delete (or edit) upcoming scheduled posts.

You can, however, reschedule/copy past posts - find out more here!