Removing a hashtag
If you remove a hashtag on one of your oldest media, the said media will take some time to be removed from the album associated with it.
If you remove a hashtag from one of your latest media, the said media will be quickly removed from the album associated with it.
Adding a hashtag
As our hashtag data comes directly from the Instagram API, and as Instagram does not guarantee to recognize hashtags that are added to the first comment or added to the caption later, there's no guarantee that one of your older media will be added to one of your albums if you update the hashtag.