154. If any condition on which a person has been pardoned or a punishment has been remitted is, in the opinion of the authority which granted the pardon or remitted the punishment, not fulfilled, such authority may cancel the pardon or remission, and thereupon the sentence shall be carried into effect as if such pardon had not been granted or such punishment had not been remitted: Provided that in the case of a person sentenced to imprisonment or detention such person shall undergo only the unexpired portion of his sentence.