Public infrastructure is often monitored to assess quality. Alarm thresholds indicate when actions are necessary to improve passenger convenience and operational safety. Based on safety assessments, urgent rail replacements are proposed with a new approach for physical damage threshold extraction. Railway tracks are crucial in transportation, and physical damage can greatly affect their efficiency. During regular visual inspections, inspectors can identify physical damages such as head checks or fatigue cracks. However, there are no definite thresholds to determine the severity of the damages except for rail profile deformation known as wear, which is based on references and standards. Therefore, during the survey and inspection, it is essential to establish alarm thresholds based on limit states for each case. Defects detected before the ultimate limit state can lead to a reduction in quality, which affects the convenience of travel but is not harmful to serviceability. Structure quality control methods use both qualitative and quantitative approaches to assess these cases. However, quantitative assessment reduces the probability of human errors and subjective judgments by inspectors when compared to qualitative assessment. To begin with, image processing is a quantitative tool used to detect defects on rail surfaces. However, the determination of the threshold has been neglected, leading to a lack of anticipation of rail physical damages' lifetime and cost estimation. Therefore, machine learning tools evaluate the output of image processing based on the proposed threshold, which has been expressed in this research. In order to ensure safety, contingency actions will be taken when needed based on the predetermined alarm thresholds.