To date, there is no independent and uniformly presented information available regarding detection performance of automated in-line mastitis detection systems. This lack of information makes it hard for farmers or their advisors to make informed investment decisions. This paper describes two on-farm tests that will provide farmers with an indicative performance of in-line mastitis sensors using data from early adopters of sensors of interest. The first test provides insight into a system’s ability to identifying cows treated for clinical mastitis. The second test provides insight into a system’s ability to identify cows with a high somatic cell count. This partial evaluation was applied to data from a New Zealand research farm with an in-line mastitis detection system. Results showed this system had 63% sensitivity with 87 false alerts per 1000 cow milkings for identifying cows treated for clinical mastitis. It also suggested that 10-36% of the herd should be excluded from the bulk tank to decrease bulk milk somatic cell count by 25%. These results can be used by farmers to determine if sensors of interest are likely to meet their on-farm requirements and expectations.