Bayesian test of hypothesis uses Bayesian statistics to evaluate the probability of a hypothesis being true given observed data. It involves defining a null and alternative hypothesis, updating the prior probability distribution based on the likelihood function to obtain the posterior probability, and calculating the Bayes factor, which quantifies the evidence supporting one hypothesis over the other.