Extreme of non-intersecting Brownian motions from Yang-Mills theory to directed polymers