King of the Court Matchmaking??

How does it work? I played the #1, #3, and #6 guy on the leaderboard and lost each game by 10-20+ and eliminated in 3.

Is it because I started late?