Why doesn't Fagin just report to the police that there's a madman threatening him for money?
Well, because he would probably have to confess that he made deals with a loan shark, which is not exactly legal. That and the fact that Fagin is a thief (albeit a harmless one) must have been what kept him form seeking the police. Itīs also possible that Sykes could have killed the policeman sent after him and for Fagin it would only make things worse if Sykes found out he was the one who tried to get him arrested.
Or Sykes has enough connections that an accusation from a homeless thief won't get anywhere with the police, and will certainly tip Sykes off that Fagin ratted him out, which won't end well for him.