signal is designed not to trust the server

Unfortunately this is not enough. A malicious Signal server can mount a timing correlation attack and infer the social graph of an user. Having a centralized server makes it more difficult to mitigate such risk.