There is a fallacy here. To be clothed with love is not a matter of man's love, but of Christ's love for us. Paul also talks of being clothed with Christ, which I believe means the same thing as the Colossians passage here.
I do agree with the concept, that man's love for one another cannot justify us, I would also say it is impossible. Save for being clothed with Christ, clothed in His love, immersed in it. That is what we have faith in, God's love for us, that is what drove Him to the cross, the idea of saving His beloved. And that is the love which creates the relationship based on faith, on trust.