Which vendor should I choose? What makes the vendor credible?

Proprietary vendors work on long term lock in philosophy. They would make you invest so much initially on their system that you will not think about switching to a better technology for years. And while you are using a proprietary solution, you would be purchasing equipments/other software that work with it only. Proprietary vendors do not give you freedom to choose and flexibility to upgrade. These critical decisions are usually with them only.

How much am I willing to spend?
How can I project my future costs?

Do not take the initial cost into account only. Ask your vendor to explain the possible expansion scenarios.
Proprietary vendors do not give you an option to expand or customize the systems. The only option that they give you is replacement of the current system with a newer and bigger one by introducing newer versions and discontinuing support for older versions. There is no concept of spend as you expand.

How can I make expansion easy and less costly?
How do I know which option is more scalable?

You can make expansion less costly and much easier if you abide by standards compliant software. Proprietary vendors tend to lock you in for a long time because their solutions usually do not work with any other hardware or software. You must have the freedom to choose hardware from any vendor to avoid lock-in.
You simply need to calculate incremental costs that you are incurring among the options you need to choose. Cost of expansion must be less than cost of acquisition.

How important is compatibility and interoperability?

Proprietary solutions do not provide interoperability with other software because of their locking in policies. This is one of the most important features of open source software. Open source software are developed keeping compatibility and interoperability under due consideration. Because enhancements and development is much faster in open source software than in any other proprietary software, it is only a matter of time that your requirements are met. Community based support and development plus no licensing issues give you a much better solution.

How can I make management easier?

Management and internal support becomes easier when you have trained individuals working for you. You can find answers and solutions to any problem that you are facing on the internet because of community support available. You are not dependent on the proprietary vendor to support you.