Is power proportional to V or V^2?

It depends on the circumstances. Without knowing anything else, we don't know whether power is proportional to \$V\$, \$V^2\$, or neither.

If a variable voltage source is connected to a resistor, then the power is proportional to \$V^2\$.

If a variable voltage source is connected to a constant-current load (something which admits the same amount of current regardless of the voltage across it), then the power is proportional to \$V\$.

If a variable voltage source is connected to a constant-power load (one which admits more current when the voltage is less, and less current when the voltage is more, so as to consume a constant amount of power, and thereby acting as a negative resistance), then the power is independent of \$V\$. A switched-mode power supply may behave in a manner similar to this. It would be pretty straightforward to design a device which consumes \$1\ \mathrm{W}\$ of power when connected to any voltage between \$5\ \mathrm{V}\$ and \$24\ \mathrm{V}\$, for example.

If (iff) R is fixed the power is indeed proportional to \$V^2 ~ and~ I^2\$.

You may easily prove the latter by similar substitution.

If the resistance is constant, then increasing the voltage increases the current proportionally, exactly proportionally in the ideal case.

Power is not proportional to voltage squared in cases where the load is not a constant resistance. For example, with a rectifier diode it will increase considerably faster than voltage squared. If you measure power into a current regulator diode, it will increase more like proportional to voltage since current will be more-or-less constant over a range of voltages.