What is the fastest way to write a matrix to a text file in Octave?

The following applies to MATLAB, but I suggest you try it in Octave. First of all, if you can - transpose the matrix. Here are examples using fprintf and csvwrite (essentially dlmwrite)

A = rand(3, 1e6);
tic;
fid = fopen('data.txt', 'w+');
for i=1:size(A, 1)
    fprintf(fid, '%f ', A(i,:));
    fprintf(fid, '\n');
end
fclose(fid);
toc

tic;
csvwrite('data.txt', A);
toc;

Elapsed time is 1.311512 seconds.
Elapsed time is 2.487737 seconds.

If not transposed, it will take ages indeed. By default, fprintf flushes the buffer after every call. You can try to use W instead of w to open the file, but it does not improve the situation here too much.

Having a variable data you can save it in a text format with space separated values (including a header):

save out.txt data

The header can be simply removed using basic Unix command tail e.g. (on any Linux/Mac OS):

tail -n +6 out.txt > file.csv

Did you try this? I'm not sure about it's speed as compared to dlmwrite.

a = [1 2;3 4];
save temp.txt a;