Fast MATLAB, slow Mathematica: Export a matrix as TSV
After using them for several years, I have to admit that many Export routines in Mathematica are either broken or way too slow (a factor of 10 to 25 when compared to a "C++" routine, e.g. for such a simple task as exporting an array).
It is very unfortunate. However, the following can export the data within 1.6 seconds while the OP's version took 15.985 seconds on my machine. It allows for specifying how many digits have to be exported.
colsep = "\t";
rowsep = "\n";
prec = 16;
SetAttributes[doubletostring, Listable];
doubletostring[x_] := Internal`DoubleToString[x, False, prec];
Export[
"a.txt",
StringJoin[
Map[
row \[Function] StringJoin[Riffle[doubletostring /@ row, colsep], rowsep],
e]
]
]; // AbsoluteTiming // First
1.60592
This timing is from Mathematica 11.3 on macOS v10.13.4 (High Sierra).
A hand-written C++ routine linked to Mathematica needs about 0.6 seconds to export the matrix e, so in principle, there is still room for improvement.
Since As ilian told me in this post, the second argument controls whether Internal`DoubleToString is undocumented, I have no idea what its second argument does (it appears to me that it has no effect which would be pretty weird).NumberMark shall be printed (False suppresses it). The third argument seems to specify how many leading digits have to be printed.
Some further speedup can be obtained by using ParallelMap instead of Map. Once the parallel kernels are set up and started, the timing is pretty close to the (unparallelized) C++ timing on my Quad Core machine:
Export["a.txt",
StringJoin[
ParallelMap[
row \[Function]
StringJoin[Riffle[doubletostring@row, colsep], rowsep],
e
]
]
]; // AbsoluteTiming // First
0.875412
First rerunning the OP's demo for a baseline timing.
e = RandomReal[{-10, 10}, {1000, 1000}];
AbsoluteTiming[
Export[FileNameJoin[{$InitialDirectory, "vxFolder", "vx7.txt"}], e, "Table"];
]
{34.0312, Null}
As you can see in the code below, most of the time is spent formatting the data but the writing takes only 0.7 seconds. If the OP's matrix can be generated as a string in the right format writing can be done quickly.
AbsoluteTiming[
str = OpenWrite[FileNameJoin[{$InitialDirectory, "vxFolder", "vx8.txt"}]];
]
AbsoluteTiming[
e2 = StringTake[StringReplace[
ToString[InputForm@e], {"}, {" -> "\n", ", " -> "\t"}], {3, -3}];
]
AbsoluteTiming[
WriteString[str, e2];
Close[str];
]
{0.00266855, Null}
{27.4393, Null}
{0.684855, Null}
An alternative formatting function takes about the same time.
AbsoluteTiming[
e2 = StringRiffle[StringRiffle[#, "\t"] & /@ Map[
StringReplace[Internal`DoubleToString[#], "`" -> ""] &, e, {2}], "\n"];
]
{29.589, Null}
This really bugged me so I used the opportunity to learn a bit more about LibraryLink. I have to say that having a close look at the internals of Szabolcs' LTemplate package was significantly more helpful than the actual LibraryLink documentation...
Setting up the C++ code:
Needs["CCompilerDriver`"];
Needs["GeneralUtilities`"];
LibraryLink`BoilerPlate[funname_String] :=
StringJoin["#include \"math.h\"
#include \"WolframRTL.h\"
static WolframCompileLibrary_Functions funStructCompile;
static mbool initialize = 1;
DLLEXPORT int Initialize_", funname, "(WolframLibraryData libData)
{
if( initialize)
{
funStructCompile = libData->compileLibraryFunctions;
initialize = 0;
}
return 0;
}
DLLEXPORT void Uninitialize_", funname, "(WolframLibraryData libData)
{
if( !initialize)
{
initialize = 1;
}
}
"]
funname = "writeRealMatrixToFile";
preamble = "
#include <iostream>
#include <fstream>
#include <boost/lexical_cast.hpp>
using namespace std;
using namespace boost;
";
code = StringJoin[preamble, "\n\n", LibraryLink`BoilerPlate[funname],
"extern \"C\" DLLEXPORT int ", funname,
"(WolframLibraryData libData, mint Argc, MArgument * Args, MArgument Res)
{
int err = 0;
const char * file = MArgument_getUTF8String(Args[0]);
const MTensor a0 = MArgument_getMTensor(Args[1]);
const mreal * a = libData->MTensor_getRealData(a0);
const char * prefix = MArgument_getUTF8String(Args[2]);
const char * infix = MArgument_getUTF8String(Args[3]);
const char * suffix = MArgument_getUTF8String(Args[4]);
const char * linesep = MArgument_getUTF8String(Args[5]);
mint const* dims;
dims = libData->MTensor_getDimensions(a0);
mint i, j;
ofstream outputstream;
try
{
outputstream.open(file, ofstream::out | ofstream::app);
for( i=0; i < (dims[0]-1); i++)
{
outputstream << prefix;
for( j=0; j < dims[1]-1; j++)
{
outputstream << lexical_cast<string>(*a) << infix; \
a++;
}
outputstream << lexical_cast<string>(*a) << suffix;
a++;
outputstream << linesep;
}
{
outputstream << prefix;
for( j=0; j < dims[1]-1; j++)
{
outputstream << lexical_cast<string>(*a) << infix;
a++;
}
outputstream << lexical_cast<string>(*a) << suffix;
}
outputstream.close();
MArgument_setInteger(Res, err);
}
catch (...)
{
return LIBRARY_FUNCTION_ERROR;
}
return LIBRARY_NO_ERROR;
}
"];
Creating the library in a temporary directory. (If you'd like to reuse the library, just specify a value of choice for "TargetDirectory".) Since we require boost and it is not located at a path that can be found automatically by the compiler, I also specify "IncludeDirectories". Of course you have to make sure that boost is installed on your system.
libfile = CreateLibrary[code, funname,
"Language" -> "C++",
"TargetDirectory" -> $TemporaryDirectory,
"IncludeDirectories" -> "/opt/local/include"
];
Loading the library function and creating a wrapper function as rudimentary user interface.
If[TrueQ[Head[writeRealMatrixToFile] === LibraryFunction],
LibraryFunctionUnload[writeRealMatrixToFile]];
ClearAll[writeRealMatrixToFile];
If[TrueQ[FindLibrary[libfile] =!= $Failed],
writeRealMatrixToFile = LibraryFunctionLoad[libfile, funname,
{
"UTF8String", {Real, 2, "Constant"}, "UTF8String", "UTF8String",
"UTF8String", "UTF8String"
},
Integer]
];
Options[WriteRealMatrixToFile] = {
"Prefix" -> "",
"Infix" -> "\t",
"Suffix" -> "",
"RowSep" -> "\n"
};
WriteRealMatrixToFile[file_String,
A_?(MatrixQ[#, Developer`MachineRealQ] &), OptionsPattern[]] :=
writeRealMatrixToFile[file, A,
OptionValue["Prefix"],
OptionValue["Infix"],
OptionValue["Suffix"],
OptionValue["RowSep"]
];
Running the test example.
A = RandomReal[{-1, 1}, {1000, 1000}];
file = "test.txt";
Put[file];
WriteRealMatrixToFile[file, A]; // AbsoluteTiming // First
B = Import[file, "Table"]; // AbsoluteTiming // First
Max[Abs[A - B]]
0.668737
1.5853
0.
This is over 20 times faster than using Export with "Table" as export format:
Export["A.txt", A, "Table"]; // AbsoluteTiming // First
16.0703
Final remark:
I wrote writeRealMatrixToFile such that it appends to existing files (creating a new one if needed). This way, it can be used to export multiple data sets with other chunks of code in between. Typical file formats where this might be helpful are, e.g., povray and obj.
In particular, povray needs lists of vectors specified in the format < x1, y1, z1 >,< x2, y2, z2 >,... and to put < and > is the purpose of the "Prefix" and "Suffix" option. "Infix" specifies the column separator (e.g. ","; default is a tabulator ("\t")) and "RowSep" specifies the row seperators (default is newline ("\n")).
Edit
New version employs boost::lexical_cast on the C++ side for conversion of double to string, preventing a loss of relative precision.