Recommendations for database with R

I also need to be able to directly call the database from R

I suggest setting up MySQL with RMySQL interface.

Once the DB connection is open, you can query the database and get the the data into R, example:

# Run an SQL statement by creating first a resultSet object
rs <- dbSendQuery(con, statement = paste(
                      "SELECT w.laser_id, w.wavelength, p.cut_off",
                      "FROM WL w, PURGE P",
                      "WHERE w.laser_id = p.laser_id",
                      "SORT BY w.laser_id")
# we now fetch records from the resultSet into a data.frame
data <- fetch(rs, n = -1)   # extract all rows

RMySQL: R interface to the MySQL database

Database interface and MySQL driver for R. This version complies with the database interface definition as implemented in the package DBI 0.2-2.

MySQL Database:

Available for all the platforms you cited in the question, and more, download here.

Do you really need a database solution for your purpose? You say you want a "solution for storing data at intermediate steps " -- how about simply saving the data array to disk at the required time points?

Edit: to make it possible to retrieve the information, you can embed meta-information, e.g. trial index and/or timestamp, in the filename. Then later you can locate and load the file using the correct filename.

Quick comments:

• R is good at this, as a language for programming with data, there are plenty of interfaces
• There is an entire manual devoted to data import/export, and it has a section on relational databases, so start there.
• R has the widely-used DBI package which provides a unified interface for many backends, among them SQLite, MySQL, PostgreSQL, Oracle, ... Use that, maybe with RSQLite to get something going quickly. You can still switch backends afterwards.
• There is also RODBC but I find ODBC tedious to work with.
• R also has a specialised variant in the TSdbi package by Paul Gilbert which brings the DBI-alike abstraction to timeseries databases. It also supports multiple backends.
• The data.table package was written for this and is very fast on indexing and aggregation.