6

I'm running a lot of experiments that give their output as CSV files. An experiment might be running for hours, with a new line being added to the CSV every 10 seconds.

Right now I'm opening these CSV files in a text editor, which isn't too convenient. I'm looking for a better way.

Here are some features I want:

  1. View CSV files as a table.
  2. Automatically display numbers in a reasonable way (i.e. instead of 10 significant digits after the dot, show 2.
  3. Automatically tail the file, i.e. show new lines as the file is updated.
  4. Allow me to hide columns, and remember my selection for new CSV files of the same format.
  5. Allow me to show the data in a plot, and remember the plot format I used for the new CSV files of the same format.

Does anything like that exist?

Ram Rachum
  • 255
  • 1
  • 6

4 Answers4

1

It is my first reply, and it already starts with the holly war. Please don't kill me.

I don't like EXCEL much, but maybe it answers all the requirements.

  1. Open a blank excel file (.xlsx)
  2. Go to Data -> From Text, select a comma delimiter
  3. Choose numeric columns, right click and change the Format Cells category to Number with 2 significant digits.
  4. Hide any columns you want. Save the file, your selection will be remembered when you open the file next time.
  5. Create a plot. When creating a plot, select the range that includes the last filled row. When you save the file the plot's format will be remembered.
  6. To update the information (or load another CSV in the same format) press Alt+F5. The sheet contents both with the plot will be refreshed and will contain all of the CSV's data (including the lastly added rows).
griko
  • 31
  • 2
1

If you do not like Excel, consider using the Pandas library. You can use the pandas.read_csv command to read the file in a DataFrame. After you have imported the data you can use the pd.options.display.float_format = "{:,.2f}".format to get it to show only 2 significant digits. If you want the script to show new records, you can write a simple code to re-read the csv in a time period you want. Hiding columns is also really easy in Pandas, you simply have to selected_columns = all_columns["selected_1", "selected_2"]. The library is also capable of drawing simple plots, so they might do the job, depending on how complex diagrams you want. For the different formats just write if/else functions.

Nemo_the_scientist
  • 612
  • 2
  • 11
1

With pandas to format your file as you want Visidata (https://github.com/saulpw/visidata) is a must have !

it is a open source project, run in terminal and rocks ! more info here https://www.visidata.org

marcopovitch
  • 46
  • 3
0

Why don't you try an interpreted language, something like Octave, R, Julia, Python etc. with an IDE? Below is a series of screenshots showing the use of Octave to do the equivalent of what you want.

The background to this is that every 10 minutes data is appended to CSV files via a cronjob - this is analogous to your experiment updating and writing to file every 10 seconds. A script, "rolling_h1_cyclic_plot," is then run which then allows you to "Select Instrument," in your case this would be "Select Relevant file," and then "Number of 10 min Bars" allows selection of the last N lines in "Relevant file." The script loads the last N lines into the workspace, plots the data, and then using the variable editor you can view the raw data.

This approach meets most of your requirements and with a bit of refinement could probably meet all of them.screen1screen2screen3screen4screen5

babelproofreader
  • 264
  • 1
  • 4