TxtSushi

TxtSushi is a collection of command line utilities for processing comma-separated and tab-delimited files (AKA flat files, spreadsheets). By focusing exclusively on processing text files with a tabular structure, TxtSushi simplifies common tasks like filtering, joining and transformation that would take some effort to accomplish with a more powerful scripting language.

The source code is available under the GPLv3 open source license. I am willing to consider requests for licensing the code under other licenses.

TxtSushi is still under active development and has not yet reached 1.0. This means that you can expect it to be a little bit rough around the edges but it should generally work as advertised.

Building/Revision Control/Issue Tracking/Announcements:

• TxtSushi can be built with GHC >= 6.10.1 and Cabal. Since TxtSushi is on hackage you should be able to just do:

  cabal update
  cabal install txt-sushi

  Make sure that the cabal bin directory (usually ~/.cabal/bin) is on your PATH
• Revision Control: The codebase is available in a darcs repository on patch-tag
• Issue Tracking: hosted on the TxtSushi page on google code.
• Announcements: TxtSushi Announcement RSS feed.

tssql:

The tssql utility allows you to use a subset of the SQL SELECT grammar on your flat files. See the tssql documentation for details.

csvtotab:

Converts a comma-delimited file to a tab-delimited file. This command takes one argument which is either a '-' for standard input or a filename.

Pretty format a comma-delimited file. Formats columns so that all data in column lines up (see examples above). This command takes one argument which is either a '-' for standard input or a filename.

Transpose a CSV file. This command takes one argument which is either a '-' for standard input or a filename.

Converts a tab-delimited file to a comma-delimited file. This command takes one argument which is either a '-' for standard input or a filename.

Pretty format a tab-delimited file. Formats columns so that all data in column lines up. This command takes one argument which is either a '-' for standard input or a filename.

Transpose a tab-delimited file. This command takes one argument which is either a '-' for standard input or a filename.

Adds a header row to a CSV file. Uses column names like col1, col2, col3... This command takes one argument which is either a '-' for standard input or a filename.

Links

There are many other great tools that help you work with flat files. Email me (keithshep@gmail.com) if you think there are other tools that should be added to this list:

Utility	Notes
csvfix	Very similar features to TxtSushi, uses many small utilities rather than SQL syntax
MS Log Parser	Does not join tables, only works with MS Windows, closed source, works with many more data types than just flat files (XML, windows registry ...)
Fsdb	Similar features to TxtSushi but uses many utilities rather than SQL
shsql	Intended to work on mutable persistent CSV data where TxtSushi is more inteded to work on "data streams".
HaExcel	An interesting tool that can be used to intelligently refactor spreadsheets into real databases
MySQL CSV Storage	You can now use CSV files as the storage backend for MySQL