"Reading" a proprietary format file

ntg · 19 Mar 2012 at 21:41

Fellow computer wizards, I'm in need of some help.

I have files which are created in a proprietary format, read only by a specific software application. Now, through this application I have the option to export the data contained within the files in a .csv format (which I can then go on to process in excel etc.).

These files are akeen to database files (i.e. loads of different columns with numerical data or a select number of text strings such as ON, OFF etc.) so I know that they essentially contain pretty much text data. The software reads the data and creates charts and graphical representations.

Everything is fine so far, I got the software and I can export my data in .csv to manipulate through excel. My problem though is that this software does not offer batch processing and I need to analyse about 50 files per day for periods of about 3-6 months. This means that I have to manually open all the files and manually click through to export in .csv.

Looking at the files with a hex editor it's all a bunch of rubbish, clearly obfuskated somehow (or compressed perhaps?) by the software that creates them. Since I have the output and I can see (through the software) what they contain, is there a way to "read" the original files myself, without the need of the software, and dump their innards into a .csv file? If so, please direct me to where I can find more info on this.

Disclaimer: this has nothing to do with any sort of illegal or questionable activity. I'm just trying to make my work a bit easier.

chaosophy · 19 Mar 2012 at 22:49

Might be easier to look into seeing if the app is callable as a COM object and then look at scripting something with VB or PowerShell.

Other wise you're looking into trying to reverse engineer/decrypt it's file format, which while I guess is technically possible seeing as you have the source and the output, would probably take a couple of code breaking maths heads and lot of computing power.

What is the software package? Is this something in house or off the shelf? Might be quicker to go back to the vendor/developer.

ntg · 19 Mar 2012 at 23:05

chaosophy said:
Might be easier to look into seeing if the app is callable as a COM object and then look at scripting something with VB or PowerShell.

Other wise you're looking into trying to reverse engineer/decrypt it's file format, which while I guess is technically possible seeing as you have the source and the output, would probably take a couple of code breaking maths heads and lot of computing power.

What is the software package? Is this something in house or off the shelf? Might be quicker to go back to the vendor/developer.

I think the first option might be possible - although I don't have the knowledge to do it myself, it will be some other smart cookie colleague. I was thinking that it might be obfuskated by some popular piece of software (packager?packer?), of which info might be publicly available that would make the RE process more straightforward. Any way to "test" that? I don't know if what I'm referring to even exists but, are there any utilities that can identify popular packer signatures? That could be a starting point.

As for the developer, it's quite an obscure, highly niche software package and the developer does not plan to add such functionality (and the costs would be much more than any business case I could put together to justify for).

Pho · 19 Mar 2012 at 23:44

Trying to decode the file format yourself will probably be impossible (or rather, take far too long to work it out unless the specs are published).

Could you not create some sort of macro with say AutoHotKey - alt-tab to a folder, press enter to open file, switch to your software and export, go back to your folder, press down to select the next item and repeat? I use it to add macro functionality to some old software so I can press CTRL+F1 say and it goes off and does several actions.

ntg · 20 Mar 2012 at 10:33

Pho said:
Trying to decode the file format yourself will probably be impossible (or rather, take far too long to work it out unless the specs are published).

Could you not create some sort of macro with say AutoHotKey - alt-tab to a folder, press enter to open file, switch to your software and export, go back to your folder, press down to select the next item and repeat? I use it to add macro functionality to some old software so I can press CTRL+F1 say and it goes off and does several actions.

That looks like a very good idea. Will it be able to figure out that it needs to open the 2nd, 3rd...Zth file in a folder by itself though? The filenames will change every day (i.e. date is appended to the filename).

Pho · 20 Mar 2012 at 10:56

Well it's a scripting language so you could probably code it but it''ll probably be a pain.

You could always write a batch file which copies the files out of the subfolders into a single flat folder first maybe?

ntg · 20 Mar 2012 at 13:58

Pho said:
Well it's a scripting language so you could probably code it but it''ll probably be a pain.

You could always write a batch file which copies the files out of the subfolders into a single flat folder first maybe?

I'll look into that, thank you very much for this suggestion.

Pho · 20 Mar 2012 at 14:36

Out of interest what is the software? Never know, someone here might have used it.

ntg · 20 Mar 2012 at 15:48

Pho said:
Out of interest what is the software? Never know, someone here might have used it.

I'm afraid I don't want to say as it's a very niche product and there are very-very few companies in UK that could be using it at the moment, and I don't want to "photograph" where I work.

Pho · 20 Mar 2012 at 17:15

Sounds ominous, and I'm guessing you don't work for Tesco

.

Moredhel · 20 Mar 2012 at 19:59

Sounds very much like Lotus notes exports. Had a project a few years back migrating a client to SharePoint from Lotus notes and had that exact pain with their database dumps. >_<