"Reading" a proprietary format file

ntg

ntg

Associate
Joined
24 Nov 2008
Posts
2,499
Fellow computer wizards, I'm in need of some help.

I have files which are created in a proprietary format, read only by a specific software application. Now, through this application I have the option to export the data contained within the files in a .csv format (which I can then go on to process in excel etc.).

These files are akeen to database files (i.e. loads of different columns with numerical data or a select number of text strings such as ON, OFF etc.) so I know that they essentially contain pretty much text data. The software reads the data and creates charts and graphical representations.

Everything is fine so far, I got the software and I can export my data in .csv to manipulate through excel. My problem though is that this software does not offer batch processing and I need to analyse about 50 files per day for periods of about 3-6 months. This means that I have to manually open all the files and manually click through to export in .csv.

Looking at the files with a hex editor it's all a bunch of rubbish, clearly obfuskated somehow (or compressed perhaps?) by the software that creates them. Since I have the output and I can see (through the software) what they contain, is there a way to "read" the original files myself, without the need of the software, and dump their innards into a .csv file? If so, please direct me to where I can find more info on this.

Disclaimer: this has nothing to do with any sort of illegal or questionable activity. I'm just trying to make my work a bit easier.
 
Might be easier to look into seeing if the app is callable as a COM object and then look at scripting something with VB or PowerShell.

Other wise you're looking into trying to reverse engineer/decrypt it's file format, which while I guess is technically possible seeing as you have the source and the output, would probably take a couple of code breaking maths heads and lot of computing power.

What is the software package? Is this something in house or off the shelf? Might be quicker to go back to the vendor/developer.
 
Might be easier to look into seeing if the app is callable as a COM object and then look at scripting something with VB or PowerShell.

Other wise you're looking into trying to reverse engineer/decrypt it's file format, which while I guess is technically possible seeing as you have the source and the output, would probably take a couple of code breaking maths heads and lot of computing power.

What is the software package? Is this something in house or off the shelf? Might be quicker to go back to the vendor/developer.

I think the first option might be possible - although I don't have the knowledge to do it myself, it will be some other smart cookie colleague. I was thinking that it might be obfuskated by some popular piece of software (packager?packer?), of which info might be publicly available that would make the RE process more straightforward. Any way to "test" that? I don't know if what I'm referring to even exists but, are there any utilities that can identify popular packer signatures? That could be a starting point.

As for the developer, it's quite an obscure, highly niche software package and the developer does not plan to add such functionality (and the costs would be much more than any business case I could put together to justify for).
 
Trying to decode the file format yourself will probably be impossible (or rather, take far too long to work it out unless the specs are published).

Could you not create some sort of macro with say AutoHotKey - alt-tab to a folder, press enter to open file, switch to your software and export, go back to your folder, press down to select the next item and repeat? I use it to add macro functionality to some old software so I can press CTRL+F1 say and it goes off and does several actions.
 
Trying to decode the file format yourself will probably be impossible (or rather, take far too long to work it out unless the specs are published).

Could you not create some sort of macro with say AutoHotKey - alt-tab to a folder, press enter to open file, switch to your software and export, go back to your folder, press down to select the next item and repeat? I use it to add macro functionality to some old software so I can press CTRL+F1 say and it goes off and does several actions.

That looks like a very good idea. Will it be able to figure out that it needs to open the 2nd, 3rd...Zth file in a folder by itself though? The filenames will change every day (i.e. date is appended to the filename).
 
Well it's a scripting language so you could probably code it but it''ll probably be a pain.

You could always write a batch file which copies the files out of the subfolders into a single flat folder first maybe?
 
Well it's a scripting language so you could probably code it but it''ll probably be a pain.

You could always write a batch file which copies the files out of the subfolders into a single flat folder first maybe?

I'll look into that, thank you very much for this suggestion.
 
Out of interest what is the software? Never know, someone here might have used it.
 
Out of interest what is the software? Never know, someone here might have used it.

I'm afraid I don't want to say as it's a very niche product and there are very-very few companies in UK that could be using it at the moment, and I don't want to "photograph" where I work.
 
Sounds very much like Lotus notes exports. Had a project a few years back migrating a client to SharePoint from Lotus notes and had that exact pain with their database dumps. >_<
 
Back
Top Bottom