Handling gigantic XML files

Keith Barrows — Fri, 15 Dec 2006 22:42:42 GMT

My XML is not a standard XML layout (standard = parent, child, grandchild, etc data). The client has taken all 1400 tables and called them generically "a category". A category has a name (table name in this case) and a GUID. Each category is broken into attributes (column names in this case). Then the attributes are joined to the categories via a many-to-many join table. So, we've condensed (screwed up) the relational layout into 3 tables. This is dumped to a schema.xml with <ATTRIBUTES>, <CATEGORIES> & <CATEGORY_ATTRIBUTES> as the main sections.

The data is then exported from all 1400 tables into a single xml document that is comprised of "data point" nodes. (I do not have an XSD of any sort so am waiting patiently for my system to load the 8+GB file). I am assuming it is structured something like:

There is probably more to it than that but let's go with this right now.

I now need to write an app that reads in the schema.xml file (10KB in size) and display the Table (category) list. User selects a table and gets the column (attribute) list. I then take that category/attribute and have to search the data.xml doc for all nodes with the same value. The idea is to give a developer a look into the data, let them pick what the PK (unique value) column should be and see if there are dups. If there are it is a bad data.xml file...

Any suggestions? The logic is pretty straight forward. I just want the user experience to be a little better than molasses in a cryogenic tank...

Testing DataSet -> XML - DataSet

Keith Barrows — Tue, 25 Jul 2006 16:45:09 GMT