Data-round-tripping: moving chemical data around.

For those of us who were around in 1985, an important chemical IT innovation occurred. We could acquire a computer which could be used to draw chemical structures in one application, and via a mysterious and mostly invisible entity called the clipboard, paste it into a word processor (it was called a Macintosh). Perchance even print the result on a laserprinter. Most students of the present age have no idea what we used to do before this innovation! Perhaps not in 1985, but at some stage shortly thereafter, and in effect without most people noticing, the return journey also started working, the so-called round trip. It seemed natural that a chemical structure diagram subjected to this treatment could still be chemically edited, and that it could make the round trip repeatedly. Little did we realise how fragile this round trip might be. Years later, the computer and its clipboard, the chemistry software, and the word processor had all moved on many generations (it is important to flag that three different vendors were involved, all using proprietary formats to weave their magic). And (on a Mac at least) the round-tripping no longer worked. Upon its return to (Chemdraw in this instance), it had been rendered inert, un-editable, and devoid of semantic meaning unless a human intervened. By the way, this process of data-loss is easily demonstrated even on this blog. The chemical diagrams you see here are similarly devoid of data, being merely bit-mapped JPG images. Which is why, on many of these posts, I put in the caption Click for 3D, which gives you access to the chemical data proper (in CML or other formats). And I throw in a digital repository identifier for good measure should you want a full dataset.

It is only now that we (more specifically, this user) understand what had happened under-the-hood to break this round-tripping. In 1984, when Apple produced the Mac, they also produced a most interesting data format called PICT. A human saw the PICT as a PICTure, but the computer saw more. It (could) see additional data embedded in the PICT. The clipboard supported the PICT format, which meant that both picture and data could be transferred between programs. And ChemDraw and Word also understood this. Hence the ability to round-trip noted above (it has to be said between specifically these programs).

Times moved on and the limitations of PICT set in. Apple refocussed on the PDF format. Related, notice, to the Postscript format that Adobe had introduced in order to allow high quality laserprinting. PICT support was abandoned, and the various components no longer carried recognisable data (specifically the clipboard or the ability of Word to recognise the data). Round-tripping broke. Does this matter? Well, one colleague where I work had accumulated more than 1000 chemical diagrams, which he decided to store in Powerpoint (and yes, he threw the original Chemdraw files away). The day came when he wanted to round trip one of them. And of course he could not. He was rather upset I have to say!

PDF was not really a format designed to carry data (see DOI: 10.1021/ci9003688). But, bless their hearts, the three vendors involved in this story all agreed to support data embedded in the PDF hamburger (and Abobe to tolerate it) and now once again, a structure diagram can move into an Office program (on Mac) and out again and retain its chemical integrity. What lessons can be learnt?

  1. Firstly, out of side, out of mind. The clipboard is truly mostly out of sight, and it was not really designed from the outset to preserve data properly. Nowadays I wonder whether clipboards in general recognise XML (and hence CML) and preserve it. I truly do not know. But they should.
  2. Secondly, any system which relies on three or four commercial vendors, who at least in the past, devised proprietary formats which they could change without warning, is bound to be fragile.
  3. We have learnt that data is valuable. More so than the representation of it (i.e. a 2D or 3D structure diagram). But when its lost, the users should care! And tell the vendors.
  4. Peter Murray-Rust and his team have produced CML4Word (or as Microsoft call it, Chemistry add-in for Word). At its heart is data integrity. Fantastic! But I wonder if it survives on Microsoft’s clipboard (I know it does not on Apple’s, since CML4Word is not available on that OS. And is unlikely to ever become so).
  5. And I can see history about to repeat itself. The same seems about to happen on new devices such as the Apple iPad. It too has copy/paste via a clipboard. I bet this will not round trip chemistry (or much other) data! Want to bet that the lessons of this story have not yet been learnt?

Oh, for those who wish to round-trip chemistry on a Mac, you will have to acquire ChemDraw 12.0.2 and Word 2011 (version 14.01), as well as OS X 10.6 for it to work.

Tags: , , , , , , , , , , , , , , , , , , , , ,

5 Responses to “Data-round-tripping: moving chemical data around.”

  1. […] an important blog post on whether we can expect to copy data (sic) from one environment to another: http://www.ch.ic.ac.uk/rzepa/blog/?p=2874 Excerpts: For those of us who were around in 1985, an important chemical IT innovation occurred. We […]

  2. Glenn Howes says:

    Just to let people know, the changes to PDF are pretty standardized and fit well within the design of PDF. I’ve a blog post on how this embedding will be done in 10.7 and above:
    http://sprinkleofcocoa.blogspot.com/2010/09/finally-apple-embraces-standard-for.html

    ChemDraw’s PDF clipboard format embeds a compressed (ZLib) CDXML document in the Metadata stream for the PDF document as a whole. The use of CDXML in a standard location should allow people some peace of mind in terms of data archiving.

  3. Henry Rzepa says:

    That is a great explanation on your blog Glenn. You describe the PDF clipboard as beautiful! But how about that other elephant in the room, Microsoft Office on Windows? Will the data be recognised if it finds itself there? Also, any information on OpenOffice?

  4. Glenn Howes says:

    Well, I described PDF as beautiful mainly because of how nicely you can render splines and text in comparison to how ugly PICT can be.

    I’m afraid I don’t have any experience with the relationship between Microsoft Office and PDF on Windows. ChemDraw for Windows doesn’t generate a PDF clipboard (or PDF at all), nor does ChemDraw for Windows render PDF clippings. When new versions of Microsoft Office come out, we do what we can to maintain compatibility, as this compatibility is one of the most important features of ChemDraw.

    I can’t speak to Open Office, I don’t believe we officially support interoperability with it.

    (I am not speaking as an official voice of CambridgeSoft. I just implemented ChemDraw for Mac’s PDF support.)

  5. […] use in whichever context it finds itself must be a holy grail for all scientists and chemists. I posted earlier on the fragile nature of molecular diagrams making the journey between the editing program used to […]

Leave a Reply