Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Preserving Twine metadata in Twee sources

This was triggered by the following discussion:

[quote author=Morgan_R link=topic=1393.msg2117#msg2117 date=1391185558]
A collaboration could be fun -- I'd like to do something for the Spring Thing, but at the moment I don't have a particular idea in mind. Unfortunately Twine itself doesn't seem particularly amenable to collaboration -- I just tried to copy-paste across documents, and no go. :/

mth wrote:

TWS files are not very suitable for collaboration, but Twee sources are: changes to them can be merged automatically by a version control system such as Git or Mercurial.

[quote author=Morgan_R link=topic=1393.msg2142#msg2142 date=1391271956]
mth: I don't think I'm willing to give up the node-based interface, but thank you for pointing that out!

The problem was also noted by Matthew Gallant on his blog:
[quote]
Twine is a graphical interface wrapper for a simple plaintext format (called twee). However, the application saves stories in a .tws file, which is just a Python pickle serialization. Since this format is not human readable, it discourages the use of source control. Why bother tracking your changes when you cant understand the incremental differences?


The problem being that we have two file formats, both of which have shortcomings. The TWS format is a binary format on which it is not possible to show differences ("diff" operation) between versions or merge changes from different contributors. The Twee format is plain text and easy to diff and merge, but does not have information about the story layout, so if you export to Twee and import again you lose all the work you did on organizing the story board.

So, what is in the TWS file? I dumped the output from a very short test story and found this:

Global:
  • buildDestination: HTML file for testing
  • saveDestination: location of the TWS file
  • target: story format
  • scale: zoom factor
  • snapping: snap-to-grid
Per passage:
  • selected: is this passage tile currently selected?
  • pos: (x, y) coordinates of the passage in the story board
  • created: creation date/time
  • modified: modification date/time
  • title: passage name
  • tags: passage tags
  • text: passage body text
These can be separated into workspace and story related data. By "workspace", I mean the data that is preserved such that when you open a story, the Twine IDE looks the same as it did when you saved it. In a collaboration scenario, the workspace data would not be shared.

Workspace data of the TWS:

Global:
  • buildDestination: HTML file for testing
  • saveDestination: location of the TWS file
  • scale: zoom factor
Per passage:
  • selected: is this passage tile currently selected?
Story data of the TWS:

Global:
  • target: story format
  • snapping: snap-to-grid
Per passage:
  • pos: (x, y) coordinates of the passage in the story board
  • created: creation date/time
  • modified: modification date/time
  • title: passage name
  • tags: passage tags
  • text: passage body text
I put the story format with the story data, because not all stories can run in any story format: both functionality and presentation might only work in one particular story format.

I put snap-to-grid with the story data since in my opinion consistency is very important when organizing things, so whether the tiles are grid-aligned or not should be a per-project decision and not a per-developer decision. It is a matter of opinion though.

The creation and modification date are tracked by Twine, but are not visible in any way. Are these useful bits of information that are currently under-used or are they an artifact from using a TiddlyWiki as the storage for the passages? In my opinion it would be better to have an external version control system keep track of modification dates rather than have that information in the story itself.

Passage title, tags and body text are already present in the Twee file format. If creation and modification date are considered unimportant and workspace related data are considered out of scope, that leaves the following data from TWS files that is lost when converting to Twee format:
  • (global) target: story format
  • (global) snapping: snap-to-grid
  • (per-passage) pos: (x, y) coordinates of the passage in the story board
Having the target (story format) in the Twee source would help the command line compiler select the right default target, instead of having "sugarcane" as the hardcoded default. So this is an addition that would be useful even when not doing conversions between TWS and Twee.

Having the snap-to-grid setting and the passage positions on the story board in a Twee file would be useful when doing conversions between TWS and Twee, but would not be useful when working in Twee exclusively. However, the amount of extra data would be relatively small, so I don't think it would be problematic.

Therefore, I would like to propose to extend the Twee file format so it can contain the listed extra data, to make round trips between TWS and Twee possible without losing important data. I'd like to hear other people's opinions on whether this is indeed the right approach and whether there are any aspects I have overlooked.

If this approach is agreed upon, the next step would be to design the changes to the file format. The obvious choice for global data would be to put them in a special passage, either an existing one (StorySettings) or a newly created one. For per-passage data there is no obvious solution; I have some ideas but none of them are perfect.

The most important question is what to do about compatibility between versions. There is both backward compatibility and forward compatibility to be taken into account. As an example, let's say that the new format is implemented in Twine 1.5, then dealing with Twine 1.4 is backward compatibility and dealing with Twine 1.6 is forward compatibility.

In terms of backward compatibility, importing 1.4 Twee sources into Twine 1.5 will work exactly like Twee imports work in 1.4: the story board layout information is lost, but there is nothing we can do about that since the information is simply not there in the 1.4 Twee format. When it comes to exporting from Twine 1.5 to Twee, there are a few options:
[list type=decimal]
export to a Twee 1.5 format which can only be compiled with the 1.5 (or newer) Twee compiler
export to a Twee 1.5 format which can be compiled with any version Twee compiler
upon export, let the user choose whether to include layout information or not

While option 2 might look attractive at a first glance, I don't think there is an elegant way to make the file format itself backward compatible, so this option would probably lead to some kind of kludge in the format, meaning that the additional data would look ugly in Twee files. Unless someone has a clever idea here?

The advantage of option 3 would be that if the author chooses to prototype in Twine and finish the story in Twee, it is very easy to strip out the irrelevant data. The disadvantage is that the export process would include an additional dialog step asking the user what data should and should not be exported.

Forward compatibility means that the format is designed in such a way that Twine 1.5 will be able to deal with Twee files from Twine 1.6 without problems. This means that it must be able to load and preserve settings even if those settings don't exist in 1.5. This suggests a generic key-value system; StorySettings is already designed like that, it would make sense to do the same for the per-passage settings.

Comments

  • This would be helpful.

    Option 2 can be done in tags.

    :: Start [test #posx=500 #posy=600]
  • For what it's worth, the upcoming Twine 2 tackles this sort of issue. A published Twine 2 story has all editor data encoded into it (e.g. passage positions). We're doing it by adding data- prefixed attributes to the HTML.

    This doesn't help you directly as far as Twine and twee interoperating goes, though...
  • Does Twine 2 have a plain text representation of its stories? While I don't mind running a node.js application instead of a wxPython application for prototyping, I don't want to give up my workflow (command line + syntax highlighting editor + git) for actual coding.
  • Not in the way I think you want it to. A passage looks like this in the generated HTML:

    <script data-role="passage" data-id="1" data-name="Hello" type="text/twine" data-twine-position="300,300">
    This is my passage source code.
    </script>
  • I'm still a tiny newbie here, but I did want to say I really like this idea.

    Actually, because I *started* with Twine, I never really knew that Twee was just plain text until reading some threads here at Twinery.  (I tried opening up the files created by Twine but, of course, that wasn't so pretty XD)

    FWIW, I wonder if there isn't something that could be between option 1 and 2?  That is, Twine's new "export" feature would create a Twee file with additional metadata that may not work in Twee 1.4, but would be handled (ignored?) in Twee 1.5+  But, because we know what the metadata is, and since it's mostly important for command line workflows, perhaps we could create a small script that would strip out that data so the story could be compiled with Twee 1.4?

    That is, something like:

    (twee-backward) twine-1.5-story.twee | twee ...

    Perhaps I'm underestimating the effort that would need to go into that script, but if it did work like a pre-processor....  Cool right?
  • If backwards compatibility is really wanted, I don't see that there are many choices.  The current twee format consists of two data chunks per passage, the title/tags line and the passage body.  You can't put the data into the latter, as it would be compiled into the passage.  And the former gives you exactly one choice, another square-bracketed section after the tags.  For example:
    ::Passage Title [Tags] [Twine Data]
    The last time I looked at the Twee code, the initTwee() method split on the opening square-bracket ([) to separate the title from the tags.  Used on the above example, that would produce a three element list.  Only the first two are referenced and parsed by current twee compilers, so they'd completely ignore the embedded Twine data.  If the only per-passage info you need to store is the passage's coordinates, that would probably work alright.  You would always have to export the tag container, even it it was empty, to keep the Twine data container safe from parsing though.

    As far as the global data goes, it could be inserted as a special passage.
  • TheMadExile wrote:

    If backwards compatibility is really wanted, I don't see that there are many choices.  The current twee format consists of two data chunks per passage, the title/tags line and the passage body.  You can't put the data into the latter, as it would be compiled into the passage.


    Being able to compile new-style sources with an old Twee is nice, but as you write pretty much the only option is to do tricky things with tags. So I'm not sure it is worth it if we'd be stuck with awkward syntax forever.

    TheMadExile wrote:

    And the former gives you exactly one choice, another square-bracketed section after the tags.


    I was going to write that that wouldn't work, but after checking the code again indeed the text after the second '[' is not parsed for tags, so it will work.

    Let's do some example code fragments:

    :: Style [stylesheet] [twine-position=800,10]
    body { color: black; background-color: white; }

    :: Start [] [twine-position=500,600]
    So it has come to this.
    nomdepony earlier suggested another trick: adding a special character in front of tag names:

    :: Style [stylesheet #twine-position=800,10]
    body { color: black; background-color: white; }

    :: Start [#twine-position=500,600]
    So it has come to this.
    This has the advantage that there is no need for an empty tag list on an untagged passage. However, having the separate list makes the actual tags stand out better, in my opinion. Also, in this case Twine and Twee will be parsing the tags and although it is unlikely existing stories depend on tags starting with such a special character, it will show up in the story board in old Twine versions and might cause other inconveniences. So I prefer the separate lists idea.

    If we don't care about the Twee source being usable in old Twine releases, we can put the metadata on a separate line:

    :: Style [stylesheet]
    :. twine-position=800,10
    body { color: black; background-color: white; }

    :: Start
    :. twine-position=500,600
    So it has come to this.
    This makes it easier to filter out the metadata if it is no longer desired at a certain point in time. Also it would avoid having very long lines if more per-passage metadata is added later.

    I'm not sure if I prefer the second bracketed list idea or the separate lines. Opinions please.

    Note that I used "twine-position=500,600" for the data itself, since that is what Twine is already inserting into HTML today. Also it is similar to what Chris said Twine 2 would be using.

    TheMadExile wrote:

    As far as the global data goes, it could be inserted as a special passage.


    I'm thinking of adding them (story format and snap-to-grid) to StorySettings, or do you think a new special passage would be better?

    nicoles wrote:

    FWIW, I wonder if there isn't something that could be between option 1 and 2?  That is, Twine's new "export" feature would create a Twee file with additional metadata that may not work in Twee 1.4, but would be handled (ignored?) in Twee 1.5+  But, because we know what the metadata is, and since it's mostly important for command line workflows, perhaps we could create a small script that would strip out that data so the story could be compiled with Twee 1.4?


    That's possible, but I think it may not be the most practical solution. If the scenario is someone with Twine 1.4 installed receiving a 1.5 format file, the conversion tool being part of Twine 1.5 is inconvenient. If the scenario is someone with Twine 1.5 installed preparing a 1.4-compatible file for someone else, having an "Export to Twee in 1.4 format" option would be more useful than a separate conversion step.

    When putting the metadata on separate lines, the conversion is as easy as grep -v '^:\.' though, so perhaps all you need is a wiki page describing the process, not an actual conversion tool.
  • mth wrote:
    I'm thinking of adding them (story format and snap-to-grid) to StorySettings, or do you think a new special passage would be better?


    I think it makes sense to use StorySettings.  It's an existing special passage, which is specifically for story settings (currently limited to header settings, but I see no issue with storing compiler setting there, just prefix them or something).  Additionally, there's already a setting which is used by both the compiler and the vanilla headers (obfuscate), so it would really just be a continuation/extension of an existing practice, more or less.
  • mth wrote:

    I'm not sure if I prefer the second bracketed list idea or the separate lines. Opinions please.


    As you write later, I think I like the second option better: using a separate line.  Even though using the second bracket would allow it to be backwards compatible, using the second line would allow more flexibility with options/parameters, particularly if Twee and Twine could "reserve" some, or perhaps, like "data-" prefixes in HTML attributes, it would allow other tools to be created, save their own metadata, and still have the twee compiler work on the stories.

    Does that make sense?

    mth wrote:

    nicoles wrote:

    FWIW, I wonder if there isn't something that could be between option 1 and 2?  That is, Twine's new "export" feature would create a Twee file with additional metadata that may not work in Twee 1.4, but would be handled (ignored?) in Twee 1.5+  But, because we know what the metadata is, and since it's mostly important for command line workflows, perhaps we could create a small script that would strip out that data so the story could be compiled with Twee 1.4?


    That's possible, but I think it may not be the most practical solution. If the scenario is someone with Twine 1.4 installed receiving a 1.5 format file, the conversion tool being part of Twine 1.5 is inconvenient. If the scenario is someone with Twine 1.5 installed preparing a 1.4-compatible file for someone else, having an "Export to Twee in 1.4 format" option would be more useful than a separate conversion step.

    When putting the metadata on separate lines, the conversion is as easy as grep -v '^:\.' though, so perhaps all you need is a wiki page describing the process, not an actual conversion tool.


    Yah, that makes sense.

    I guess I was mostly thinking of the "conversion tool" as being something that could also be used separately?  Like, in theory there's no reason not to update from 1.4 to 1.5, but if there were a bug or some different processing in 1.5, the person may not want to upgrade for their existing stories.  The script (or instructions) would allow them (or their collaborators) to use Twine 1.5 but still compile the story with 1.4

    It's an edge case, and the idea of using :. as a line prefix for these data points (particularly if we can use multiple lines?) would make it rather trivial to manage without an official tool.
  • I don't care for using a separate line. This is information that is totally ancillary to the Twee code, and it makes more sense to me to stick it in a (clearly less important) tag.

    I do like the overall idea of allowing Twee and Twine to share metadata, though, since it means that Twine can be used effectively for visualizing stories written in Twee. Twine is definitely not something I'd want to write in, but I can see the benefit in getting a visual graph of something already written. (To that end, it might be nice to allow for hidden passages that don't show up in Twine at all; if you're using Twine as a simple visualization tool, it would be nice to avoid having to move all of your non-story passage to one side.)
  • Just wanted to say that I also think sharing metadata between Twine and Twee would be useful...
  • So having just finished my first real game in Twine, I have to say that towards the end making changes was a bit of a nightmare. I've gotten used to version control, but as plenty of people have pointed out, git doesn't work particularly well with tws story files. The obvious solution is to write in Twee, and that's what I will likely do for my next project... but then that kills one of the coolest features of Twine, which is the spatial visualization of story passages and connections.

    Since nobody has posted here for a while, I assume nothing has been done with this idea of preserving Twine metadata in Twee output, but I just wanted to check in in case anyone has made any progress privately, and to say that I think this would be a really awesome feature, since it would allow authors to take advantage of version control while still being able to visualize their stories.
  • Twee2 extracts the position metadata from html output into the twee file, and can also put it back into an html file for import.
Sign In or Register to comment.