Support Zotero citations in docx writer

[retorquere]

@jgm suggested I open the issue here (from a discussion at retorquere/zotero-better-bibtex#2862 (comment)).

proposed improvement and the problem it solves.

I currently produce zotero-compatible citations during conversion from markdown to docx. pandoc can parse these as -f docx+citations, I'd love to be able to also produce them in a filter ran while converting from markdown to docx. I currently produce them by replacing the Cite with a RawInline, but I'd much prefer it if I could do it using AST manipulation (which would also save me the trouble of parsing locators and such).

Describe alternatives you've considered.

I've implemented it manually at https://retorque.re/zotero-better-bibtex/exporting/pandoc/index.html#from-markdown-to-zotero-live-citations

[jgm]

I think what you're asking for is for -t docx+citations to include Zotero citations.

There is a problem, though. -f docx+citations will currently handle citations of all sorts, including also EndNote and Mendeley. So we'd either have an odd asymmetry (if the writer just does Zotero) or we'd need to change the extensions so that there's a separate one for each type.

[retorquere]

I'd be OK if I can just manipulate the Cite nodes in the AST such that the result is zotero-compatible citations. It'd be more an filter api change than a command line option change. As far as I can tell, generating live citations is going to require some sort of interaction with zotero/mendeley/etc, not something I see as a strong candidate for building into pandoc itself.

But maybe my current approach suffices. I'd just be happy of being rid of my own prefix/suffix parser when I know Pandoc already does this. If just that could be exposed to the filter that'd simplify filters doing live citations a whole lot.

[jgm]

I'm not really clear on what you want. You can already manipulate Cite nodes in the AST, in Lua filters. What exactly do you think is missing in the Lua API?

generating live citations is going to require some sort of interaction with zotero/mendeley/etc

Not necessarily. Pandoc can use its own citeproc to generate the formatted citation; it would just be a matter of including the extra fields that link it to Zotero. I can't remember how all of this is done, but if you had a sample of a docx with Zotero citations, I could take a look.

[retorquere]

What exactly do you think is missing in the Lua API?

The only way I knew how was to replace the Cite using RawInline. I don't know of a way to modify/create AST nodes to achieve the same results. Plus, there's the parsing of the prefix/postfix/locator stuff, which in my script is currently a fragile heap of lpeg expressions. If just that could reuse the parsing that pandoc undoubtedly already has, that would be huge.

it would just be a matter of including the extra fields that link it to Zotero

That would be the item URI in the case of Zotero, but if your starting point is a Markdown document with citation keys, you won't have them available. It could be done if the user adds a references meta section I suppose. But I reckon users would prefer being able to skip that step.

(edit: the formatted citation would be nice but it isn't actually required to get live citations. My script doesn't)

[jgm]

The only way I knew how was to replace the Cite using RawInline. I don't know of a way to modify/create AST nodes to achieve the same results.

You can modify the AST node perfectly well. https://pandoc.org/lua-filters.html#type-citation
It sounds like you're trying to produce a docx that includes embedded Zotero fields? If so, then this is about docx writer support as I suggested, not about the AST.

That would be the item URI in the case of Zotero, but if your starting point is a Markdown document with citation keys, you won't have them available. It could be done if the user adds a references meta section I suppose. But I reckon users would prefer being able to skip that step.

That would be the item URI in the case of Zotero, but if your starting point is a Markdown document with citation keys, you won't have them available.

That's true. It would require that the citation-key is defined in the database. So maybe this wouldn't be too useful in practice.

[retorquere]

You can modify the AST node perfectly well. https://pandoc.org/lua-filters.html#type-citation
It sounds like you're trying to produce a docx that includes embedded Zotero fields? If so, then this is about docx writer support as I suggested, not about the AST.

I'm trying to describe situation best I can, but I'm very likely misunderstanding the technology and/or using the wrong terminology.

I am indeed trying to produce a docx the includes embedded zotero fields. If the Lua filter can play a role in in the docx writer (I'm not even sure this sentence is meaningful) and in that way avoid duplicating functionality already present in pandoc, that'd be great. If that doesn't fit your strategy for pandoc, my script works. It's just not very elegant.

That's true. It would require that the citation-key is defined in the database. So maybe this wouldn't be too useful in practice.

But it can be made easier to implement. A lot of the code I wrote in my script is going to be generic between implementations for mendeley, zotero, etc.