Centering figures and the index

[aburkov]

Hi! I'm working on a book and I have 2 issues. In the past, I used LaTeX because my books have some math in them. However, Amazon KDP rejected my last book on the premise that they cannot print the PDF that was generated from a LaTeX source. I tried all possible PDF tricks hoping they will accept the PDF but no luck.

This time I decided to use a different tool than LaTeX. After multiple tests, there are no perfect options. Writing in MS Word from scratch is not an option. I need sources so that I can work with publishers if needed. Asciidoc cannot render the inline math characters: they aren't aligned with the rest of the characters in the sentence in the PDF. So, I decided to work in markdown and then use pandoc to convert markdown to docx, and then docx to PDF if needed.

I managed to find various filters that implement the referencing of figures and equations, but I couldn't find anything for the following:

1. Centering figures and figure captions (they are always left-aligned) in the resulting docx.
2. Generating the alphabetical index at the end of the book. With LaTeX, I used makeindex and annotated the index terms in the text with \index{term} but when I convert a markdown file with these annotation to docx with pandoc, these annotations disappear. At the same time, pandoc generates the table of contents without problems. Is there a way to get the index too somehow?

If I cannot solve these two issues, I will have to manually add the index and manually fix the figure alignment which is a problem because if I have to change the source markdown once again, I will have to fix the docx from scratch once again.

If anyone can help me here, I will be delighted.

[jgm]

Amazon KDP rejected my last book on the premise that they cannot print the PDF that was generated from a LaTeX source.

Did they say why?

For docx, did you try using a --reference-doc that modifies the Figure or Captioned Figure style? You should be able to modify this style to make figures be centered.

I can't help with an index; presumably Word has a way to generate an index but I'm not familiar.

[aburkov]

Did they say why?

"We cannot successfully and consistently print files created with LaTeX/pdfTex software. It will be necessary to create a new manuscript file using a new PDF producer. "

For docx, did you try using a --reference-doc that modifies the Figure or Captioned Figure style?

It worked! Thanks a lot. Now I only have the index problem :-) Tried to make ChatGPT and Opus write me a Python script that would add index items to the docx after pandoc but nothing worked so far.

[jgm]

My guess is that there is a way to make it work via LaTeX. They are probably rejecting it for some specific reason (e.g. the fonts aren't embedded or the color table is wrong). I'll bet there's a LaTeX incantation that would work.

[jgm]

E.g. on this thread it's reported that embedding fonts in the latex-generated PDF using ghostscript allowed it to be uploaded. You might look into this further.

[aburkov]

I tried all tricks, including embedding fonts, using PDF/X-1a, flattening the file using Adobe Distiller and converting it back and forth, opening the PDF in a browser and then printing it using a PDF printer. You can imagine that I tried everything given that I spent months typesetting the book in LaTeX and I've got a rejection from basically a monopoly in self-publishing.

They don't tell what's wrong. They repeat this standard sentence "We cannot successfully and consistently print files created with LaTeX/pdfTex software. It will be necessary to create a new manuscript file using a new PDF producer." and "You must ensure all transparencies and layers are flattened." Oh, I flattened everything flattenable and embedded everything embeddable :-)

The biggest problem with KDP is that they have a robotic rejection process. Any attempt to get a real human look into the issue fails.

This time, I will not risk it, so I will convert my markdown to LaTeX and try it with KDP as plan A. If they reject, I will convert my markdown to docx and this will be my plan B. Relying solely on LaTeX is too risky.

The only drawback of plan B is that I must build the index manually. Looks like I have some progress in my Python script that adds index items to the docx after pandoc. I managed to add one at the end of the document, but it still bugs when I try to add it in the middle. Will continue trying tomorrow.

[iandol]

See #6415 for an existing issue on native Pandoc indices. There are some Pandoc filter solutions there. For DOCX you could easily parse some custom markup and inject in the raw XML to turn a list of words into indexable ones I think...

EDIT: ODT has much simpler markup for this and IMO a nicer index system in its GUI, you may want to use ODT then convert to DOCX if neccesary...

ODT index mark:

This is a <text:alphabetical-index-mark-start text:id="IMark105553265462528"/>baby<text:alphabetical-index-mark-end text:id="IMark105553265462528"/>

DOCX index mark:

<w:t xml:space="preserve">This </w:t></w:r><w:r w:rsidR="00ED0D90" w:rsidRPr="00861EFF"><w:t xml:space="preserve">is a </w:t></w:r><w:r w:rsidRPr="00861EFF"><w:t>baby</w:t></w:r><w:r w:rsidRPr="00861EFF"><w:fldChar w:fldCharType="begin"/></w:r><w:r w:rsidRPr="00861EFF"><w:instrText xml:space="preserve"> XE "baby" </w:instrText></w:r><w:r w:rsidRPr="00861EFF"><w:fldChar w:fldCharType="end"/></w:r><w:r w:rsidR="006C0245" w:rsidRPr="00861EFF"><w:t>

[iandol]

If you use Scrivener for writing (which for more complex projects is an amazing tool), then you can use Scrivener styles which can be converted into raw XML by its compiler:

https://forum.literatureandlatte.com/t/making-an-index/94742/29

[aburkov]

@iandol Thank you! Questions:

• Can scrivener file be converted to LaTeX or Word? Most technical book publishers accept only these two formats of manuscripts. Some also accept asciidoc.
• I pasted the following code directly into the markdown file:

<w:t xml:space="preserve">This </w:t></w:r><w:r w:rsidR="00ED0D90" w:rsidRPr="00861EFF"><w:t xml:space="preserve">is a </w:t></w:r><w:r w:rsidRPr="00861EFF"><w:t>baby</w:t></w:r><w:r w:rsidRPr="00861EFF"><w:fldChar w:fldCharType="begin"/></w:r><w:r w:rsidRPr="00861EFF"><w:instrText xml:space="preserve"> XE "baby" </w:instrText></w:r><w:r w:rsidRPr="00861EFF"><w:fldChar w:fldCharType="end"/></w:r><w:r w:rsidR="006C0245" w:rsidRPr="00861EFF"><w:t>

The output docx just contain this text, no index mark:

The one on top is from your code. The one on the bottom is what you get if you use the Word button "Mark Entry" manually.

• Do you know the name of the pandoc filter that creates the index entries? I couldn't find them.

[jgm]

Instead of pasting the code directly (it would then be treated as raw HTML and ignored in docx) you could try including it with a raw attribute.

regular text `<w:t> etc.`{=openxml} and more text.
```

[aburkov]

Then the resulting docx cannot be open:

[jgm]

Yeah, injecting XML in this way is very delicate; you have to make sure that the result is something that would be well formed in the context into which it will be injected. This requires a good knowledge of the format and of how the pandoc writer works.

I can see a problem already with what you have above:

<w:t xml:space="preserve">This </w:t></w:r><w:r w:rsidR="00ED0D90" w:rsidRPr="00861EFF"><w:t xml:space="preserve">is a </w:t></w:r><w:r w:rsidRPr="00861EFF"><w:t>baby</w:t></w:r><w:r w:rsidRPr="00861EFF"><w:fldChar w:fldCharType="begin"/></w:r><w:r w:rsidRPr="00861EFF"><w:instrText xml:space="preserve"> XE "baby" </w:instrText></w:r><w:r w:rsidRPr="00861EFF"><w:fldChar w:fldCharType="end"/></w:r><w:r w:rsidR="006C0245" w:rsidRPr="00861EFF"><w:t>

There is no opening <w:r> to be closed by the </w:r> near the beginning.
Similarly at the end there is no closing </w:r>.

[aburkov]

This is a <w:r w:rsidRPr="00861EFF"> <w:t>term</w:t> </w:r> <w:r w:rsidRPr="00861EFF"> <w:fldChar w:fldCharType="begin"/> </w:r> <w:r w:rsidRPr="00861EFF"> <w:instrText xml:space="preserve"> XE "term" </w:instrText> </w:r> <w:r w:rsidRPr="00861EFF"> <w:fldChar w:fldCharType="end"/> </w:r>{=openxml} for the index.

This is a minimal piece of code that works.

Now if I find a way to insert it into the docx using a Python script, it will be Christmas! :-)

[aburkov]

Ok, I'm not proud of this code but it does the job.

The filter that replaces \index{term} with !!!INDEX(term)!!!:

#!/usr/bin/env python3

import panflute as pf
 
def action(elem, doc):
    if isinstance(elem, pf.RawInline):
        # Check if it contains the \index{term} pattern
        if elem.text.startswith('\\index{'):
            term = elem.text[7:-1]  # Extract the term inside \index{}
            # Create a new placeholder span with visible text
            new_text = f"!!!INDEX({term})!!!"
            return pf.Span(pf.Str(new_text), classes=['index'])

def main(doc=None):
    return pf.run_filter(action, doc=doc)

if __name__ == '__main__':
    main()

The docx modifier that replaces !!!INDEX(term)!!! with the index entry:

from docx import Document
from docx.oxml import OxmlElement
from docx.oxml.ns import qn
import re

def replace_text_with_field(doc_path):
    pattern = re.compile(r'!!!INDEX\((.*?)\)!!!')
    doc = Document(doc_path)
    for paragraph in doc.paragraphs:
        for run in paragraph.runs:
            matches = pattern.findall(run.text)
            if matches:
                match = matches[0]
                run_2 = paragraph.add_run('')
                run._r.addprevious(run_2._r)
                fldChar_begin = OxmlElement('w:fldChar')  # Create a new element for the field character
                fldChar_begin.set(qn('w:fldCharType'), 'begin')  # Set the type to 'begin'

                # Create the instruction text element
                instrText = OxmlElement('w:instrText')
                instrText.text = f' XE "{match}" '  # Set the field instruction

                # Append elements to the run
                run_2._r.append(fldChar_begin)
                run_2._r.append(instrText)

                # Create the separate field character
                fldChar_separate = OxmlElement('w:fldChar')
                fldChar_separate.set(qn('w:fldCharType'), 'separate')
                run_2._r.append(fldChar_separate)

                # End field character
                fldChar_end = OxmlElement('w:fldChar')
                fldChar_end.set(qn('w:fldCharType'), 'end')
                run_2._r.append(fldChar_end)
                run.text = pattern.sub('', run.text)

    # Save the document with modifications
    doc.save('../output_with_index_items.docx')

replace_text_with_field('../input_with_placehoders.docx')

[iandol]

If you see my lua filter below you could make this all one filter, i.e. you take the rawinline and output an equivalent openxml rawinline. ODT is much easier to work with so the code is simpler but the idea is the same...

[iandol]

Also in general Lua filters are much faster and flexible as they are built-in to the Pandoc engine. I don't really know Lua but it is easy enough to work with (I use Codeium in VSCode to answer how-to questions), and using mobdebug as a remote debugger so you can step debug the filter even when it is running inside pandoc.

[iandol]

Can scrivener file be converted to LaTeX or Word? Most technical book publishers accept only these two formats of manuscripts. Some also accept asciidoc.

Scrivener is awesome. You work using semantic styles (block and inline), and an outliner where each chunk of a document can get a semantic type. These styles and types are styled containers that can be wrapped in markup at compile time, where the native Scrivener project is converted to any output you desire. It does DOCX/ODT/EPUB/LaTeX etc. natively, but the power lies in combining it to pandoc. It spits out pandoc-flavoured markdown, which can then be processed automatically (Scrivener can run pandoc command for you).

For example this is an index entry in the Scrivener editor:

This will get converted to \index{warp} at compile time and then using my lua filter can be converted to an ODT index entry automatically, no other fussing required...

I pasted the following code directly into the markdown file

Right, as @jgm mentions, this needs wrapping and careful addition.

Do you know the name of the pandoc filter that creates the index entries? I couldn't find them.

There are a couple linked to #6415 as workarounds. However, they do some more complex things. I modified an older lua filter to convert \index{key} syntax to ODT and Typst (and it passes through so it is still compatible with LaTeX):

--[[
	convertIndex.lua: convert \index{item} markup for marking up words for an 
	index into ODT and Typst native alternatives.
	For ODT it uses the native XML markup for index items.
	For Typst, you also need https://typst.app/universe/package/in-dexter
	Author: Ian Max Andolina
	Version:   1.00
	Copyright: (c) 2024 Ian Max Andolina License=MIT, see LICENSE for details
]]

local stringify = (require 'pandoc.utils').stringify

local counter = nil
local function formatindex(format, key) --returns a rawinline for tex, odt and typst
	if not counter then counter = 1 end
	if format:match 'latex' then
		key = "\\index{" .. key .. "}"
		return pandoc.RawInline('tex', key)
	elseif format:match 'odt' or format:match 'opendocument' then
		key = "<text:alphabetical-index-mark-start text:id=\"" .. stringify(counter) .. "\"/>" .. key .. "<text:alphabetical-index-mark-end text:id=\"" .. stringify(counter) .. "\"/>"
		counter = counter + 1
		return pandoc.RawInline('opendocument', key)
	elseif format:match 'typst' then
		key = key .. "#index[" .. key .. "]"
		return pandoc.RawInline('typst', key)
	else
		return key
	end
end

function RawInline(r) -- parse rawinlines looking for a tex \index{key} to convert
	local fmt = FORMAT
	if fmt:match("odt") then fmt = "opendocument" end
	if fmt:match("tex") then fmt = "latex" end
	kw = stringify(r.text)
	if string.match(r.format, "tex") and string.match(kw, "\\index") then
		kw = kw:match("\\index{(.-)}")
		if string.len(kw) > 0 then
			return formatindex(fmt, kw)
		end
	end
end

You run it like this:

 ▶︎ pandoc --verbose -L convertIndex.lua -t typst

This is a \index{label}. More text.                                   

[INFO] Running filter /Users/ian/.local/share/pandoc/filters/convertIndex.lua
[INFO] Completed filter /Users/ian/.local/share/pandoc/filters/convertIndex.lua in 10 ms

This is a label#index[label]. More text

 ▶︎ pandoc --verbose -L convertIndex.lua -t opendocument

This is a \index{label}. More text.

[INFO] Running filter /Users/ian/.local/share/pandoc/filters/convertIndex.lua
[INFO] Completed filter /Users/ian/.local/share/pandoc/filters/convertIndex.lua in 10 ms

<text:p text:style-name="Text_20_body">This is a
<text:alphabetical-index-mark-start text:id="1"/>label<text:alphabetical-index-mark-end text:id="1"/>.
More text.</text:p>

 ▶︎ pandoc --verbose -L convertIndex.lua -o out.odt     

This is a \index{label}. More text.

[INFO] Running filter /Users/ian/.local/share/pandoc/filters/convertIndex.lua
[INFO] Completed filter /Users/ian/.local/share/pandoc/filters/convertIndex.lua in 9 ms

The ODT generated by pandoc has the correct markup to generate an index, no more tweaking necessary:

[aburkov]

No luck:

This is the command I use: pandoc book.md -L index.lua -L include-files.lua -M chapters -F pandoc-crossref --metadata-file=style.yaml --toc -s -o book.docx --reference-doc template.docx

Archive.zip

Word for Mac: Version 16.84 (24041420)

ProductName: macOS
ProductVersion: 14.4.1
BuildVersion: 23E224

[aburkov]

Ok, it works if I don't use --lua-filter=include-files.lua. Can filters interfere?

Also, this "unreadable content" warning still appears.

[iandol]

Yes, filters can interfere, you can change the order of them (the order matters, also try to put pandoc-crossref first etc.).

I tried include-files.lua with a simple testcase pandoc -L include-files.lua -L convertIndex.lua -F pandoc-crossref -o out.docx using stdin:

---
title: Test
author: Jane Doe
---

This is an \indext{index} term, and direct \index{term}. A final term\index{Parent:term}.

$$ x = y $$ {#eq:label}

See [@eq:label].

'''{.include}
test.md
'''

More text.

And the DOCX opens in Word fine... Also tried a 20 page Typst document, also worked... At least it isn't a fundamental problem between index.lua and include-files.lua or pandoc crossref...

Try to simplify your test document to a bare minimum and then work forwards. There is an interaction somewhere in what you include (i.e. it could be some content you include). I have only minimally tested the filter so it can certainly be the case there is a bug somewhere.

[aburkov]

It works in that order! The error message is still there though, but the file can be opened nevertheless.

I noticed that the lua code doesn't process the index instance like \index{rectified linear unit|see{ReLU}} correctly. I updated it for the openxml case but because I don't know lua, the fix is not perfect:

local function replaceSeeClause(input)
    -- Replace |see{term} with " \t "See: term in the given string
    return input:gsub("|see{(.-)", '" \\t "See: %1')
end
...
key = prefix .. replaceSeeClause(key) .. suffix

[iandol]

Great, the |see{blah} is a LaTeX feature, and plain text replacement is OK I assume, I will also make this change later for the other output formats, thanks.

I don't understand the error message and don't see it in my Word, perhaps this is a setting or something?