Skip to content

destructure.js: fix @transifexKey nesting - #1764

Draft
matthew-white wants to merge 1 commit into
masterfrom
transifex-key-nesting
Draft

matthew-white wants to merge 1 commit into
masterfrom
transifex-key-nesting

Conversation

@matthew-white

Copy link
Copy Markdown
Member

Closes getodk/central#2127.

There's some description of the problem in getodk/central#2127 and in code comments, but I plan to add more detail to this PR. It's kind of a complex case.

What has been done to verify that this works as intended?

After this change (along with #1751), destructure.js at last runs without error. 🎉 I used this code in the previous release to run destructure.js, and the resulting Vue I18n JSON matched my expectations.

Given the complexity of the case, I'm tempted to add the first tests of our Transifex scripts. Doing so would probably help document this case.

Why is this the best possible solution? Were any other approaches considered?

I'll add more detail later explaining the underlying problem, after which I think the code approach will make more sense. The code itself isn't optimized, which is one reason why I've marked this PR as draft.

getodk/central#2127 proposes two main approaches: handling this case vs. detecting and disallowing it. This PR implements the former approach (handling this case), but the latter approach would probably be more straightforward. I'll probably continue to run with the former approach for now, since I've already written code for it. However, I may turn to the latter approach (just detecting this case) if I feel like it'd require fewer new tests.

@changeset-bot

This comment was marked as resolved.

}
}
if (changed)
i = 0; // Start over

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure that we really need to start over here, so I may remove this line. I wrote this code pretty fast shortly before release. This is definitely not the most efficient sort algorithm, but I'm also pretty sure that .sort() can't handle this case. Given that, and given the small size of transifexPaths, I think code clarity is more important than sort efficiency.

Comment on lines +697 to +698
const ancestor = ordered[i]; // Subpath
const descendant = ordered[j]; // Longer/deeper path

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like I'm mixing metaphors here, referencing both subpaths and ancestors.

Comment on lines +726 to +729
if (hasPath(sourcePath, translated))
throw new Error(`@transifexKey: attempted to copy the value at ${transifexPath.join('.')} to ${sourcePath.join('.')}, but there is already a value at ${sourcePath.join('.')}.`);
if (!hasPath(transifexPath, translated))
throw new Error(`@transifexKey: attempted to copy the value at ${transifexPath.join('.')} to ${sourcePath.join('.')}, but there is no value at ${transifexPath.join('.')}.`);

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These errors aren't thrown right now, but they helped me track down the problem.

@matthew-white

Copy link
Copy Markdown
Member Author

@alxndrsn, you may be interested in this PR, since you've been helping to get destructure.js back to a runnable state. I was thinking to tag Sadiq for code review, since he's familiar with the @transifexKey mechanism. He'll also be somewhat familiar with this specific case, since he wrote the FormUpload component. Though as I mentioned in the PR description, I'm also considering taking a different approach entirely and just detecting/disallowing this case. In any case, this PR is in a draft state and may change or be closed, but I wanted to let you know that it was up. Definitely feel free to leave any questions or comments.

@matthew-white matthew-white left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding TODO comments

Comment on lines +687 to +688
// representing an ancestor message object. We want to rekey those shorter
// subpaths only after first rekeying the longer/deeper paths. Here, we

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to rekey those shorter subpaths only after first rekeying the longer/deeper paths.

TODO: Explain why.

// Delete the old transifexPath as soon as possible (as soon as the count is
// zero) in order to account for subpaths. We don't want to move a
// descendant message along with its ancestor message object.
if (!hasPath(transifexPath, source) && decrementCount(transifexPath) === 0)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: It was preexisting code, but maybe let's add a code comment explaining !hasPath(transifexPath, source). I think it's the difference between copying vs. moving a message. If a message is just copied, it will still exist at its original path in source.

@matthew-white

Copy link
Copy Markdown
Member Author

I'll add more detail later explaining the underlying problem

OK, here we go! The issue arises for @transifexKey comments involving component.FormNew. We used to have a FormNew component, but in #1535, that component was refactored and renamed to FormUpload.

The FormNew component had a bunch of translations already. By default, translations' Transifex keys are automatically derived from the component name. We didn't want to lose the translations after renaming the component, so we used @transifexKey to preserve the previous Transifex keys, moving the translations in FormUpload to the component.FormNew key:

{
// @transifexKey component.FormNew
"en": {
"introduction": [
// The words "XForms" and "XLSForm" should not be translated.
{
"create": "To create a Form, upload an XForms XML file or an XLSForm Excel file.",
"update": "To update the Draft, upload an XForms XML file or an XLSForm Excel file."
},
{
"full": "If you don’t already have one, there are {tools} to help you design your Form.",
"tools": "tools available"
},
"If you have Form Attachments, you will be able to provide those on the next page, after the Form has been created."
],
"dropZone": {
"full": "Drop a file here, or {chooseOne} to upload.",
"chooseOne": "choose one"
},
"action": {
"uploadAnyway": "Upload anyway"
},
"alert": {
"fileNotReadable": "The file could not be read. It may have been modified or deleted. Please choose the file again."
},
"problem": {
"400_8": "The Form definition you have uploaded does not appear to be for this Form. It has the wrong formId (expected “{expected}”, got “{actual}”).",
// The word "XLSForm" should not be translated.
"400_15": "The XLSForm could not be converted: {error}",
"409_3": "A Form already exists in this Project with the Form ID of “{xmlFormId}”."
},
// Sub-heading for a warning details, followed by the list of fields
"fields": "Fields:",
"warningsText": [
"This file can be used, but it has the following possible problems:",
"Form design warnings:",
"Workflow warnings:",
{
"deletedFormExists": "There is a form with ID \"{value}\" in the Trash. If you upload this Form, you won’t be able to restore the other one with the matching ID.",
"structureChanged": "The following fields have been deleted, renamed or are now in different groups or repeats. These fields will not be visible in the Submission table or included in exports by default.",
"oldEntityVersion": "Entities specification version “{version}” is not compatible with Offline Entities. We recommend using version 2024.1.0 or later."
},
"Please correct the problems and try again.",
{
"create": "If you are sure these problems can be ignored, click the button to create the Form anyway:",
"update": "If you are sure these problems can be ignored, click the button to update the Draft anyway:"
}
]
}
}

The problem is that those translations are a large message object, and we already had individual messages that were separately moved to component.FormNew:

// @transifexKey component.FormNew.action.upload
"upload": "Upload",

// @transifexKey component.FormNew.title.create
// This is the title at the top of a page.
"title": "Create Form",

What I think the problem is (what happens without this PR):

  • The transifexPaths array passed to rekeyTranslations() is sorted by depth (by the path length of the @transifexKey)
  • That means that @transifexKey component.FormNew is executed first, moving component.FormNew in strings_en.json to component.FormUpload for the Vue I18n JSON.
  • When @transifexKey component.FormNew.action.upload is later executed, it doesn't actually move that message, because component.FormNew has already been moved to component.FormUpload: no message is found at component.FormNew.action.upload.
    • I've added a check of this case to the code.
  • So there ends up being a message at component.FormUpload.action.upload for the Vue I18n JSON. But that leads to a structural mismatch, because the FormUpload component doesn't actually contain an action.upload message: that message is supposed to go to en.json5 . That's why the error message says No value found for component.FormUpload.action.upload even though no such key exists in either the Transifex JSON or the Vue I18n JSON.

What this PR does instead:

  • When one @transifexPath is a prefixing "subpath" of another (e.g., component.FormNew is a subpath of component.FormNew.action.upload), we need to execute @transifexPath for the longer path before doing so for the subpath. E.g., we first need to move component.FormNew.action.upload, then component.FormNew. If we did it in the opposite order (component.FormNew, then component.FormNew.action.upload), the result would be that component.FormNew.action.upload would get moved along with component.FormNew, which doesn't work, since we'd lose track of component.FormNew.action.upload (as described above).
  • A move is a copy + a delete. Previously, we would execute all copy operations, then execute all delete operations, but that doesn't work for this case. Even if we copy component.FormNew.action.upload to its correct destination in en.json5 (copying in the correct order), we also need to delete the message at component.FormNew.action.upload before @transifexKey component.FormNew is executed. Otherwise, it will be copied to the FormUpload component as well, leading to a structural mismatch.
    • An additional twist: a single message in the Transifex JSON can be copied to multiple paths in the Vue I18n JSON. So we can only delete a message once all the copy operations are executed, not just the first. (That's probably why previously we executed all copies, then all deletes: to ensure that the delete comes only after the last copy.) So we need to keep track of how many times each message will be copied, and delete it immediately after the last copy (before the first copy for the next key).

@alxndrsn

Copy link
Copy Markdown
Contributor

@alxndrsn, you may be interested in this PR, since you've been helping to get destructure.js back to a runnable state.

I hadn't realised it was fundamentally broken! I think it's definitely time this code got some tests.

@matthew-white

Copy link
Copy Markdown
Member Author

Yeah, this case introduced with #1535 isn't something that destructure.js is able to handle. 😢 We could easily resolve it just by removing @transifexKey for the two problematic messages in en.json5 and FormNewPage. But I think we should either fix it for real or add detection for this sort of case. Either way, adding tests does sound sensible.

@matthew-white

Copy link
Copy Markdown
Member Author

I think it's definitely time this code got some tests.

Related: getodk/central#2151. Once we get the required testing infrastructure in place, we can add one or more tests about the specific case that this PR fixes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

destructure.js fails for complex uses of @transifexKey

2 participants