--- /dev/null
+# The --encode-output-strings Flag
+
+## What's this about?
+
+This is about making perltidy work better as a filter when called from other
+Perl scripts. For example, in the following example a reference to a
+string `$source` is passed to perltidy and it stores the formatted result in
+a string named `$output`:
+
+```
+ my $output;
+ my $err = Perl::Tidy::perltidy(
+ source => \$source,
+ destination => \$output,
+ argv => $argv,
+ );
+```
+
+For a filtering operation we expect to be able to directly compare the source and output strings, like this:
+
+```
+ if ($output eq $source) {print "Your formatting is unchanged\n";}
+```
+
+Or we might want to optionally skip the filtering step and pass the source
+directly on to the next stage of processing. This requires that the source
+and output strings be in the same storage mode.
+
+The problem is that in versions of perltidy prior to 2022 there was a use case
+where this was not possible. That case was when perltidy received an encoded
+source and decoded it from a utf8 but did not re-encode it before storing it in
+the output string. So the source string was in a different storage mode than
+the output string, and a direct comparison was not meaningful.
+
+This problem is an unintentional result of the historical evolution of perltidy and needs to be fixed.
+
+The same problem occurs if the destination is an array rather than a string,
+so for simplicity we can limit this discussion to string destinations, which
+are more common.
+
+## How will the problem be fixed?
+
+A fix is being phased in over a couple of steps. The first step was to
+introduce a new flag in in version 20220217. The new flag is
+**--encode-output-strings**, or **-eos**. When this is set, perltidy will fix
+the specific problem mentioned above by doing an encoding before returning.
+So perltidy will behave well as a filter when **-eos** is set.
+
+To illustrate using this flag in the above example, we could write
+
+```
+ my $output;
+
+ # Make perltidy versions after 2022 behave well as a filter
+ $argv .= " -eos" if ($Perl::Tidy::VERSION > 20220101);
+ my $err = Perl::Tidy::perltidy(
+ source => \$source,
+ destination => \$output,
+ argv => $argv,
+ );
+```
+
+With this modification we can make a meaningful direct comparison of `$source` and `$output`. The test on `$VERSION` allows this to work with older versions of perltidy (which would not recognize the flag -eos). An update such as the above can be made right now to facilitate a smooth transition to the new default.
+
+In the second step, possibly later in 2022, the new **-eos** flag will become the default.
+
+## What can go wrong?
+
+The first step is safe because the default behavior is unchanged. But the programmer has to set **-eos** for the corrected behavior to go into effect.
+
+The second step, in which **-eos** becomes the default, will have no effect on programs which do not require perltidy to decode strings, and it will make some programs start processing encoded strings correctly. But there is also the possibility of **double encoding** of the output, or in other words data corruption, in some cases. This could happen if an existing program already has already worked around this issue by encoding the output that it receives back from perltidy. It is important to check for this.
+
+To see how common this problem might be, all programs on CPAN which use Perl::Tidy as a filter were examined. Of a total of 45 programs located, one was identified for which the change in default would definitely cause double encoding, and in one program it was difficult to determine. It looked like the rest of the programs would either not be affected or would start working correctly when processing encoded files. Here is a slightly revised version of the code for the program which would have a problem with double encoding with the new default:
+
+```
+ my $output;
+ Perl::Tidy::perltidy(
+ source => \$self->{data},
+ destination => \$output,
+ stderr => \$perltidy_err,
+ errorfile => \$perltidy_err,
+ logfile => \$perltidy_log,
+ argv => $perltidy_argv,
+ );
+
+ # convert source back to raw
+ encode_utf8 $output;
+```
+
+The problem is in the last line where encoding is done after the call to perltidy. This
+encoding operation was added by the module author to compensate for the lack of an
+encoding step with the old default behavior. But if we run this code with
+**-eos**, which is the planned new default, encoding will also be done by perltidy before
+it returns, with the result that `$output` gets double encoding. This must be avoided. Here
+is one way to modify the above code to avoid double encoding:
+
+```
+ my $has_eos_flag = $Perl::Tidy::VERSION > 20220101;
+ $perltidy_argv .= ' -eos' if $has_eos_flag;
+
+ Perl::Tidy::perltidy(
+ source => \$self->{data},
+ destination => \$output,
+ stderr => \$perltidy_err,
+ errorfile => \$perltidy_err,
+ logfile => \$perltidy_log,
+ argv => $perltidy_argv,
+ );
+
+ # convert source back to raw if perltidy did not do it
+ encode_utf8($output) if ( !$has_eos_flag );
+```
+
+A related problem is if an update of Perl::Tidy is made without also updating
+a corrected version of a module such as the above. To help reduce the chance
+that this will occur the Change Log for perltidy will contain a warning to be
+alert for the double encoding problem, and how to reset the default if
+necessary. This is also the reason for waiting some time before the second step is made.
+
+If double encoding does appear to be occuring after the default change for some program which calls Perl::Tidy, then a quick emergency fix can be made by the program user by setting **-neos** to revert to the old default. A better fix can eventually be made by the program author by removing the second encoding using a technique such as illustrated above.
+
+## Summary
+
+A new flag, **-eos**, has been added to cause Perl::Tidy to behave better as a
+filter when called from other Perl scripts. This flag will eventually become
+the default setting. Programs which use Perl::Tidy as a
+filter can be tested right now with the new **-eos** flag to be sure that double
+encoding is not possible when the default is changed.
+
+## Reference
+
+This flag was originally introduced to fix a problem with the widely-used **tidyall** program (see https://github.com/houseabsolute/perl-code-tidyall/issues/84).
+
+## Appendix, a little closer look
+
+A string of text (here, a Perl script) can be stored by Perl in one of two
+internal storage formats. For simplicity let's call them 'B' mode (for 'Byte')
+mode and 'C' mode (for 'Character'). The 'B' mode can be used for text with
+single-byte characters or for storing an encoded string of multi-byte
+characters. The 'C' mode is needed for actually working with multi-byte
+characters. Thinking of a Perl script as a single long string of text, we can
+look at the mode of the text of a source script as it is processed by perltidy
+at three points:
+
+ - when it enters as a source
+ - at the intermediate stage as it is processed
+ - when it is leaves to its destination
+
+Since 'C' mode only has meaning within Perl scripts, a rule is that outside of
+the realm of Perl the text must be stored in 'B' mode.
+
+The source can only be in 'C' mode if it arrives by a call from another Perl
+program, and the destination can only be in 'C' mode if the destination is a
+Perl program. Otherwise, if the destination is a file, or object with a print
+method, then it will be assumed to be ending its existance as a Perl string and
+will be placed in an end state which is 'B' mode.
+
+Transition from a starting 'B' mode to 'C' mode is done by a decoding operation
+according to the user settings. A transition from an intermediate 'C' mode to
+an ending 'B' mode is done by an encoding operation. It never makes sense to
+transition from a starting 'C' mode to a 'B' mode, or from an intermediate 'B'
+mode to an ending 'C' mode.
+
+Let us make a list of all possible sets of string storage modes to be sure that
+all cases are covered. If each of the three stages list above (entry,
+intermedite, and exit) could be in 'B' or 'C' mode then we would have a total
+of 2 x 2 x 2 = 8 combinations of states. Each end point may either be a file
+or a string reference. Here is a list of them, with a note indicating which
+ones are possible, and when:
+
+ # modes when
+ 1 - B->B->B always ok: (file or string )->(file or string)
+ 2 - B->B->C never (trailing B->C never done)
+ 3 - B->C->B ok if destination is a file or -eos is set [NEW DEFAULT]
+ 4 - B->C->C ok if destination is a string and -neos is set [OLD DEFAULT]
+ 5 - C->B->B never (leading C->B never done)
+ 6 - C->B->C never (leading C->B and trailing B->C never done)
+ 7 - C->C->B only for string-to-file
+ 8 - C->C->C only for string-to-string
+
+So three of these cases (2, 5, and 6) cannot occur and the other five can
+occur. Of these five possible cases, only four are possible when the
+destination is a string:
+
+ 1 - B->B->B ok
+ 3 - B->C->B ok if -eos is set [NEW DEFAULt]
+ 4 - B->C->C ok if -neos is set [OLD DEFAULT]
+ 8 - C->C->C string-to-string only
+
+The first three of these may start at either a file or a string, and the last one only starts at a string.
+
+From this we can see that, if **-eos** is set, then only cases 1, 3, and 8 can occur. In that case the starting and ending states have the same storage mode for all routes through perltidy which end at a string. This verifies that perltidy will work well as a filter in all cases when the **-eos** flag is set, which is the goal here.
+
+The last case in this table, the C->C->C route, corresponds to programs which
+pass decoded strings to perltidy. This is a common usage pattern, and this
+route is not influenced by the **-eos** flag setting, since it only applies to
+strings that have been decoded by perltidy itself.
+
+Incidentally, the full name of the flag, **--encode-output-strings**, is not
+the best because it does not describe what happens in this case. It was
+difficult to find a concise name for this flag. A more correct name would have
+been **--encode-output-strings-that-you-decode**, but that is rather long. A
+more intuitive name for the flag might have been **--be-a-nice-filter**.
+
+Finally, note that case 7 in the full table, the C->C->B route, is an unusual
+but possible situation involving a source string being sent directly to a file.
+It is the only situation in which perltidy does an encoding without having done
+a corresponding previous decoding.
projects / perltidy.git / blobdiff