Specialized ExpressionPattern classes

[mmatera]

This PR adds two ingredients to the Pattern refactor:

• Simple ordered pattern rules are now implemented by FixedBlankTupleExpressionPattern and SimpleOrderedExpressionPattern, which avoid dealing with the special code for handling orderless, flat, or oneidentity attributes.
• After loading all the Definitions, Deferred rules are resolved before starting the evaluations. With this, pattern matching is always done using the simplest pattern matching algorithm.

As motivation, this is the number of typical patterns created in the definitions object at load time:

Shape	Occurences	State
head[__]	746	✅ this round
head[_]	510	✅ this round
head[lit] (single literal atom)	275	already optimal (isliteral)
head[,]	260	✅ this round
head[___]	211	not implemented
head[]	85	alredy optimal (isliteral, vaccum)
head[,,_]	55	✅ this round
head[_,OptionsPattern]	47	more specific
head[_String]	45	typed
head[_List]	44	typed
head[_Integer]	29	typed

[rocky]

Where can I see benchmarks comparing with the existing code in master?

It would be interesting to see what improvements there are with the gs (Gries-Schneider) tests and with the Rubi package.

[mmatera]

Where can I see benchmarks comparing with the existing code in master?

It would be interesting to see what improvements there are with the gs (Gries-Schneider) tests and with the Rubi package.

Just to be clear, the goal of this is not to speed up the evaluations, but to split the complicated general pattern match algorithm into pieces that can be debugged separately. So, if we have a rule of the form Subtract[a_,b_], avoid passing for all the code that decides if Subtract is "flat" or "orderless", or "oneidentity", so we can identify in an easier way how the match happens, and what happens under the hood.
I wanted to avoid doing this to produce more slowdown, so I added some benchmarks to check if the changes make the problem worse. But yes, eventually I want to also see if there were significant improvements, and then I would look at these sets of tests.

[rocky]

Where can I see benchmarks comparing with the existing code in master?
It would be interesting to see what improvements there are with the gs (Gries-Schneider) tests and with the Rubi package.

Just to be clear,

Just to be clear. For the last year or two, you said you wanted to work on Boxing and Form handling. There are these digressions and tangents like this that keep coming up.

Okay, somehow you saw this via MakeBoxes, and then wanted to do something to make this clearer.

But continuing this at this point is now pushing a bit beyond elucidating the existing code so that it can be used.

Just to be clear, there has been a history of, well, I'll just add this improvement without a discussion or understanding of the ways this is typically done, big-picture alternatives. The current approach to pattern matching is one of a couple of approaches. It might be right.

But it might not be.

The more that is done here, may mean more that may need to be rewritten later should we decide to go a different route in handling pattern matching.

This project has suffered greatly from misguided ideas along the lines of working in a myopic fashion like this. Expression caches and implementing triangular meshes for graphics surfaces are two examples. But there are numerous other examples.

[mmatera]

@rocky, it is just I went as far I could go with MakeBoxes refactor, without facing other aspects, like parsing, pattern matching, evaluation, or the internal representation of some numeric classes. Look, for instance, at the remaining fails in

 DEBUGMAKEBOXES=1 pytest  test/format

The first four errors comes from differences in how we represent precision in real numbers, and how we propagate information on how numbers must be formatted.
Then, there are four errors related to parsing, and finally, test_makeboxes_custom and test_makeboxes_custom2 errors are related on how we handle options and patterns. This is why I propose some changes to see more clearly where the differences come from. The proposal here and in the previous rounds points to define what is the API of the pattern module is, what the helper functions for the internal implementation are, and which routes are used to do pattern matching according to the attributes associated with the pattern. If someone else wants to rework the implementation, can just remove everything from mathics.core.pattern apart from the init file, and provide a different implementation. Tests and benchmarks done in this round could also be helpful.

In any case, I am OK with leaving this as a draft for a while, and eventually discarding it if it does not prove useful.

[rocky]

@rocky, it is just I went as far I could go with MakeBoxes refactor, without facing other aspects, like parsing, pattern matching, evaluation, or the internal representation of some numeric classes. Look, for instance, at the remaining fails in

 DEBUGMAKEBOXES=1 pytest  test/format

The first four errors comes from differences in how we represent precision in real numbers, and how we propagate information on how numbers must be formatted. Then, there are four errors related to parsing, and finally, test_makeboxes_custom and test_makeboxes_custom2 errors are related on how we handle options and patterns. This is why I propose some changes to see more clearly where the differences come from. The proposal here and in the previous rounds points to define what is the API of the pattern module is, what the helper functions for the internal implementation are, and which routes are used to do pattern matching according to the attributes associated with the pattern. If someone else wants to rework the implementation, can just remove everything from mathics.core.pattern apart from the init file, and provide a different implementation. Tests and benchmarks done in this round could also be helpful.

In any case, I am OK with leaving this as a draft for a while, and eventually discarding it if it does not prove useful.

Ok. Thanks for the update. I'll take a look at it when I can, and perhaps together we can come up with a plan or isolate and contain problems.