[Koha-bugs] [Bug 20589] New: Add field boosting and use elastic query fields parameter instead of depricated _all

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

Bug ID: 20589
Summary: Add field boosting and use elastic query fields
parameter instead of depricated _all
Change sponsored?: ---
Product: Koha
Version: master
Hardware: All
OS: All
Status: NEW
Severity: enhancement
Priority: P5 - low
Component: Searching - Elasticsearch
Assignee: koha-***@lists.koha-community.org
Reporter: ***@gmail.com

Generate a list of fields for the query_string query fields parameter, with
possible boosts, instead of using "_all"-field. Also add "search" flag in
search_marc_to_field table so that certain mappings can be excluded from
searches. Add option to include/exclude fields in query_string "fields"
parameter depending on searching in OPAC or staff client. Refactor code to
remove all other dependencies on "_all"-field.

(the atomicupdate sql-file is a bit obfuscated since had to be able to run in
multiple time without error because of our local deploy-script, when moving
this all procedure related code can be removed, only the "ALTER TABLE.."-lines
are relevant.)
--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

--- Comment #1 from David Gustafsson <***@gmail.com> ---
Created attachment 74245
-->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=74245&action=edit
Bug 20589: Add field boosting and use query_string fields parameter

Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.

How to test:
1) No reindex is really required, but still, reindex authorities and
biblios to verify that this still works.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
"Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
some biblios and/or authorities searches.

Sponsorded-by: Gothenburg Univesity Library
--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

David Gustafsson <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #74245|0 |1
is obsolete| |

--- Comment #2 from David Gustafsson <***@gmail.com> ---
Created attachment 74249
-->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=74249&action=edit
Bug 20589: Add field boosting and use query_string fields parameter

Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.

How to test:
1) No reindex is really required, but still, reindex authorities and
biblios to verify that this still works.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
"Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
some biblios and/or authorities searches.

Sponsorded-by: Gothenburg Univesity Library
--
You are receiving this mail because:
You are the assignee for the bug.
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

David Gustafsson <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |Needs Signoff
--
You are receiving this mail because:
You are the assignee for the bug.
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

David Gustafsson <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #74249|0 |1
is obsolete| |

--- Comment #3 from David Gustafsson <***@gmail.com> ---
Created attachment 74301
-->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=74301&action=edit
Bug 20589: Add field boosting and use query_string fields parameter

Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.

How to test:
1) No reindex is really required, but still, reindex authorities and
biblios to verify that this still works.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
"Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
some biblios and/or authorities searches.

Sponsorded-by: Gothenburg Univesity Library
--
You are receiving this mail because:
You are the assignee for the bug.
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

--- Comment #4 from David Gustafsson <***@gmail.com> ---
Added a *.perl update script instead of the custom SQL-script.
--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

David Gustafsson <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #74301|0 |1
is obsolete| |

--- Comment #5 from David Gustafsson <***@gmail.com> ---
Created attachment 74373
-->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=74373&action=edit
Bug 20589: Add field boosting and use query_string fields parameter

Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.

How to test:
1) No reindex is really required, but still, reindex authorities and
biblios to verify that this still works.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
"Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
some biblios and/or authorities searches.

Sponsorded-by: Gothenburg Univesity Library
--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

--- Comment #6 from David Gustafsson <***@gmail.com> ---
Had forgotten to take new "search" field into account for mappings, and also
fixed where had not removed all of the _all-field mapping (which would have
caused indexing to crash).
--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

David Gustafsson <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #74373|0 |1
is obsolete| |

--- Comment #7 from David Gustafsson <***@gmail.com> ---
Created attachment 74418
-->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=74418&action=edit
Bug 20589: Add field boosting and use query_string fields parameter

Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.

How to test:
1) No reindex is really required, but still, reindex authorities and
biblios to verify that this still works.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
"Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
some biblios and/or authorities searches.

Sponsorded-by: Gothenburg Univesity Library
--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

--- Comment #8 from David Gustafsson <***@gmail.com> ---
Discovered search where missing in '+as', and '+select' clause, which causes
indexing to crash, will fix.
--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

David Gustafsson <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #74418|0 |1
is obsolete| |

--- Comment #9 from David Gustafsson <***@gmail.com> ---
Created attachment 74419
-->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=74419&action=edit
Bug 20589: Add field boosting and use query_string fields parameter

Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.

How to test:
1) No reindex is really required, but still, reindex authorities and
biblios to verify that this still works.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
"Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
some biblios and/or authorities searches.

Sponsorded-by: Gothenburg Univesity Library
--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

Nick Clemens <***@bywatersolutions.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
See Also| |https://bugs.koha-community
| |.org/bugzilla3/show_bug.cgi
| |?id=18969
--
You are receiving this mail because:
You are the assignee for the bug.
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

Nick Clemens <***@bywatersolutions.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
See Also| |https://bugs.koha-community
| |.org/bugzilla3/show_bug.cgi
| |?id=18316
--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

Nick Clemens <***@bywatersolutions.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@bywatersolutions.com
See Also| |https://bugs.koha-community
| |.org/bugzilla3/show_bug.cgi
| |?id=19604
--
You are receiving this mail because:
You are the assignee for the bug.
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

Nick Clemens <***@bywatersolutions.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Assignee|koha-***@lists.koha-commun |***@gmail.com
|ity.org |
--
You are receiving this mail because:
You are watching all bug changes.
You are the assignee for the bug.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

Katrin Fischer <***@bsz-bw.de> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@bsz-bw.de
Summary|Add field boosting and use |Add field boosting and use
|elastic query fields |elastic query fields
|parameter instead of |parameter instead of
|depricated _all |deprecated _all
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

Alex Arnaud <***@biblibre.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@biblibre.com

--- Comment #10 from Alex Arnaud <***@biblibre.com> ---
Searchable parameter on search fields (staff/opac) is nice.
But, is searchable parameter on mapping useful? We can just remove the mapping,
right?

This patch could be rebased on bug 18316, that brings boost for simple search.
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

--- Comment #11 from David Gustafsson <***@gmail.com> ---
Yes, I think the best option would be to try getting bug_18316 merged, and then
refactor this upon that. I completely missed bug_18316 before starting to work
on this. I tried applying bug_18316 but was unable to do so because of some
git-error, think there might be something wonky with the current patches.
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

--- Comment #12 from David Gustafsson <***@gmail.com> ---
And about the searchable parameter, I think we are going to need this in the
future so included it as did not see a major downside of doing so. The problem
this might solve is that we would want to have exclusive "sort"-field mappings,
not included in the main field. The idea is to allow mappings to be written on
the form "035a|035b|035c", and this would mean that the first field that has a
value gets mapped, the rest skipped. We have disabled sorting since it's very
unpredictable for multivalued fields, and this was seen as a possible future
solution.
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

David Gustafsson <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Depends on| |18316


Referenced Bugs:

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18316
[Bug 18316] Add weighting/relevancy options to ElasticSearch
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

--- Comment #13 from Katrin Fischer <***@bsz-bw.de> ---
Hi David, if you write bug 18316 instead of bug_18316, bugzilla will
automagically link it. :)
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

Séverine Queune <***@bulac.fr> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@bulac.fr

--- Comment #14 from Séverine Queune <***@bulac.fr> ---
I a bit lost because of last comments and allusion about merging it with bug
18316 : should this one be tested ?
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.k

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

David Gustafsson <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|Needs Signoff |BLOCKED
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

Martin Renvoize <***@ptfs-europe.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@ptfs-europe
| |.com
Status|BLOCKED |Needs Signoff
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

David Gustafsson <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|Needs Signoff |In Discussion

--- Comment #15 from David Gustafsson <***@gmail.com> ---
I think this patch in it's current state needs to be refactored with regard to
the changes introduces in Bug 18316 once has been pushed to master. I will do
this, but will probably wait until it has been pushed (which it seems close to
be).
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

--- Comment #16 from David Gustafsson <***@gmail.com> ---
I have started working on this now since 18316 has been merged. Will update
beginning of next week.
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589
Bug 20589 depends on bug 18316, which changed state.

Bug 18316 Summary: Add weighting/relevancy options to ElasticSearch
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18316

What |Removed |Added
----------------------------------------------------------------------------
Status|Pushed to Master |RESOLVED
Resolution|--- |FIXED
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

David Gustafsson <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #74419|0 |1
is obsolete| |

--- Comment #17 from David Gustafsson <***@gmail.com> ---
Created attachment 82503
-->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=82503&action=edit
Bug 20589: Add field boosting and use query_string fields parameter

Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.

How to test:
1) Reindex authorities and biblios.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
"Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
some biblios and/or authorities searches.

Sponsorded-by: Gothenburg Univesity Library
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

--- Comment #18 from David Gustafsson <***@gmail.com> ---
This is still work in progress. Have not adjusted tests for the refactored code
for example. There are parts of this patch regarding fixes for authorities
search that could be a separate patch, but since removing '_all' also requires
adjusting this patch such a patch would probably cause merge conflict with this
one. So it could be simple to bundle theses changes as one patch.
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

--- Comment #19 from David Gustafsson <***@gmail.com> ---
*but since removing '_all' also requires adjusting this _part_ such a patch
would probably cause merge conflict with this one. So it could be simple to
bundle theses changes as one patch.
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

--- Comment #20 from David Gustafsson <***@gmail.com> ---
I will write a summary of the changes with motivations behind each one as this
can be difficult to understand by just looking at the code. I'm also a little
bit confused my self with regards to the correct behaviour of
build_auth_query_compat for example. And would probably have to setup a Zebra
instance to find out exactly how the different options are supposed to work.
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

David Gustafsson <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #82503|0 |1
is obsolete| |

--- Comment #21 from David Gustafsson <***@gmail.com> ---
Created attachment 82544
-->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=82544&action=edit
Bug 20589: Add field boosting and use query_string fields parameter

Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.

How to test:
1) Reindex authorities and biblios.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
"Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
some biblios and/or authorities searches.

Sponsorded-by: Gothenburg Univesity Library
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

--- Comment #22 from David Gustafsson <***@gmail.com> ---
Here is a more verbose explaction of most of the changes:

- Remove syspref QueryWeightFields as this seems not to be used anyway.

- Replace hard coded 'biblios' and 'authorities' in mappings.pl with constants.

- Add _search_fields method in QueryParser for returning weighted/unweighted
search fields for OPAC/STaffClient (as optional subfield, used primarily for
authorities search).

Refactor authorities search. This part is a a litte bit iffy:

What mainly confuses me is the motivation behind the differentiation between
is/=, and "exact". With regards to elastic that operator only seems to be used
in C4/Matcher.pm. It also occurs in Koha/QueryParser/Driver/PQF.pm and
Koha/REST/Plugin/Query.pm but i don't believe Elastic is used as backend for
those. What I can gather there where actually no difference between is/= and
"exact" before my changes as is/= performed a "term" query on "<field>.phrase"
and "exact" performed a "match_phrase" on the same field, which is pointless
since the "phrase" subfield is not tokenized (used keyword analyser). So I
simplified this and use the same query for both is/= and "exact".

As "exact" queries seems to be used for matching, I also thought it would be
more fitting to perform a case insensitive match instead of also removing
punctuation (which could result in unexpected matches).

I also noticed that the Authorities type drop-down in the authorities search
was not respected, so this is not included in the query if selected.

Also noted there is an $and_or option to build_authorities_query_compat that is
not used, this should probably be addressed, but not fixed in this patch.

Tidy up the index/fields configuration a little bit:

- Rename all occurrences of misspelled "analyser" to "analyzer".
- Rename "my_normalizer" to more descriptive "nfkc_cf_normalizer".
- Rename "normalizer_keyword" to more descriptive "icu_folding_normalizer".
- Rename subfield "lc_raw" to "ci_raw" (case insensitive). I don't think this
subfield actually was case insensitive before since "my_normalizer" was used
wich is only performing UTF normalization, not case folding.
- Don't use "phrase" subfield for sorting, this is incorrect and will result in
strange results since byte order of characters is used. There is also no need
for a subfield for the sort field. I removed the subfield and changed type to
"icu_collation_keyword", which will attempt to sort in order with the least
conflicts between languages. Ideally there should be a syspref for setting
collation language to sort by. This is a pretty trivial fix so might open a new
issue for that.

As a result of this "phrase" is almost not used at all, and could probably be
removed with phrase queries on the search field instead in a later stage (which
would decrease index size considerably).
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

Séverine Queune <***@bulac.fr> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |***@bulac.fr
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-communit

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

David Gustafsson <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Status|In Discussion |Needs Signoff

--- Comment #24 from David Gustafsson <***@gmail.com> ---
Rebase against master (but currently untested).
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/

[b***@bugs.koha-community.org]

https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=20589

David Gustafsson <***@gmail.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #82544|0 |1
is obsolete| |

--- Comment #23 from David Gustafsson <***@gmail.com> ---
Created attachment 83016
-->
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=83016&action=edit
Bug 20589: Add field boosting and use query_string fields parameter

Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.

How to test:
1) Reindex authorities and biblios.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
"Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
some biblios and/or authorities searches.

Sponsorded-by: Gothenburg Univesity Library
--
You are receiving this mail because:
You are watching all bug changes.
_______________________________________________
Koha-bugs mailing list
Koha-***@lists.koha-community.org
http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs
website : http://www.koha-community.org/
git : http://git.koha-community.org/
bugs : http://bugs.koha-community.org/