SolrNet Release 0.5.1: Fix Spellcheck Parser Issue

by Xavier Comments: 0

SolrNet, the C# client for Apache Solr, has a new release: 0.5.1. The current release aims to include a breaking change with the latest versions of Solr 4.x in which multiple collations are returned by Solr. I am currently working on getting it to Nuget

This is the release:

SolrNet Release 0.5.1: Fix Spellcheck Parser Issue

Let me show you quickly with an example, here is how a single collation was returned before:
<response>
<result numFound=”1″ start=”0″>
<doc>
<str name=”Key”>224fbdc1-12df-4520-9fbe-dd91f916eba1</str>
</doc>
</result>
<lst name=”spellcheck”>
<lst name=”suggestions”>
<lst name=”hell”>
<int name=”numFound”>1</int>
<int name=”startOffset”>0</int>
<int name=”endOffset”>4</int>
<arr name=”suggestion”>
<str>dell</str>
</arr>
</lst>
<lst name=”ultrashar”>
<int name=”numFound”>1</int>
<int name=”startOffset”>5</int>
<int name=”endOffset”>14</int>
<arr name=”suggestion”>
<str>ultrasharp</str>
</arr>
</lst>
<str name=”collation”>dell ultrasharp</str>
</lst>
</lst>
</response>

And then with later versions of Solr 4.x multiple collations were returned:

<response>
 <result name="response" numFound="0" start="0"></result>
 <lst name="spellcheck">
 <lst name="suggestions">
 <lst name="produtc">
 <int name="numFound">1</int>
 <int name="startOffset">0</int>
 <int name="endOffset">7</int>
 <arr name="suggestion">
 <str>product</str>
 </arr>
 </lst>
 <lst name="collation">
 <str name="collationQuery">product</str>
 <int name="hits">1000</int>
 <lst name="misspellingsAndCorrections">
 <str name="produtc">product</str>
 </lst>
 </lst>
 </lst>
 </lst>
</response>

The detail was that SolrNet would raise an issue because the numFound node was not found. Well, this issue was fixed now.

This is just the first release that I do on SolrNet since I was granted permission to provide new releases. I am merely getting up to speed to work my way through the backlog of improvements and including support for newer releases of Solr.

If you have any questions, don’t hesitate to contact me via this blog or @xmorera in Twitter.

 

SolrNet vs. SolrExpress

by Xavier Comments: 1

I have been working with Solr for a while, mainly from the .NET world and I basically love it. I use SolrNet which I think it is a very mature and stable library. I was asked today if I have ever used SolrExpress and if I recommend it over SolrNet.

The short answer is no, I have not used it. Therefore I can’t give a facts based recommendation, but looking over the source code of both libraries it is my opinion that SolrNet is still more complete. So I still believe SolrNet to be a more sensible choice.

It is worth mentioning that is a biased point of view,  as I have used SolrNet multiple times and it really has made my life a lot easier.

Having said that, besides using it several times, I have authored a few things around Solr and SolrNet and used it extensively. It works fine and I know it pretty well. It basically gets the job done, it is pretty mature and almost complete (pending SolrCloud and a few minor things like a breaking change on collation).

Some of the things I created

I created a Solr training for Pluralsight

2016-08-18_1009

https://www.pluralsight.com/courses/enterprise-search-using-apache-solr

Getting Started with Enterprise Search Using Apache Solr …
www.pluralsight.com
Search is one of the most misunderstood functionalities in the IT industry. Apache Solr brings high quality Enterprise Search to the masses.

And a SolrNet training for Pluralsight

https://www.pluralsight.com/courses/implementing-search-dotnet-applications

2016-08-18_1005

I wrote a book for a company called SyncFusion for their Succinctly Series for Solr and SolrNet

https://www.syncfusion.com/resources/techportal/ebooks/apachesolr

2016-08-18_1002

I’ve also done internal trainings, presentations and webcasts on Solr + SolrNet
http://www.meetup.com/Atlanta-Net-User-Group/events/222161640/

2016-08-18_1011

Learn How to Add Search to .NET with Solr & SolrNet …
www.meetup.com
Search is a functionality that most people take for granted while at the same time it is deeply misunderstood and usually poorly implemented. .NET

SolrNet does not have yet support for SolrCloud in the main repository, but there is one fork that already uses it but our current project does not use forks, only the main repository. If that is not a blocker for your customer, go ahead or like in our case, just use a load balancer for querying and a call to zookeeper api to get leader for indexing.

Hope this helps.

Stemming and Multi Language

by Xavier Comments: 0

I received a question today on stemming and multi language. Basically, “why do we need multiple fields in our Solr in different languages and how do I test multi language stemming?”.

First of all, let’s explain what stemming is. Stemming involves reducing words to their stem (or base or root) during indexing and querying in an effort to improve recall.

For example, if a document includes the following phrase “Xavier walked to work every morning from Westside Parkway” and a user searches for walk then the results will correctly include the document that has walk. Read more!

Configuring Spell Correction in Solr

by Xavier Comments: 0

Today I am configuring spell correction in Solr 5.5. Enabling it is not very hard. Simply select which spellcheck component you want to use, please see here for the alternatives: https://cwiki.apache.org/confluence/display/solr/Spell+Checking

There are several but I selected solr.IndexBasedSpellChecker which works for what I need. I replaced the one that comes in the solrconfig and then added spellcheck as lastcomponents. Reindexed, committed and it works.

Most people stop here, but I wanted to learn more, and so here is some very good recommended lecture to understand spellchecking better:

Getting started Spell Checking with Apache Lucene and Solr

Which references a more technical post

http://norvig.com/spell-correct.html

That goes even into more technical depth

http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/36180.pdf

http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=52A3B869596656C9DA285DCE83A0339F?doi=10.1.1.146.4390&rep=rep1&type=pdf

 

 

Monitoring a Solr Cluster

by Xavier Comments: 0

A couple of days ago I got asked, how do we monitor our cluster? Well, there are professional ways and other for the budget conscious deployment. Here are a few options that came to my mind:

  • You have the ping request handler which can be used to determine if a node is up and running – this is useful if you want to configure the load balancer to determine which nodes are responding
  • Additionally I’ve seen environments where a monitoring service uses several predefined queries that are issued at a predefined interval and will notify if no response is received. Something like http://www.site24x7.com/ but behind the firewall. I do not know which/if monitoring services you might have.
  • And there are more specialized tools, for example Sematext although some of them are more Linux friendly, so it is necessary to look for Windows counterparts if you don’t have Linux.
  • Also you can use the clusterstate.json (this would be the one from prod https:///solr/zookeeper?detail=true&path=/clusterstate.json) from Zookeepr which will tell you the state of the nodes. You just need to do a bit of parsing which can be done pretty easily with a bit of Json.Net which is easy to learn.
  • And regarding monitoring your cluster’s use in terms of queries done, you can definitively use the Solr logs and analyze queries.

Just a few thoughts I wanted to share.

How to Query for a Null or Empty Value in SOLR

by Xavier Comments: 0

I had to look for empty values in a mandatory field in SOLR today. Wait, what? Shouldn’t mandatory values in the index should be marked as required=”true” when you are defining the field?

Well yes, but some people forget to do it or maybe the spec was not fully completed at the time when they worked on the schema so they did not include it… just in case! (YAGNI definitively comes to mind)

Well, in any case I had to find which documents did not have the publication date (which sounds like a really really really mandatory field to me).

So how do you identify them?

Option A: Query *:* and start paginating taking down notes of which documents do not have the value… Ok this is totally brute force approach. But I wouldn’t be too impressed if I find someone doing it. The things I have seen…

Option B: Query *:* and in your fl include only id and publicationdate. Paginate or add enough rows. Very amateur but a bit better than before

Option C: Query *:*, include only the two fields in fl and sort asc! Much better as in your results you will have the ones with empty at the beginning.

Option Winner: or instead of *:* simply use the nice flexibility of a specific query and use q=-publicationdate:*

This is definitively the best approach and as I just demonstrated, there are many ways of finding a solution which go from biggest effort to most effective. I strive to do things always as efficiently as possible and so should you!

 

How to Send Solr Optimize Command

by Xavier Comments: 1

There are times when you want to optimize your Solr index. But what is optimize and why do I care?

Optimize is similar to when you defragment your hard drive. Solr will create a new index removing any deleted documents. It is simply house keeping at its best.

I usually do a commit from the Admin UI, going to the overview tab. Solr Optimize

 

However, sometimes we might want to do it programatically, a good example being when you have a spell checker configured to build the dictionary on optimize. The url to optimize is very simple, here is an example with my localhost, just replace with your Solr

http://localhost:8983/solr/yourcore/update?stream.body=<optimize><query>*:*</query></optimize>

 

Notice how the # is removed from all REST calls vs when the Admin UI loads.

Happy optimizing!

Easy way to do a Solr Core Reload

by Xavier Comments: 0

Many times have I stopped and restarted Solr to reload a core, yes it is kind of a rookie way as you can always go to the Admin UI, Core Admin and reload Core.

But what if you wanted to have a really fast way of reloading your core?

Just do it via the admin handler!
http://{SOLR IP}:{SOLR PORT}/solr/admin/cores?action=RELOAD&core={CORE NAME}

You can even add it to your code and make a simple call or better yet use SolrNet via the admin functionality found below:

https://github.com/mausch/SolrNet/blob/master/Documentation/Core-admin.md

 

Quick Tip: Find Missing Fields in Apache Solr

by Xavier Comments: 0

I had an issue raised because of a mismatch between my document results and my facet counts. The issue is basically that there is a field that is not required and in most cases, the field is added with an empty string – which is ok as empty has a meaning. However in a few cases, the field is not added at all and this is not the expected scenario. So I needed to find out why did this happen, which means finding the document id so that it can be reviewed during indexing.

Oh well..I was tired so I ran a *:* query and got all results… too much text.

Query for all: q=*:*

Added only the two fields that I needed in the fl field so that only those fields that I needed were shown and number of rows to see them all. This was kind of slow and inconvenient.

Query for all with only required fields and all rows: q=*:*&fl=title myfacet&rows=1600

So now I remembered query for missing fields! Just use the – operator on a field name.

Query for documents with missing fields: q=-myfacet:*

Problem solved. Easy as pie!

How to Start a Development Apache Solr

by Xavier Comments: 3

I am preparing for a presentation this month on Solr and SolrNet for the Atlanta .NET User Group.  Solr 5 is already out but I will be running my demos using Solr 4.10. Now that I am starting the preparation process, it really feels so good to know that starting a local Solr is SO EASY. Check out the steps which couldn’t be easier:

– Assuming you already downloaded Solr (here if you haven’t: http://lucene.apache.org/solr/downloads.html)

– Just extract into a folder. Mine is called AtlantaSolr

– Make sure you have Java running. If unsure just type java -version

2015-06-10_2152

– Now navigate to your Solr folder, in my case C:\Dropbox\Public Speaking\AtlantaSolrSolrNet

– Type the magic words java -jar start.jar and let it load.

– Voila! Navigate to localhost:8983/solr

2015-06-10_2155

It couldn’t be easier!