Architecture of an Enterprise Search Application

by Xavier Comments: 0

For those of you that landed directly on this page, I am working on a series of post that help people “Get Started with Enterprise Search using Apache Solr”. In this post I cover one specific module. If interested in watching the course please click here: http://pluralsight.com/training/courses/TableOfContents?courseName=enterprise-search-using-apache-solr . Click here to get to the starting point: https://www.xaviermorera.com/2014/06/getting-started-with-enterprise-search-using-apache-solr/ In the second module I start by trying to make my point on how

  • Beauty is in the eye of the beholder
  • Users usually don’t care about the application
  • They care about getting accurate and fast results.

Users don’t care about the how, engineers care about the how. The people that use it don’t. and that’s what counts! Then I focus for a few minutes on explaining architecture from two points of view:

  • Where, architecturally speaking, within an application does the search engine fit in

architecture

  • Solr’s architecture

Solr architecture


The third section of this module explains what makes Solr move and shake, mainly the search engine underneath: Lucene. And finally I explain to you the parts of a search application. This part may be a bit subjective depending on who looks at it, but most principles apply across the board. If you are interested on viewing this module you can start here:  enterprise-search-apache-solr-m2 Here is the table of contents:

You can also go back to the main page of this series by clicking here: https://www.xaviermorera.com/2014/06/getting-started-with-enterprise-search-using-apache-solr/

Solr Configuration

by Xavier Comments: 0

For those of you that landed directly on this page, I am working on a series of post that help people “Get Started with Enterprise Search using Apache Solr”. In this post I cover one specific module. If interested in watching the course please click here: http://pluralsight.com/training/courses/TableOfContents?courseName=enterprise-search-using-apache-solr . Click here to get to the starting point: https://www.xaviermorera.com/2014/06/getting-started-with-enterprise-search-using-apache-solr/

The third module is made up of four sections:

2014-07-30_1821

Installation

In this section I will show you how to install Solr. It is not as simple as solrinstaller.exe, next, next next… but it not too complicated either.

Demo

Watch, learn and follow. You will be able to get Apache Solr running in a few minutes.

Admin UI

In this section I go over each one of the options in the Admin UI. I don’t spend too much time as there are many secitons, but enough for you to understand what they are for.

Tomcat

Solr comes with Jetty server out of the box. However Jetty is not for production purposes, instead it should be for development only. I will show you how to run Solr within Tomcat. BTW: if you do not know how to do this, it may take you a few hours of research to learn on your own. Watch this clip and you will learn in a few minutes!

If you are interested on viewing this module you can start here:  enterprise-search-apache-solr-m3 

This is the table of contents:

You can also go back to the main page of this series by clicking here: https://www.xaviermorera.com/2014/06/getting-started-with-enterprise-search-using-apache-solr/

How to Add Data to a Different Solr Core

by Xavier Comments: 0

Today a good friend asked while watching my Getting Started with Enterprise Search Using Apache Solr course in Pluralsight on how to index data in different cores in Solr. Solr Cores   Here is the answer.

  • Let’s say you want to feed data to Solr’s default collection, called collection1, using the update handler, then do this:

java -Durl=http://localhost:8983/solr/update  -Dtype=text/csv -jar post.jar “Two sample courses with author test.csv”

  • But if you have a different core, i.e. psdemo, then do this:

java -Durl=http://localhost:8983/solr/psdemo/update  -Dtype=text/csv -jar post.jar “Two sample courses with author test.csv”

Easy as pie! If interested in watching the course please click here: http://pluralsight.com/training/courses/TableOfContents?courseName=enterprise-search-using-apache-solr .

Why Solr & Enterprise Search

by Xavier Comments: 0

For those of you that landed directly on this page, I am working on a series of post that help people “Get Started with Enterprise Search using Apache Solr”. In this post I cover one specific module. If interested in watching the course please click here: http://pluralsight.com/training/courses/TableOfContents?courseName=enterprise-search-using-apache-solr .Click here to get to the starting point: https://www.xaviermorera.com/2014/06/getting-started-with-enterprise-search-using-apache-solr/

I start my Enterprise Search Apache Solr training course by conveying my beliefs to you. I truly believe that a good search implementation can help tremendously with the adoption of your application or providing huge savings by letting your users find what they are looking for. I can even say that a Search Engine is a very straightforward and easily understood name, but maybe sometimes I think that you should have a Find Engine!

Anyway, in this module I start with why. I really hope we both are in the same lane. Immediately I tell you what I promise you will get from this course. It is a pretty powerful promise, but believe me it is true.

I promise… That in the next couple of hours I will teach you to build things that might take you weeks to learn on your own and we will create together a search experience that could cost thousands of dollars to build

And they I take you in a few minutes into some interesting topics, for example Why Enterprise Search. This might seem normal to you, but most people have no idea about search. Search is one of the most misunderstood functionalities in IT!

And then of course we touch on a bit of history of search engines, famous Solr sites so that you can see with your own eyes how the big guys really are using Solr! If they are, why shouldn’t you?

Also, Apache Solr is not alone. I have a personal opinion at this point that it is the best, but others disagree. Let’s all agree to disagree then, but in this training I will focus solely on Apache Solr.

Are you ready to watch? If so you can watch it here after you log in to Pluralsight: http://pluralsight.com/training/Player?author=xavier-morera&name=enterprise-search-apache-solr-m1&mode=live&clip=0&course=enterprise-search-using-apache-solr

Here is the table of contents for this module:

You can also go back to the main page of this series by clicking here: https://www.xaviermorera.com/2014/06/getting-started-with-enterprise-search-using-apache-solr/

 

These 5 things are destroying your meetings

by Xavier Comments: 0

This post originally appeared in Pluralsight’s blog in http://blog.pluralsight.com/tips-for-meetings I am including it here as future reference of all my writings

Meetings can be wildly useful for things like brainstorming and communicating within — and between — teams. They can also be great for sparking collaboration and moving projects forward. But when they get  misused or abused, which happens all too often, they end up being little more than a gigantic waste of time. Let’s look at the top five things that tend to go wrong in meetings, and how you can fix them.

1. Busyness vs. Productivity. For some people, the mere act of sitting in a meeting is considered work. This becomes an issue when those in attendance use the time to appear busy, rather than actively, and constructively, contributing. Meetings that go over in this manner tend to end with scheduling another meeting just to “continue the discussion.” This is all too common among project managers or product managers.

The fix: Make sure every meeting you attend or lead has a real and very clear objective. Moreover, make sure that this meeting is actually necessary. If the objective is not important or required, then simply defer the meeting until the right time.

2. The time trap: Have you ever been in a one hour meeting with 15 other folks and quietly agonized over your workload, rather than actively listening and waiting for your two minute turn to talk? And what about the guy next to you who seems to be thinking the same thing, while he plays Candy Crush or answers emails? An hour long meeting with this many people is actually two days of work lost forever; gone in time. I shiver just to think of how many dollars are wasted because of overcrowded meetings.

The fix: Not everyone needs to be invited to every meeting. Make sure each person has a clear understanding of why they’re there. Reserve a friendly kick out of the room for anyone who doesn’t need to be around. Also, as the old adage says, “divide and conquer.” A very broad topic where too many people are involved can be broken down into smaller chunks with smaller groups. You only need one person to coordinate among teams, and one is always much better than a committee. This is very common with Scrums that get abused and end up looking more like office parties.

3. No direction: Another reason why some meetings take much longer than expected is that there isn’t a clear agenda or objective. This leads to endless talking, tangents and scheduling new meetings to continue the conversation. People also confuse a meeting agenda with the subject of the meeting. For example, “Discussion on data” makes a fine subject line, but it’s far too vague to serve as the agenda. This scenario also comes into play when the overall vision for the project is not made clear or has many unknown factors.

The fix: Create a clear agenda with well-defined points and, if possible, use my favorite feature of Scrum: timebox. Parkinson’s Law states that “work expands so as to fill the time available for its completion,” and this applies just as well to meetings. If you don’t have a limit, people will use as much time as possible for a discussion. Timebox will help you maintain control over this. And, if the issue is overall project vision, try planning shorter term until there is a clearer view of the road ahead.

4. Too many interruptions: If you’re into self-improvement, you may be familiar with the concept of “flow.” In a nutshell, flow is when you’re totally immersed in the task at hand; when you’re your most productive. Developers usually experience this when time suddenly flies by.

Something else you should be familiar with that is the “Maker’s Schedule, Manager’s Schedule.” If you’re a maker (in this case, let’s use programmer), you likely work in large chunks of time, say from 8 a.m. through noon. During that time you focus on solving a problem, but if you’re interrupted, it may take up to 15 minutes to pick up where you left off. This means that if there are multiple meetings throughout the day, your maker’s schedule gets broken down into many small chunks that do not allow for full, focused involvement in a task.

The fix: Managers usually have smaller chunks of work that can be broken down into one hour slots, hence their impulse to “book meetings with the team to catch up.” They may have the best intentions, but this usually has a negative impact. Even worse, managers are higher in the food chain, which makes it difficult to cancel a meeting with your boss based on the fact that he was interrupting you.

If you’re a manager, try to understand the impact that you have on your whole team with these interruptions. Aim to schedule meetings when you know they’ll have the least impact on workflow – this can be near the start, middle or end of the day. If you’re a programmer, or a “maker,” find a way to approach your manager and see if he can work out a better method for scheduling these meetings. It can also be helpful to use email, chat or a tool like Jira to manage your team’s progress.

5. Too much talk, not enough action: This is a tricky one, as it requires a leader who is aware of the capabilities and responsibilities of his entire team (or at least those who will be in attendance). Thanks to extreme differences in personalities, this can be tricky. Typically you end up with some people who talk too much and others who remain totally silent. Unfortunately, talking does not always equate to doing. Someone might talk a lot and do very little, while the guy doing the work just sits and listens. This creates an unbalanced and biased view, as not all of the involved parties will communicate appropriately and the outcome of the meeting might not be the best.

The fix: It is the meeting organizer’s responsibility to understand the involvement of each person invited to the meeting in respect to the meeting agenda. Then ask questions, specifically on each one of the topics, to get the correct information and help obtain the best possible outcome for everyone involved.

To quickly recap, meetings can be the best way to help your team move forward. Just remember to schedule them only when absolutely required, instead of as a way to keep busy; invite only those who really need to be there; have a clear agenda and objective; don’t interrupt the maker’s schedule; and guide the meeting in such a way that everyone is heard.

Resource Kanban

by Xavier Comments: 0

There is no doubt that Kanban is great for software projects. With Kanban you can increase throughput by limiting work in progress, delivering value constantly this increasing trust in the team.

But Kanban does not only apply to projects. Some people use also Portfolio Kanban to manage projects.

And there is another potential use for Kanban that I think can be very useful and solve multiple problems if applied properly, i.e. in a services company: Resource Kanban.

Let’s start by outlining the problem:

  • In most companies, but especially sevices oriented companies, resourcing is always an issue. Projects usually start when customers want to or when legal decides they should
  • Also current projects might end up unexpectedly, ahead of time (yeah right!) or contracts are extended, sometimes longer terms but also little by little
  • Predicting when each project ends and others begin in such a way that resources can be allocated then becomes a really tricky proposition. On one hand if you don’t have a resource available to start another project then you might upset a customer and potentially lose little (or lots of) money. On the other hand, if there is a lot of time between one project’s end and another’s start then your developer is idle and you also lose money! Tricky indeed
  • So being able to visualize and maximize utilization as much as possible, without burning out developers, is critical

And thus it came to my mind that Kanban can be used to help this process. It is not a solution per se, but it is a method that allows to help schedule resources and understand potential idle and over utilized times.

And so I will explain what in my mind Resource Kanban can do for you. Stay tuned for a step by step explanation.

 

Disclaimer: a quick search on “Resource Kanban” indicates to me that no one has yet presented this solution to the problem in the way that I envision it. Please let me know if you already did so.

Course and Blog

by Xavier Comments: 0

And today I made it to the front of the marketing email from Pluralsight with my blog post and my course!

2014-07-02_1058_001

Bad Search Can Break a Good App – Real World Example

by Xavier Comments: 0

In my Pluralsight training, Getting Started with Enterprise Search using Apache Solr, one of the things that I make quite a bit of emphasis is on how important search is, yet it is one of the most misunderstood functions of IT and development in general. In this post I will show you an example of how a potentially good app is a pretty bad app mainly because of its search capabilities.

It is so much the case that in Twitter Pluralsight selected this phrase to tweet about the release of my course as you can see here:

searchiseverywhere

But now let’s get to the sample. Here’s the scenario:

Problem: Life is busy. No time to go to the supermarket

Solution: use your grocery store’s web site to purchase your food and it gets delivered home the next day. Charming idea, did not work with Webvan, but it seems to be doing quite well for Amazon and in my home town one of the major supermarkets is doing it in a more controlled way with a good delivery service, all for $10. Not too scalable, but for a MVP it is ok. (Read Lean Startup if you don’t know what MVP is)

It may work or maybe not mainly because of a really bad user experience, but let me get to the point. UX is important! Never forget it!

You get to the app in https://www.automercado.co.cr/aam/showMain.do and they have mainly 4 sections as you can see here

auto

And here is what they are for:
– On the left they have a directory style organized by aisle. Grouping kind of works in my opinion if you are not too sure of what you want, but it is terribly slow and inefficient. They lose cookie points for this.

2014-07-02_0638

– Then in the middle they have a section where they display the products. This is very standard so it kind of goes through, however they lose cookie points again for having products without pictures or with very weird stretching. They are a supermarket, and a big one, so I am sure they can send a guy with an iPhone to take a quick picture.

2014-07-02_0637

– The cart has a problem which is that they do not actually display the product name, only the description. Who thought of this? Not even something as simple as a tooltip!

2014-07-02_0640

And then here is the deal breaker for me: BAD SEARCH! As mentioned in the post, search is one of the most misunderstood functionalities in IT. A lot of people make huge mistakes because search can be done with a database, which it can, but the end results sucks! And it did suck here.

Let me show you this. I want to look for “jabon dial” which means “Dial Soap”. So I just type “Jabon Dial”. Should work, right? It doesn’t! Look at the message: “No results found…”. Also I hate the CAPS. There may be 1 technical reason I can think of but it is pretty dumb.

2014-07-02_0646

But why? If you look closely there are 27 types of “Jabon Dial”, type only Dial

2014-07-02_0649

The problem lies here:
– The person that implemented this application had no knowledge of how search works, which is normal as search is pretty misunderstood.
– But humans don’t do search like engineers want. Having the user do a search exactly like the engineer wants is just lazy and ineffective.
– So engineers who created this probably went for a simple exact match in a database search
– This is a terrible user experience. I can bet the farm that Amazon would have closed its doors in the 1990s if they had such a bad search

How to fix it? Well, go learn how to use a search engine. And that’s why I created my course, Getting Started With Enterprise Search Using Apache Solr: http://www.pluralsight.com/training/Courses/TableOfContents/enterprise-search-using-apache-solr