Road to elasticsearch

About me

Benjamin Cremer

Benjamin Cremer
Core Developer shopware AG
Follow me @benjamincremer

Questions?

Feel free to ask at any time

Give Feedback please

joind.in/event/scd2015

joind.in/15090

shopware AG

AntiDemo

Problems

  • Duplicate Code
  • Tight Coupling to Persistence layer
  • Tight Coupling to framework
  • Primitive extensibility
  • Big Queries that select too much
  • Performance / Scale issues
  • Undocumented Arrays with minor differences

        BaseProduct < ListProduct < Product
    
BaseProduct Product Identification
ListProduct Basic Product Data (Category Listing, Related Products)
Product Full Product data, contains ListProducts (Product Detail Page)

Separation of concerns

  • Search Definition
  • Find product numbers
  • Fetch Product data by product number
  • Bridge to framework

Search Definition

  • Description of what to search for
  • You will find no WHERE or INNER JOIN here
  • Lets call that "Criteria"

Criteria


$criteria = new \Shopware\Bundle\SearchBundle\Criteria();
    

$criteria->addCondition(
    new HasFreeShippingCondition()
);
    

$criteria->addCondition(
    new SearchTermCondition('tee')
);
    

$criteria->addCondition(
    new PriceRangeCondition(5, 50)
);
    

$criteria->addSorting(
    new PriceSorting(SortingInterface::SORT_ASC)
);
    

$criteria->limit(5);
$criteria->offset(0);
    

Condition


namespace Shopware\Bundle\SearchBundle\Condition;

class PriceRangeCondition implements ConditionInterface
{
    /**
     * @param float $minPrice
     * @param float $maxPrice
     */
    public function __construct($minPrice, $maxPrice)
    {
        Assertion::float($minPrice); // see github.com/beberlei/assert
        Assertion::float($maxPrice);

        $this->minPrice = $minPrice;
        $this->maxPrice = $maxPrice;
    }

    public function getMinPrice() {};
    public function getMaxPrice() {};
}
    

Condition

  • Value Object
  • Immutable
  • Only scalars
  • Domain Specific
  • Ensures basic consistency
  • We can easily persist that
    (ProductStreams!)
  • Search Definition ✔
  • Find Product Numbers?

Context

  • Global shop state
  • User / Runtime dependent information

/** @var ContextServiceInterface $contextService */
$contextService = $container->get('shopware_storefront.context_service_core');

$context = $contextService->getProductContext();
    

$context->getShop()->getId(); // 1
$context->getFallbackCustomerGroup()->getKey(); // EK
    

Search Interface


namespace Shopware\Bundle\SearchBundle;

interface ProductNumberSearchInterface
{
    /**
     * @return ProductNumberSearchResult
     */
    public function search(Criteria $criteria, ContextInterface $context);
}
    

Search


$criteria = Criteria();
$criteria->addCondition(new SearchTermCondition('tee'));
$criteria->limit(5);
[..]
$context = [..]
    

/** @var ProductNumberSearchInterface $search */
$search = $this->get('shopware_searchdbal.product_number_search');

/** @var ProductNumberSearchResult $result */
$result = $search->search($criteria, $context);
    

echo $result->getTotalCount();

/** @var BaseProduct $product */
foreach ($result->getProducts() as $product) {
    echo $product->getNumber();
}
    
  • Search Definition ✔
  • Find Product Numbers ✔
  • Fetch Product Data ?

Fetch Product Data


interface ListProductServiceInterface
{
    /**
     * @return Struct\ListProduct
     */
    public function get($number, ContextInterface $context);

    /**
     * @return Struct\ListProduct[] Indexed by the products order number
     */
    public function getList(array $numbers, ContextInterface $context);
}
    

interface ProductServiceInterface
{
    /**
     * @return Struct\Product
     */
    public function get($number, ContextInterface $context);

    /**
     * @return Struct\Product[] Indexed by the products order number
     */
    public function getList($numbers, ContextInterface $context);
}
    

/** @var ListProductService $productListService */
$productListService = $this->get('shopware_storefront.list_product_service');

/** @var ListProduct $product */
$product = $productListService->get('sw-555', $context);

/** @var ListProduct[] $products */
$products = $productListService->getList(['sw-555', 'sw-111'], $context);
    
  • Search Definition ✔
  • Find Product Numbers ✔
  • Fetch Product Data ✔
  • Combine both?

/** @var ProductNumberSearchResult $searchResult */
$searchResult = $this->productNumberSearch->search($condition, $context);

/** @var ListProductService $productListService */
$productListService = $this->get('shopware_storefront.list_product_service');
    

$listProducts = [];
foreach ($result->getProducts() as $product) {
    $listProducts = $productListService->get($product->getNumber(), $context);
}
    

// Better:
$listProducts = $productListService->getList(
    array_keys($result->getProducts()),
    $context
);
// Produces one single IN query (default DBAL implementation)
    

namespace Shopware\Bundle\SearchBundle;

interface ProductSearchInterface
{
    /**
     * @return ProductSearchResult
     */
    public function search(Criteria $criteria, ContextInterface $context);
}
    

class ProductSearch implements ProductSearchInterface
{
    public function __construct(
        ProductNumberSearchInterface $productSearch  // Doctrine DBAL
        ListProductServiceInterface $productService, // Doctrine DBAL
    );

    /**
     * @return ProductSearchResult
     */
    public function search(Criteria $criteria, ContextInterface $context);
}
        

class ProductSearch implements ProductSearchInterface
{
    public function __construct(
        ProductNumberSearchInterface $productSearch  // Elasticsearch
        ListProductServiceInterface $productService, // Doctrine DBAL
    );

    /**
     * @return ProductSearchResult
     */
    public function search(Criteria $criteria, ContextInterface $context);
}
        

class ProductSearch implements ProductSearchInterface
{
    public function __construct(
        ProductNumberSearchInterface $productSearch  // Elasticsearch
        ListProductServiceInterface $productService, // Elasticsearch
    );

    /**
     * @return ProductSearchResult
     */
    public function search(Criteria $criteria, ContextInterface $context);
}
        

$condition = [..];
$context   = [..];

/** @var ProductSearchInterface $search */
$search = $this->get('shopware_search.product_search');

$result = $search->search($criteria, $context);

echo "Totalcount: ". $result->getTotalCount(). "\n";

/** @var ListProduct $product */
foreach ($result->getProducts() as $product) {
    echo $product->getName();
};
    
  • Search Definition ✔
  • Find Product Numbers ✔
  • Fetch Product Data ✔
  • Combine both ✔

Implementation


interface ConditionHandlerInterface
{
    /**
     * @param ConditionInterface $condition
     * @return bool
     */
    public function supportsCondition(ConditionInterface $condition);

    /**
     * @param ConditionInterface $condition
     * @param QueryBuilder $query
     * @param ContextInterface $context
     */
    public function generateCondition(
        ConditionInterface $condition,
        QueryBuilder $query,
        ContextInterface $context
    );
}
    

class ManufacturerConditionHandler implements ConditionHandlerInterface
{
    public function supportsCondition(ConditionInterface $condition)
    {
        return ($condition instanceof ManufacturerCondition);
    }

    public function generateCondition(
        ConditionInterface $condition,
        QueryBuilder $query,
        ContextInterface $context
    ) {
        /* @var $condition ManufacturerCondition */

        $query->innerJoin(
            'product',
            's_articles_manufacturer',
            'manufacturer',
            'manufacturer.id = product.manufacturerID AND product.manufacturerID IN (:manufacturer)'
        );

        $query->setParameter(
            ':manufacturer',
            $condition->getManufacturerIds(),
            Connection::PARAM_INT_ARRAY
        );
    }
}
    

class ProductNameSortingHandler implements SortingHandlerInterface
{
    public function supportsSorting(SortingInterface $sorting)
    {
        return ($sorting instanceof ProductNameSorting);
    }

    public function generateSorting(
        SortingInterface $sorting,
        QueryBuilder $query,
        ContextInterface $context
    ) {
        $query->addOrderBy('product.name', $sorting->getDirection())
              ->addOrderBy('product.id', $sorting->getDirection());
    }
}

    

Elasticsearch

“Elasticsearch is a search server based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents.”

https://en.wikipedia.org/wiki/Elasticsearch

Elasticsearch

  • Scalable search
  • Cross-platform
  • Open Source (Apache License)
  • JSON based Query Language

ElasticsearchDSL

  • Integrated in Shopware 5.1 (CE)
  • Shopware/Bundle/SearchBundle
  • Shopware/Bundle/SearchBundleDBAL

  • Shopware/Bundle/SearchBundleES
    NCLOC: 2616
  • Shopware/Bundle/ESIndexingBundle
    NCLOC: 2772

Architecture

tideways.io/profiler/blog/high-performance-shopware-51-with-elasticsearch

Shopware/Bundle/SearchBundleES


class SalesConditionHandler implements HandlerInterface
{
    public function supports(CriteriaPartInterface $criteriaPart)
    {
        return ($criteriaPart instanceof SalesCondition);
    }

    public function handle(
        CriteriaPartInterface $criteriaPart,
        Criteria $criteria,
        Search $search,
        ContextInterface $context
    ) {
        $filter = new RangeFilter(
            'sales',
            ['gt' => $criteriaPart->getMinSales()]
        );

        $search->addFilter($filter);
    }
}
    

Shopware/Bundle/ESIndexingBundle

$ ./bin/console sw:es:index:populate

$ ./bin/console sw:es:backlog:sync

Need for speed

Performance gains

100.000 Producs / 100 Categories

MySQL

Elasticsearch

tideways.io/profiler/blog/high-performance-shopware-51-with-elasticsearch

Recap

Questions?

Thank you!

Please give feedback

joind.in/15090

@benjamincremer